PUMPKIN ECLIPSE — Mystery malware destroys 600,000 routers from a single ISP during 72-hour span An unknown threat actor with equally unknown motives forces ISP to replace routers.

Dan Goodin – May 30, 2024 2:00 pm UTC EnlargeGetty Images reader comments 0

One day last October, subscribers to an ISP known as Windstream began flooding message boards with reports their routers had suddenly stopped working and remained unresponsive to reboots and all other attempts to revive them.

The routers now just sit there with a steady red light on the front, one user wrote, referring to the ActionTec T3200 router models Windstream provided to both them and a next door neighbor. They won’t even respond to a RESET.

In the messageswhich appeared over a few days beginning on October 25many Windstream users blamed the ISP for the mass bricking. They said it was the result of the company pushing updates that poisoned the devices. Windstreams Kinetic broadband service has about 1.6 million subscribers in 18 states, including Iowa, Alabama, Arkansas, Georgia, and Kentucky. For many customers, Kinetic provides an essential link to the outside world.

We have 3 kids and both work from home, another subscriber wrote in the same forum. This has easily cost us $1,500+ in lost business, no tv, WiFi, hours on the phone, etc. So sad that a company can treat customers like this and not care.

After eventually determining that the routers were permanently unusable, Windstream sent new routers to affected customers. Black Lotus has named the event Pumpkin Eclipse. A deliberate act

A report published Thursday by security firm Lumen Technologies Black Lotus Labs may shed new light on the incident, which Windstream has yet to explain. Black Lotus Labs researchers said that over a 72-hour period beginning on October 25, malware took out more than 600,000 routers connected to a single autonomous system number belonging to an unnamed ISP.

While the researchers arent identifying the ISP, the particulars they report match almost perfectly with those detailed in the October messages from Windstream subscribers. Specifically, the date the mass bricking started, the router models affected, the description of the ISP, and the displaying of a static red light by the out-of-commission ActionTec routers. Windstream representatives declined to answer questions sent by email.

According to Black Lotus, the routersconservatively estimated at a minimum of 600,000were taken out by an unknown threat actor with equally unknown motivations. The actor took deliberate steps to cover their tracks by using commodity malware known as Chalubo, rather than a custom-developed toolkit. A feature built into Chalubo allowed the actor to execute custom Lua scripts on the infected devices. The researchers believe the malware downloaded and ran code that permanently overwrote the router firmware. Advertisement

We assess with high confidence that the malicious firmware update was a deliberate act intended to cause an outage, and though we expected to see a number of router make and models affected across the internet, this event was confined to the single ASN, Thursdays report stated before going on to note the troubling implications of a single piece of malware suddenly severing the connections of 600,000 routers.

The researchers wrote:

Destructive attacks of this nature are highly concerning, especially so in this case. A sizeable portion of this ISPs service area covers rural or underserved communities; places where residents may have lost access to emergency services, farming concerns may have lost critical information from remote monitoring of crops during the harvest, and health care providers cut off from telehealth or patients records. Needless to say, recovery from any supply chain disruption takes longer in isolated or vulnerable communities.

After learning of the mass router outage, Black Lotus began querying the Censys search engine for the affected router models. A one-week snapshot soon revealed that one specific ASN experienced a 49 percent drop in those models just as the reports began. This amounted to the disconnection of at least 179,000 ActionTec routers and more than 480,000 routers sold by Sagemcom. EnlargeBlack Lotus Labs

The constant connecting and disconnecting of routers to any ISP complicates the tracking process, because its impossible to know if a disappearance is the result of the normal churn or something more complicated. Black Lotus said that a conservative estimate is that at least 600,000 of the disconnections it tracked were the result of Chaluba infecting the devices and, from there, permanently wiping the firmware they ran on.

After identifying the ASN, Black Lotus discovered a complex multi-path infection mechanism for installing Chaluba on the routers. The following graphic provides a logical overview. EnlargeBlack Lotus Labs

Further ReadingMystery solved in destructive attack that knocked out >10k Viasat modemsThere aren’t many known precedents for malware that wipes routers en masse in the way witnessed by the researchers. Perhaps the closest was the discovery in 2022 of AcidRain, the name given to malware that knocked out 10,000 modems for satellite Internet provider Viasat. The outage, hitting Ukraine and other parts of Europe, was timed to Russia’s invasion of the smaller neighboring country.

A Black Lotus representative said in an interview that researchers can’t rule out that a nation-state is behind the router-wiping incident affecting the ISP. But so far, the researchers say they aren’t aware of any overlap between the attacks and any known nation-state groups they track. Advertisement

The researchers have yet to determine the initial means of infecting the routers. It’s possible the threat actors exploited a vulnerability, although the researchers said they aren’t aware of any known vulnerabilities in the affected routers. Other possibilities are the threat actor abused weak credentials or accessed an exposed administrative panel. An attack unlike any other

While the researchers have analyzed attacks on home and small office routers before, they said two things make this latest one stand out. They explained:

First, this campaign resulted in a hardware-based replacement of the affected devices, which likely indicates that the attacker corrupted the firmware on specific models. The event was unprecedented due to the number of units affectedno attack that we can recall has required the replacement of over 600,000 devices. In addition, this type of attack has only ever happened once before, with AcidRain used as a precursor to an active military invasion.

They continued:

The second unique aspect is that this campaign was confined to a particular ASN. Most previous campaigns weve seen target a specific router model or common vulnerability and have effects across multiple providers networks. In this instance, we observed that both Sagemcom and ActionTec devices were impacted at the same time, both within the same providers network.This led us to assess it was not the result of a faulty firmware update by a single manufacturer, which would normally be confined to one device model or models from a given company. Our analysis of the Censys data shows the impact was only for the two in question. This combination of factors led us to conclude the event was likely a deliberate action taken by an unattributed malicious cyber actor, even if we were not able to recover the destructive module.

With no clear idea how the routers came to be infected, the researchers can only offer the usual generic advice for keeping such devices free of malware. That includes installing security updates, replacing default passwords with strong ones, and regular rebooting. ISPs and other organizations that manage routers should follow additional advice for securing the management interfaces or administering the devices.

Thursday’s report includes IP addresses, domain names, and other indicators that people can use to determine if their devices have been targeted or compromised in the attacks. reader comments 0 Dan Goodin Dan Goodin is Senior Security Editor at Ars Technica, where he oversees coverage of malware, computer espionage, botnets, hardware hacking, encryption, and passwords. In his spare time, he enjoys gardening, cooking, and following the independent music scene. Advertisement Channel Ars Technica ← Previous story Related Stories Today on Ars