RC RANDOM CHAOS

NSANet went down Tuesday after a PHP patch

The NSA's classified NSANet went offline this week after a PHP patch left some systems unable to restart. No cyberattack, per the DNI.

· 3 min read
NSANet went down Tuesday after a PHP patch

The NSA’s classified internal network, NSANet, went down this week. Director of National Intelligence Jay Clayton attributed it to patching. “The patching that has gone on following that has caused some of these systems to not come back up when you patch them as expected,” he said on Fox Business. The patch addressed major PHP vulnerabilities discovered last month, the same flaws the NSA had publicly urged organizations to fix.

The agency’s own framing was softer. “The National Security Agency experienced temporary degradation during scheduled routine maintenance,” it said Tuesday afternoon, adding that “mission-critical capabilities remain fully operational” and that its infrastructure “is designed with multiple layers of redundancy.” Scheduled maintenance and a patch that stops systems coming back up are not the same story, and the day’s timeline points to the second one.

The timeline reads like a failed rollout

NSA employees track their reporting status through a three-level code system left on a voicemail inbox. Tuesday morning it was “code yellow,” the second-highest level, telling non-emergency staff to arrive in staggered waves and emergency personnel to come as usual. By Tuesday evening the inbox had reverted to “code green,” report as scheduled. Then, Tuesday night, it moved to “code orange,” closing the Fort Meade campus to most employees until further notice. The agency told staff they could return to normal and pulled that back within hours. That back-and-forth is what a restoration looks like when you think a fix has held and it hasn’t.

What was being patched

The vulnerabilities sit in PHP, which runs a large share of the web. According to the reporting, AI tools helped researchers exploit them, which set off a scramble across governments and companies over the past few weeks. The flaws let an attacker read process memory that can hold secrets like passwords. That is the class of bug you patch immediately, and the NSA did. The cost showed up on the other side: apply the fix across a fleet of classified systems and some of them do not come back.

There is no evidence of a cyberattack, according to multiple sources and to Clayton directly. This was self-inflicted, in the way that aggressive patching of a serious memory-disclosure bug tends to be.

For anyone who runs these systems

The useful lesson is about the patch, not the PHP CVE. A patch that prevents a service from restarting is a failure mode you plan for the same way you plan for the vulnerability. Validate that a patched node actually returns to service before you roll the change fleet-wide. Keep a tested rollback that does not depend on the thing you just broke. Stage the rollout so a bad patch takes down a slice instead of the whole estate. Redundancy only helps if the failure is uncorrelated; patch every node with the same broken change in the same window and the redundancy patches itself out.

A former senior NSA official described NSANet as “more of a network of networks” where “it’s unlikely that a full outage exists given the many redundancies in place.” The same official called the event “alarming, but likely not crippling,” noting it “could impact analysis and reporting, but for highest priority requirements they would find a way to work those.” Officials said the outage would not affect the ability to pass urgent intelligence to the president, who was traveling in Asia, and that Five Eyes intelligence sharing with the UK, Canada, Australia and New Zealand was not affected.

The precedent for how long this can run is the agency’s own: in 2000, a tech failure downed NSA computers for nearly three days and interrupted intelligence processing at Fort Meade.

Share

Keep Reading

Latest on the Wire

Full wire →

New signal daily · RSS

Stay in the loop

New writing delivered when it's ready. No schedule, no spam.