NSANet went down Tuesday after a PHP patch
The NSA's classified NSANet went offline this week after a PHP patch left some systems unable to restart. No cyberattack, per the DNI.
The NSA’s classified internal network, NSANet, went down this week. Director of National Intelligence Jay Clayton attributed it to patching. “The patching that has gone on following that has caused some of these systems to not come back up when you patch them as expected,” he said on Fox Business. The patch addressed major PHP vulnerabilities discovered last month, the same flaws the NSA had publicly urged organizations to fix.
The agency’s own framing was softer. “The National Security Agency experienced temporary degradation during scheduled routine maintenance,” it said Tuesday afternoon, adding that “mission-critical capabilities remain fully operational” and that its infrastructure “is designed with multiple layers of redundancy.” Scheduled maintenance and a patch that stops systems coming back up are not the same story, and the day’s timeline points to the second one.
The timeline reads like a failed rollout
NSA employees track their reporting status through a three-level code system left on a voicemail inbox. Tuesday morning it was “code yellow,” the second-highest level, telling non-emergency staff to arrive in staggered waves and emergency personnel to come as usual. By Tuesday evening the inbox had reverted to “code green,” report as scheduled. Then, Tuesday night, it moved to “code orange,” closing the Fort Meade campus to most employees until further notice. The agency told staff they could return to normal and pulled that back within hours. That back-and-forth is what a restoration looks like when you think a fix has held and it hasn’t.
What was being patched
The vulnerabilities sit in PHP, which runs a large share of the web. According to the reporting, AI tools helped researchers exploit them, which set off a scramble across governments and companies over the past few weeks. The flaws let an attacker read process memory that can hold secrets like passwords. That is the class of bug you patch immediately, and the NSA did. The cost showed up on the other side: apply the fix across a fleet of classified systems and some of them do not come back.
There is no evidence of a cyberattack, according to multiple sources and to Clayton directly. This was self-inflicted, in the way that aggressive patching of a serious memory-disclosure bug tends to be.
For anyone who runs these systems
The useful lesson is about the patch, not the PHP CVE. A patch that prevents a service from restarting is a failure mode you plan for the same way you plan for the vulnerability. Validate that a patched node actually returns to service before you roll the change fleet-wide. Keep a tested rollback that does not depend on the thing you just broke. Stage the rollout so a bad patch takes down a slice instead of the whole estate. Redundancy only helps if the failure is uncorrelated; patch every node with the same broken change in the same window and the redundancy patches itself out.
A former senior NSA official described NSANet as “more of a network of networks” where “it’s unlikely that a full outage exists given the many redundancies in place.” The same official called the event “alarming, but likely not crippling,” noting it “could impact analysis and reporting, but for highest priority requirements they would find a way to work those.” Officials said the outage would not affect the ability to pass urgent intelligence to the president, who was traveling in Asia, and that Five Eyes intelligence sharing with the UK, Canada, Australia and New Zealand was not affected.
The precedent for how long this can run is the agency’s own: in 2000, a tech failure downed NSA computers for nearly three days and interrupted intelligence processing at Fort Meade.
Keep Reading
Citrix NetScalerCitrix patches a pre-auth NetScaler Console bypass
Citrix patched 12 NetScaler and Secure Access Client flaws, including a pre-auth NetScaler Console authentication bypass to patch first.
cybersecurityZeabur breach exposes 3.22 million environment-variable records
A practitioner-focused method for separating systemic security failures from routine breach-cycle operations.
rubyTwenty-one ß characters walk one byte past Ruby's heap
A one-byte heap overflow in Ruby's single-byte case conversion gives pre-auth RCE; fixed in Ruby 3.3.14, 3.4.13 and 4.0.10.
Latest on the Wire
Full wire →- 15 Million Web Servers: Ports Aren't the Real BottleneckHacker News
- Anthropic Prices $83 IPO, Shares to Trade on NasdaqHacker News
- Building an 8-page zine from A4: Local-first editor automates layoutHacker News
- Claude Cracks Two Black Hole Physics ProblemsHacker News
New signal daily · RSS
Stay in the loop
New writing delivered when it's ready. No schedule, no spam.