A SQLite underflow, and the flood behind it

In November 2024, an LLM agent named Big Sleep found an exploitable stack buffer underflow in SQLite before the code shipped. Google’s Project Zero and DeepMind ran the harness. The bug sat in a development branch, caught pre-release, an issue that existing fuzzers and human review had walked past. It was the first public case of an AI agent surfacing a memory-safety defect in widely-deployed software that conventional tooling missed. That is the fact. The narrative built on top of it - AI is hollowing out defender skill, the early results are in, the results are bad - misreads what actually happened at the mechanism level. The skill did not leave. The volume changed. And volume is the thing nobody budgeted for.

The claim under examination is specific. Automation is making analysts worse. Junior researchers lean on copilots and never learn to read a crash dump. SOC operators trust model-generated triage and stop reading raw events. The craft atrophies. Some of that is real and observable. But the framing treats AI as a replacement function - a thing that does what the human did, slightly worse, until the human forgets how. That is not the operating condition. AI is not a replacement function for the defender. It is a volume multiplier on the defender’s input. Discovery is automated. Code generation is automated. Vulnerability report generation is automated. Each of those scales the raw material an analyst has to process. The burden moves off reactive patching and onto proactive detection and response. The skillset that retains value moves with it. The researchers who decline are not declining because AI replaced a skill. They are declining because they kept practising the old skill against an input rate that broke it.

Start with the code-generation side, because it sets the supply rate. In 2021, researchers at NYU published the study commonly cited as ‘Asleep at the Keyboard.’ They drove GitHub Copilot through 89 security-relevant completion scenarios. Roughly 40 percent of the generated programs contained a weakness - CWE-89 SQL injection, CWE-79 cross-site scripting, CWE-798 hardcoded credentials, CWE-22 path traversal, CWE-787 out-of-bounds write. The mechanism is not mysterious. A code model predicts the statistically likely continuation of a prompt. Its training corpus includes a decade of vulnerable Stack Overflow answers, abandoned tutorials, and pre-patch source. The model reproduces the center of that distribution. The center of that distribution is not secure. The model is not malicious and it is not stupid. It is average, and average code has bugs.

That 40 percent figure is from a single study under controlled prompts, and later assistants with security-tuned post-training shifted the number. Hold the precise percentage loosely. Hold the direction tightly. The volume of code entering production went up - every engineer with a copilot ships more lines per day. Review capacity did not move at the same rate. The delta between code generated and code actually read by a human who understands the bug class is the new exposure. It is the same structural failure as an unmonitored CI pipeline. Output exceeded inspection. The defect rate per line may even have dropped. The defect rate per line is irrelevant when the line count tripled and nobody is reading.

This is where the supply-chain mechanics turn ugly, because the model does not only reproduce vulnerable patterns - it invents dependencies that do not exist. Researchers documented the behaviour through 2024 and 2025. Ask a model to write code that needs a package, and it will sometimes import a library that was never published. The hallucinated name is plausible. It follows the ecosystem’s naming conventions. It looks like something that should exist. Across large sample runs, a meaningful fraction of these phantom package names repeat - the model hallucinates the same non-existent dependency consistently, because the hallucination is also a function of the training distribution. The attack writes itself. Register the hallucinated name on PyPI or npm. Populate it with a working payload. Wait. The model keeps recommending it. Developers keep installing it. This is slopsquatting, and it maps cleanly to MITRE T1195.001, compromise of software dependencies and development tools. The trust boundary is no longer the developer’s typo, as in classic typosquatting. The trust boundary is the model’s confidence, and the model is confident about things that are not real. No human typed the wrong name. The machine recommended a name that an attacker pre-positioned to be wrong.

Now the offensive discovery side, which is where the volume claim gets concrete. Daniel Stenberg maintains curl. Through 2024 and into 2025 he publicly described the state of the project’s bug-bounty inbox on HackerOne as flooded with AI-generated submissions - reports that are fluent, well-formatted, confident, and wrong. He called it AI slop. The mechanism is the same one that produces hallucinated packages, pointed at vulnerability disclosure instead of code completion. A model is asked to find a security bug in curl. It produces a report that has the shape of a real finding - a CVE-style title, a CVSS vector, a plausible function name, a description of a use-after-free or a buffer overflow - and none of the underlying mechanism is true. The function does not do what the report says. The overflow does not exist. But triaging the report to confirm that it is false still costs a human maintainer real time. The asymmetry is the point. Generating a plausible false positive costs the submitter a single API call. Refuting it costs a senior engineer twenty minutes of reading source. That ratio is the operational burden shift in its purest form, and it lands on the smallest teams hardest.

The inverse exists too, and it is not slop. In 2025, autonomous offensive tooling - XBOW the most public example - climbed the HackerOne US leaderboard to the top position by submitting valid, reproduced vulnerabilities at machine rate. Not hallucinations. Confirmed findings, triaged and accepted. The relevant detail is not that an AI placed first. The relevant detail is the throughput. A system that runs reconnaissance, identifies candidate injection points, generates a test case, validates the response, and files a structured report - at a rate no human researcher matches - changes the arithmetic of every program it touches. MITRE catalogues this as T1595, active scanning, and T1190, exploit public-facing application, executed at a cadence the defender’s patch cycle was never designed to absorb. The defender’s mean-time-to-patch was calibrated against human-paced discovery. Human-paced discovery is no longer the input.

Feed that into the vulnerability-management pipeline and the second-order failure appears. CVE publication volume hit record levels through 2024, with the year’s published identifiers crossing forty thousand. In February 2024, NIST announced it was reducing the enrichment of incoming CVE records in the National Vulnerability Database. The downstream effect was a large and growing backlog of published CVEs that lacked the analysis data - CVSS scoring, CPE applicability, CWE mapping - that vulnerability-management tools consume to make prioritisation decisions. The identifier exists. The structured context behind the identifier lags. Every scanner, every SBOM tool, every patch-prioritisation workflow that assumed enriched NVD data as a free input inherited the gap. This is what the volume shift looks like at the ecosystem layer. It is not that any single bug got worse. It is that the rate of bugs outran the human institution built to characterise them, and the institution publicly said so.

Set the deskilling thesis against that backdrop and it inverts. The skill that is becoming less valuable is the one-bug-at-a-time skill - the manual reverse-engineering of a single crash, performed start to finish by hand, as the primary daily activity. That skill is not worthless. It is the foundation, and a researcher who cannot read a crash dump cannot validate anything a model produces. But it does not scale to the input rate, and a defender who only has that skill is now permanently behind the queue. The skills gaining value are different in kind. First, verification at speed - the ability to take a model-generated finding or a model-generated patch and determine in minutes whether the underlying mechanism is real. Second, telemetry engineering - building detection that scales to a volume of activity no analyst can watch event by event. Third, adversarial simulation - generating attacker activity on purpose, in a controlled environment, to find the detection gaps before the real volume does. None of those is a softer skill than manual reversing. Each requires more, not less, understanding of the underlying mechanism. The narrative that AI deskills the defender assumes the only real skill was the manual one. The manual one was always table stakes.

This is the point where the real-world actor data matters, because it disciplines the hype in both directions. In February 2024, Microsoft and OpenAI jointly published findings on nation-state actors using large language models. The named clusters included Forest Blizzard, the actor tracked elsewhere as APT28; Charcoal Typhoon and Salmon Typhoon, both PRC-linked; Crimson Sandstorm, Iran-linked; and Emerald Sleet, DPRK-linked. The finding that matters is what these actors used the models for. Reconnaissance on targets and technologies. Scripting assistance and debugging. Translation and social-engineering content generation. Drafting phishing lures with fewer language tells. What the report explicitly did not find was novel capability - no new exploit primitive, no class of attack that did not previously exist, no automated zero-day chain. The models delivered productivity, not new physics. That is the honest framing, and it is the one to hold. The threat is not that AI invents a new category of attack. The threat is that AI removes the friction from existing categories, and friction was doing more defensive work than anyone measured.

The 2025 data sharpened it. Anthropic published an account of disrupting misuse of its Claude models, including a campaign it tracked as GTG-2002 - described as a ‘vibe hacking’ extortion operation in which an actor used the model agentically across the intrusion lifecycle, from reconnaissance and credential handling to drafting extortion demands calibrated against the victim’s finances. Again, the techniques are not new. T1071 for command-and-control over standard protocols, T1567 for exfiltration over web services, T1657 for financial theft and extortion. What is new is that one operator drove the full chain with model assistance at each stage, compressing what historically required a team with distinct specialisations into a single agent-augmented workflow. The capability ceiling did not rise. The capability floor did. More operators can now reach a competent intrusion, which means the volume of competent intrusions rises even if no single intrusion is more sophisticated than what a skilled team could already do. The defender does not face a smarter adversary. The defender faces more adversaries operating at the old skilled level, and ‘more at the same level’ is its own failure mode for any detection program tuned to a lower base rate.

That is the entire defensive problem in one line. Detection programs are tuned to base rates. AI moves the base rate. Which brings the analysis to telemetry, because telemetry is where the volume either gets caught or gets missed, and it is where the deskilling risk is most acute and most misunderstood.

Consider what AI-augmented intrusion actually produces on the wire and in the host logs, because the answer is deflating for anyone expecting an exotic signature. A model-assisted attack does not generate model-shaped telemetry. It generates the telemetry of the technique it executed. A PowerShell loader written with model assistance still produces Sysmon Event ID 1 on process creation, with the command line captured, and Windows Security Event ID 4688 alongside it. If that loader spawns from an Office process, the parent-child relationship - winword.exe spawning powershell.exe - is the same anomaly it was in 2018, T1059.001, and it fires the same correlation rule. Process injection produced by model-generated code still trips Sysmon Event ID 8 on CreateRemoteThread and Event ID 10 on process access, with the granted-access mask that flags a handle opened to lsass.exe, T1003.001, T1055. A model-drafted phishing payload that beacons out still opens a Sysmon Event ID 3 network connection and a Sysmon Event ID 22 DNS query that a competent egress rule should surface. The model wrote cleaner code, faster. The behaviour on the host is behaviour, and behaviour is what the EDR sees. This is the load-bearing fact for the defender. AI does not give the attacker invisibility. It gives the attacker speed and polish. The detection logic that keys on technique behaviour rather than static signature degrades far less than the static-signature crowd fears.

The places it does degrade are specific and worth naming precisely. Signature-based and reputation-based controls degrade, because the model trivially produces novel variants - every generated loader is a unique string, so any detection keyed on file hash, on a known byte sequence, or on a previously-seen template is bypassed by default. That was already true with commodity packers and crypters; AI lowers the cost of variant generation to near zero, which means the half-life of any static IOC shrinks. Phishing-content heuristics degrade, because the linguistic tells that classifiers and trained humans relied on - awkward phrasing, translation artifacts, formatting errors - are precisely what a language model removes. A lure generated by a competent model reads like internal corporate English, which is the whole point of T1566. And volumetric thresholds degrade, because tooling that operates at machine rate can either blow past a rate-based rule loudly or, more dangerously, be tuned to stay under it deliberately. The gap is not that the EDR went blind. The gap is that the controls which depended on the attacker being slow, sloppy, or repetitive lost the assumption they were built on.

Now the deskilling risk in the SOC, stated without softening, because this is the part of the narrative with genuine substance. The dangerous failure mode is not the analyst who uses AI to triage. It is the analyst who uses AI to triage and stops reading the underlying events. A model-driven triage layer that auto-closes alerts it scores as benign is a control that fails silently. When it is wrong, it is wrong in the direction of suppression - it closes the true positive, and no human ever sees the Event ID 4624 logon from an impossible geography or the Event ID 4672 special-privilege assignment that should have escalated. The skill that atrophies here is real and the consequence is real. But the fix is not to ban the AI triage layer. The fix is to treat it as an untrusted detection source - to sample its decisions, to red-team its suppressions, to keep a human in the loop on the closes rather than the opens. The analyst who can no longer read raw telemetry cannot perform that supervision, which is exactly why raw-telemetry literacy becomes more valuable as triage automation spreads, not less. The skill is not obsolete. The skill changed jobs. It moved from first-pass triage, where the machine is genuinely faster, to supervision and validation of the machine, where the machine cannot supervise itself.

This is the new core competency and it deserves a name without marketing attached to it. AI-driven telemetry analysis is not the SOC buying a product with ‘AI’ on the box. It is the discipline of running detection logic over a volume of events no human can read sequentially, while retaining the human capacity to validate what that logic surfaces and to notice what it suppresses. It has concrete components. Detection-as-code, where detections are version-controlled, tested, and deployed like software, with Sigma as the portable rule format that compiles to the backend in use. Behavioural baselining, where the system learns the normal distribution of process lineage, network egress, and authentication patterns so that a model-assisted intrusion executing standard techniques still deviates from the host’s own established baseline even when it matches no static signature. And alert validation, where the output of any automated triage is itself monitored as a fallible component. A defender practising this is not less skilled than the analyst who read every event by hand in 2015. They are differently skilled, against a volume that would have buried the 2015 analyst by lunch.

The second new competency is adversarial simulation, and it is the direct counter to the offensive volume problem. If autonomous offensive tooling can generate attack activity at machine rate against external targets, the defender’s answer is to generate attack activity at machine rate against their own environment, on purpose, under control, to find which techniques their telemetry catches and which it does not. The tooling for this is mature and largely open. Atomic Red Team provides a library of small, discrete tests mapped directly to MITRE ATT&CK technique IDs - run the T1003.001 test, confirm the Sysmon Event ID 10 with the lsass handle fires in the SIEM, confirm the correlation rule triggers, confirm the alert reaches a human. MITRE Caldera automates adversary emulation across a full chain. Breach-and-attack-simulation platforms run this continuously. The purple-team model - offensive simulation feeding directly into detection engineering rather than into a report that sits in a drawer - is the operational loop that closes the gap. The simulation reveals the detection gap. The detection engineer writes the rule. The next simulation confirms the rule fires. The loop runs continuously because the offensive volume runs continuously. A program that simulates its own defences against the techniques it actually faces does not deskill its operators. It is the most demanding thing a defensive team can do, and it requires every operator on it to understand the mechanism of each technique well enough to both execute it and detect it.

There is a hard distinction inside the simulation work that the volume narrative obscures, and it determines whether a team’s results improve or just get noisier. Generating attack activity is now cheap. Validating that the activity was both realistic and correctly detected is not. A model can produce a thousand variant payloads in the time it takes to write this sentence. Confirming that each one exercises the technique it claims to exercise, that the telemetry fired, that the alert was a true positive and not a coincidental match - that is the expensive, irreducible human-supervised step. It is the same asymmetry that made the curl bug-bounty inbox a problem, observed from the defensive side. Generation is cheap. Validation is dear. The team that automates generation and also automates validation without keeping a competent human on the validation output has not solved the volume problem. It has moved the volume problem one layer down and hidden it, which is worse, because now the unvalidated output wears the authority of a passed test.

The regulatory frame in Australia makes this concrete rather than abstract, because the obligations assume the defender can actually see what happened. Under the Security of Critical Infrastructure Act, responsible entities for critical infrastructure assets carry risk-management program obligations and mandatory cyber-incident reporting to the Australian Signals Directorate - within twelve hours for a critical incident having a significant impact, within seventy-two hours for one having a relevant impact. Those timelines presume detection and characterisation capability that operates at incident speed. A detection program that has quietly outsourced its triage to an unsupervised model, and that auto-closed the alert chain leading to the incident, cannot meet a twelve-hour reporting obligation it never knew it had triggered. Under the Privacy Act and the Notifiable Data Breaches scheme, an entity must assess a suspected eligible data breach and notify the Office of the Australian Information Commissioner and affected individuals where serious harm is likely. Assessment requires knowing what was accessed, which requires telemetry that was retained, parsed, and actually examined. The volume shift is not only an operational problem. It is a compliance exposure, because the obligations were written assuming a defender who can see, and the failure mode of unsupervised automation is a defender who cannot.

Return to the original claim and resolve it against the mechanism. Is AI ruining defender skills. The honest answer is that AI is ruining one specific practice - the habit of treating automated output as finished work rather than as raw input requiring validation - and it is doing so precisely in the teams that adopted the automation without adopting the supervision discipline that has to ride alongside it. That is a real decline and it is measurable in suppressed true positives, in accepted hallucinated findings, in slopsquatted dependencies that shipped, in CVE backlogs treated as authoritative when they were never enriched. But it is not the decline the narrative describes. The narrative says the machine took the skill. The mechanism says the machine took the friction, and the friction was holding up a workload the human was never going to clear by hand.

The residual exposure after every AI control is deployed is the same one that has always governed this work. Output exceeding inspection. The model generates code faster than it is reviewed. The scanner finds candidates faster than they are validated. The triage layer closes alerts faster than they are sampled. The simulation produces attack activity faster than it is confirmed. Every one of those is the same failure observed in a different layer - generation outrunning validation - and the fix is not less automation and it is not more. The fix is keeping a human with mechanism-level understanding on the validation side of every automated generator, and building the detection engineering and adversarial simulation capacity to do that validation at the rate the generators now run. That is harder than the work it replaces. It requires more understanding of memory safety, of process behaviour, of protocol mechanics, of detection logic, than the manual one-bug-at-a-time practice it supersedes.

The early results are in. They are not that AI replaced the defender. They are that the defenders who kept practising the old skill against the new volume are losing, and the defenders who moved their skill to the validation and detection-engineering layer are holding the line. The discriminator is not whether a team adopted AI. The discriminator is whether the team kept a human who understands the mechanism on the output. Volume is the new primitive. The skill that counts is the one that validates what the volume produced - and that skill was never the thing the machine could take.

---

Contains a referral link.

See also: NordVPN for tunneled traffic when operating outside controlled networks.

---

#ad Contains an affiliate link.