Picture a midsize fintech on a Tuesday morning in March. The SOC dashboard glows green across the board. Their XDR is humming. The ML-based code review tool flagged six minor issues earlier in the week, all patched by Friday. And yet they are losing data. They have been losing data for six weeks. The exfiltration looks, to the model that watches their network, like routine analytics traffic. It looks routine because the attackers studied what routine looked like to that specific model, and shaped their work to match.

This is not a hypothetical. It is the shape of a recurring incident pattern across organizations that leaned hard into AI-driven defense over the last eighteen months. The dashboards stay green. The models do exactly what they learned to do. The business bleeds anyway.

The shift no one warned us about

For most of the last two decades, defenders ran rule-based systems. A YARA signature for a known malware family. A regex for SQL injection. A firewall ACL listing forbidden destinations. These systems had obvious limits. They could only catch what someone had thought to write a rule for. But they had a strange virtue. They did not leak information about themselves the way machine learning models do.

Query a rule-based system, and you learn whether you matched a rule. That is all. You cannot reverse-engineer the rule from one match. The logic is discrete and opaque to the prober.

Deep learning detectors are different. They are continuous functions over a high-dimensional input space, and anyone willing to send enough probes can map their decision boundaries. A classifier that returns a confidence score is generous to the attacker. One that returns only a binary decision is enough too, given query budget. The structure of what gets flagged and what does not becomes a map. And maps get followed.

How the probe actually works

The literature on this is no longer new. Florian Tramèr and colleagues showed in 2016 that you could extract a functionally equivalent copy of a production ML classifier with a few thousand queries. Nicholas Carlini's work over the following decade demonstrated that adversarial examples crafted to evade a specific classifier often transfer to other classifiers trained on similar data. By 2023, defenders watching the offensive security space were already documenting attackers who kept sandboxed copies of commercially available EDR products and pre-tested payloads against those copies before deployment.

The 2026 escalation is that this practice is no longer the province of nation-states. Off-the-shelf model extraction tooling exists. Public benchmarks of commercial defense models exist. Threat actors with moderate resources can now characterise a defender's ML stack with enough fidelity to engineer evasions that look like nothing the model was trained to find.

The blind spot is not a bug. It is a region of the input space where the model's training data did not adequately cover the manifold of real inputs. There will always be such regions. The model cannot tell you where they are. The attacker, given enough queries and enough patience, can.

Three patterns showing up in 2026

Three patterns dominate the incidents we are seeing this year.

• Pre-deployment evasion pipelines (mature). Commodity malware operators ship their builds through a pipeline that holds copies of EDR products, recent SOC analytics platforms, and the major ML-based code scanners. They iterate anything that does not survive contact with that pipeline until it does. By design, defenders only ever detonate the samples their tools could not see.

• Repository poisoning of AI code review (recent). An attacker submits a benign-looking pull request whose subtle vulnerability lands in a region of the embedding space the reviewer model has historically scored as low-risk. The model approves, the human reviewer trusts the model’s confidence and approves too, and the vulnerability ships. This year’s disclosures around the mcp-server-git findings showed the pattern, where attacker-controlled README content drove agent behaviour into territory the model treated as benign.

• Blind-spot corridors (subtler). A scanner that consistently misses a class of issue becomes a known-safe corridor. Internal teams start staging artifacts there because that is where the scanner reports clean, and the blind spot hardens into an unintended architectural pattern. Adversaries who learn the same blind spot get a quiet place to operate, inside the part of the environment defenders have, paradoxically, learned to trust most.

What these patterns share is that the model is functioning correctly. It is doing what it was trained to do. The failure is not at the model. The failure is in treating the model's output as authoritative when its limits have become public information.

A framework for engineering leaders

Three practices are emerging from teams that have lived through these incidents. Each one assumes your model will be mapped, and plans for that day rather than against it.

• Rotate your blind spots. Treat detection models the way you treat cryptographic keys. Assume an attacker will eventually characterise the current model, and run ensembles whose members differ in data slices, architecture, and retraining schedule, so whoever maps one model meets an unfamiliar next one. Schedule full retraining and replacement, not just incremental fine-tuning. Make the map go stale faster than the attacker can refresh it.

• Red-team the defender, not just the app. Stand up an internal capability that probes your ML defences the way external attackers will. Measure how few queries it takes your own team, working without internal access, to characterise the detection surface. If that number is small, your detection surface is, for practical purposes, public information. Build the muscle to know the number, and drive it up over time.

• Build layers that fail differently. Keep rule-based controls as a floor. They are obvious and attackers will evade them, but they also catch what the ML stack confidently misclassifies, because they are not playing the same game. The teams that weathered 2026 best did not own the most sophisticated single model. They owned detection layers that fail differently, so an attacker who breaks one still has to solve a different kind of problem to break the next.

Closing

The argument is not against AI-driven defense. These systems are too good at the things they are good at to abandon. The argument is that defense AI is not a finished product the way a firewall is a finished product. It is a continuously probed system whose security properties decay every day an attacker has access to its outputs. Treating it otherwise, with green dashboards and periodic vendor updates and business as usual, is how organisations have found themselves bleeding in front of monitors that show nothing wrong.

The dashboards do not show what the model cannot see. The model does not see what the attacker has learned to hide. The gap between those two facts is where 2026's quietest incidents are happening.