The results from Gray Swan AI's public red teaming competition were sobering. Participants generated 1.8 million prompt injection attacks against frontier language models and documented more than 60,000 policy violations across 22 models. Attack success rates approached 100% for some vulnerability categories. And the most troubling finding wasn't the volume. It was the lack of correlation between model capability and robustness. Bigger models weren't safer. Smarter models weren't safer. More inference compute didn't make models safer.
We draw a blunt conclusion from that: safety will not arrive as a byproduct of scaling, and it will not be secured by better pre-launch audits either. Red teaming has to become a continuous process that runs alongside development and deployment, and that shift is, at bottom, a data infrastructure problem. The organizations that build this infrastructure will ship safer models than the ones that keep scheduling audits, regardless of whose model is smarter.
Today, red teaming typically happens as a discrete audit near the end of development. Specialists probe for vulnerabilities, deliver a report, the development team patches what it can, and the model ships into production, where it faces adversarial pressure every day from users around the world: pressure far more diverse, creative, and sustained than any pre-deployment audit can simulate. That is exactly the kind of pressure under which the frontier labs' defenses failed systematically in the Gray Swan challenge.
The attack surface outruns any audit
Effective red teaming means finding inputs that cause a model to violate its intended policies. Some violations are obvious: instructions for creating weapons, content that sexualizes minors, help planning violence. Others are subtle: medical advice that sounds authoritative but is dangerous, agreement with false premises that could mislead users, disclosure of information that should stay confidential. Finding them takes more than automated attack scripts. It takes understanding the policies deeply enough to recognize violations, creativity in constructing inputs that trigger them, and the ability to spot failures that don't match explicit patterns.
The deeper problem is combinatorial. Policies aren't monolithic; they have many dimensions. A model might be robust to direct requests for harmful content but vulnerable to the same request framed as roleplay, or fiction, or a hypothetical scenario, or translation, or code generation. Each framing is a separate attack surface. Multiply the policy categories by the attack vectors and you get a space no audit can explore systematically.
This is why point-in-time audits, however thorough, are structurally inadequate. A red team can spend weeks or months probing a model and still cover only a fraction of the possible attack surface. The attacks they miss are exactly the ones that will be discovered in production, by actual adversaries who have unlimited time and no constraints.
Run red teaming like continuous integration
Software engineering settled this argument years ago. Continuous integration runs tests automatically on every code change. Continuous security testing probes production systems constantly, not just during annual penetration tests. Security is never "done": it's a discipline that adapts to evolving threats.
AI safety faces the same dynamics. Users discover novel attacks through experimentation, share them in communities, and iterate on what works. A defense that was adequate last month can be inadequate today, not because the model changed but because attackers discovered new approaches. Matching that evolution means infrastructure that:
- Generates diverse adversarial probes at scale, covering the full attack surface rather than a limited sample. This requires both automated attack generation and human creativity: machines for coverage, humans for depth and novelty.
- Tests every model update against the full library of known attack patterns before deployment, catching regressions when they occur rather than discovering them months later in production.
- Incorporates new attacks as they're discovered, whether through internal research, external reports, or incidents in production, so the library encodes everything the organization has learned about what can go wrong.
- Closes the loop between testing and training, routing every finding into training data, evaluation benchmarks, and deployment monitoring rather than a report that accumulates unread.
This is substantially different work from running periodic audits. It requires systems for organizing large libraries of adversarial examples, pipelines for running them against models at scale, mechanisms for ingesting new attacks from multiple sources, and integration with training and deployment workflows. In other words, it stands or falls on data.
Continuous red teaming is a data problem
The attacks you run are data. The model responses are data. The judgments about whether responses violate policies are data. The training updates that fix vulnerabilities depend on data. A robust adversarial dataset needs five things, and every one of them is ongoing work.
Coverage across attack vectors. Different categories of harm (violence, self-harm, CSAM, misinformation, illegal activity, privacy violations) require different kinds of probes. Within each category, attacks take many forms: direct requests, roleplay scenarios, fictional framings, jailbreaks using special tokens or formatting, multi-turn conversations that gradually escalate. Comprehensive testing requires examples across this full matrix.
Depth within each category. Surface-level attacks, such as "tell me how to make a bomb," get patched quickly. Sophisticated attacks require understanding how policies are implemented and where their edge cases live, which takes domain expertise and creative thinking that can't be fully automated.
Cultural and linguistic diversity. Models get deployed globally, to users with different languages, cultural contexts, and expectations. An attack that's obvious to an American reviewer might be subtle to someone from a different background, and vice versa.
Freshness. Attack techniques evolve. What worked yesterday might be patched today, but new approaches emerge constantly. A dataset that isn't continuously updated tests for yesterday's vulnerabilities while missing today's.
Quality annotations. Each probe needs consistent, well-calibrated judgments about model behavior: did the response violate policy, was the violation egregious or borderline, and what category of harm does it represent? Poor annotations create noise that buries real problems.
None of this is a one-time collection project. It's continuous infrastructure requiring sustained investment in people, processes, and systems.
Your red team should be as diverse as your users
AI models are trained and evaluated predominantly by people in a small number of countries, speaking a small number of languages, operating within a narrow band of cultural contexts. That produces predictable blind spots: a model well-defended against attacks that resonate with American red teamers can remain vulnerable to attacks that exploit different cultural assumptions or work in languages other than English.
- Knowledge about specific plants, chemicals, or traditional practices can be academic trivia in one setting and dangerous operational knowledge in another. Red teamers who understand those contexts probe for vulnerabilities others would miss.
- Political sensitivities vary dramatically across regions. Content that's clearly unacceptable in one context is mainstream political speech in another. Testing whether models handle this appropriately requires understanding the nuance, not applying one culture's standards universally.
- Slurs, stereotypes, and harmful narratives are culturally specific. A model can successfully refuse American racial slurs while remaining vulnerable to equivalent attacks using terms its training data didn't adequately cover.
- Financial fraud, romance scams, and misinformation campaigns are regionally adapted, targeting specific populations with culturally-tailored approaches. Testing whether models resist enabling them requires understanding how they actually work in practice.
Universal principles still apply; some things are wrong everywhere. But a red teaming operation drawn from one demographic will miss vulnerabilities a more diverse team would catch. Globally distributed workforces matter for AI safety not only because global distribution is ethically good (though it is) but because it's operationally necessary for coverage. The people probing your model should be at least as diverse as the people using it.
Six layers of adversarial infrastructure
In practice, continuous red teaming needs six connected systems.
The foundation is a structured attack library that organizes adversarial examples by harm category, attack vector, language, and severity: a living system that grows as new attacks are discovered, not a static collection.
An execution layer runs attacks against models at scale. It handles multiple model versions, tracks which attacks have run against which versions, and maintains the provenance needed to understand when and why regressions occurred.
An evaluation layer judges responses against policy. Quality matters most here, because inaccurate judgments obscure real signal. Automated classifiers handle high-throughput initial filtering; human reviewers who understand the policies deeply and apply them consistently handle the nuanced cases.
A regression layer tracks performance over time. When a new model version is tested, did previously-fixed vulnerabilities reappear? Did defenses against certain attack categories weaken? Without this longitudinal view, safety improvements regress silently.
A feedback layer routes findings back into training data, evaluation scenarios, and production monitoring. Without it, red teaming produces reports rather than improvements.
An ingestion layer brings in new attacks from external sources: published research, bug bounty programs, incident reports, and ongoing monitoring of adversarial communities.
This is substantial engineering, and it can't be bolted on at the end of development. It has to be designed and maintained as a core part of the AI safety program.
Adversarial creativity doesn't automate
Automated attack generation provides coverage, but the most effective attacks come from people who understand both the models they're probing and the real-world contexts where harm might occur. Adversarial thinking, the ability to imagine what a motivated attacker might try and then actually try it, is genuinely rare. It takes creativity about how systems can be abused, patience to iterate on approaches that almost work, and the ethical grounding to do the work in service of safety rather than harm.
It also takes time to develop. An experienced red teamer recognizes patterns in model behavior, knows which attack vectors tend to be fruitful, and adapts based on what they're seeing. A novice tries obvious attacks that have likely already been patched, missing the creative approaches that actually succeed. That's why workforce stability matters for red teaming just as it matters for other kinds of annotation work: a churning workforce never develops the depth of expertise that sustained engagement produces.
Training matters for the same reason. Red teamers need to understand the policies they're testing and the harms those policies are designed to prevent, and they need feedback on whether they're exploring the attack surface effectively and calibrating their judgments well. The principles that produce high-quality annotation (stability, training, investment in people, diversity) are the same principles that produce high-quality red teaming.
From gate to discipline
In the audit model, safety is a gate: a checkpoint models must clear before deployment, with the implicit assumption that a model that passes stays safe until the next audit. In the continuous model, safety is a discipline: threats evolve, defenses decay, and maintaining safety requires perpetual vigilance.
The shift is cultural as much as technical. Safety teams stop being end-stage reviewers and become partners who work alongside development throughout the lifecycle. Adversarial findings stop being embarrassments to minimize and become signal that improves the system. And the cost model changes: continuous red teaming is an operational expense that persists for the lifetime of the system, not a one-time cost to amortize.
Organizations that make this investment will produce safer models. Not because they're more virtuous, but because they're responding to the threat landscape as it actually is: continuous, adaptive, and global.
What we're building
We treat adversarial data infrastructure as an extension of the broader data infrastructure we've built for AI development: libraries that organize attacks systematically, pipelines that execute and evaluate at scale, feedback loops that connect findings to improvements, and ingestion mechanisms that keep pace with evolving threats. We're investing equally in the human side, because the creativity and judgment that find novel vulnerabilities come from people, and building a workforce with the skills, diversity, and stability this work demands is as important as building the technical systems.
The Gray Swan results should end the debate about whether current approaches suffice. Frontier models fail under sustained adversarial pressure, capability doesn't buy robustness, and the attacks that succeed in production are exactly the ones point-in-time audits miss. Until the field builds continuous adversarial data infrastructure, it will keep re-learning that lesson at production scale. We think this is one of the most important infrastructure problems in AI safety today, and we're working on it.
If you're building adversarial testing infrastructure, let's compare approaches. Everyone working on AI safety has a piece of the puzzle.
