Every successful scientific project eventually transitions into a battle-testing phase in which the project team is actively trying to show that their results don’t hold. Did we overlook a source of counterexamples? Have biases in our data conspired to deliver these results for spurious reasons? Are there fatal bugs in our code? The best scientists ask these questions relentlessly, and they celebrate the “yes” answers, because each one is a potential disaster averted and a step closer to the truth.

What does this battle-testing phase look like when AI agents are writing all the experiment code and conducting the majority of the analyses (Lu et al. 2024; Si et al. 2024)? What we see in the public discourse around AI and scientific discovery leaves us concerned. However, we have discovered that a practice now standard in agentic coding – adversarial review by a fresh-context agent – can be adapted to scientific work. When it is, the battle-testing can become even more rigorous than in our all-human collaborations, especially if the review process is adapted over time to your specific scientific context.

Our focus here: straight-up errors in experimental design

Our focus is not on inadvertent p-hacking or the garden of forking paths (Gelman and Loken 2014). When people send their agents on large-scale expeditions in search of exciting discoveries, they are certainly increasing the risks of false positives deriving from multiple comparisons and selective reporting of results (see Miao et al. 2026), even if the individual experiments are executed impeccably. This is a pressing problem. However, our concern in this post is lower-level: flawed experimental designs leading to meaningless results (Luo et al. 2025).

We suspect everyone is aware of the risks here, but when the agents announce that their findings are going to change the world, it can be hard to remain skeptical. Their language can be absurdly confident (Zhou et al. 2024), and we’ve been promised that they are superintelligent (or will be in 2–3 years). Also, they do so many experiments!

A false sense of security from the most verifiable domains

Some areas of science seem better protected from outright errors than others. For example, when current AI products are asked to prove theorems, they seem to automatically go into a mode of repeatedly checking their own work using symbolic theorem provers. This really came home to us when Levent Alpöge posted a counterexample to the Jacobian conjecture that Claude found for him. We have not seen Alpöge’s session with Claude, but others posted their versions with hints based on Alpöge’s result, and you could see agents checking and rechecking their work, apparently with no prompting to do so, and Alpöge has said publicly that he uses a “generic prompt”.

Such high-profile examples could create the impression that AIs will be thorough in this way for all tasks. However, in our experience, this is dramatically not the case (and even for math, of course, checking is not a panacea). The scientific work we do with agents is generally on topics that blend AI, machine learning, and data science. The experiments tend to be highly verifiable in the general sense that, once the questions are set, they can be answered objectively, and mistakes are going to unavoidably and non-negotiably count as mistakes. However, in these contexts, current AI products do not check their work or their conclusions without significant prompting to do so (Huang et al. 2023).

The Adversarial Protocol Review

Our initial response was a traditional one: when the agents make a claim, go into the battle-testing mode we are familiar with from our all-human collaborations. Our core directive: find what could manufacture this result. This is certainly productive: agents reliably respond by doing work that reliably leads them to better evidence. However, one sees quickly that it isn’t scalable, because agents not only produce vastly more claims than humans, but they also retract their claims at a much higher rate (and often with shocking rationales). This led us to The Adversarial Protocol Review.

The Adversarial Protocol Review works roughly as follows: before any result is adopted – and, just as crucially, before any expensive run is launched – an independent thread is started with the built artifacts (code, reports, etc.) and a hostile brief: relentlessly seek to answer the question “what could manufacture this result?” by enumerating, for every number the study will report, (1) the design flaws that could produce it with no effect in the world and (2) the design flaws that could produce a null effect with a real one present. Any problems that this process surfaces need to be fixed before the project proceeds, and the process itself applies recursively.

We should say that adversarial review has become quite standard in agentic coding. Anthropic includes advice along these lines in its best practices guide, and people seem to benefit from it greatly. Bringing these practices into your scientific work is critical in our view. However, this step alone will not suffice. Clean code doesn’t guarantee sound methodology; we’ve seen plenty of code reviews that end in victory only to produce scientific results we later discovered to be fundamentally flawed. Our prime directive summarizes the key difference: it’s not “find all the bugs”, but rather “what could manufacture this result?” When we distill the entire process into a skill.md file (see below), the core procedure is broken down into a scope gate (the conditions under which the skill should be used), a six-step procedure, an extensive manufacture catalog of high-level patterns to watch for, and discipline notes.

A brief case study

The Adversarial Protocol Review burns a lot of tokens. Is it worth it? We conducted a review of the documents relating to one of our ongoing projects. Across 18 review documents, we found 323 filed findings, of which 31 had flaws that would have destroyed our core claims. Here is a small sample of these scientific disasters averted:

1. A permutation test was applied to a column containing only one label, so all permutations reproduced the observed data exactly. What was initially reported to us as a substantive robustness check was in fact structurally guaranteed to find no effect.

2. A result about style transfer was confounded by the fact that a user’s own sessions have much greater topic overlap with each other than with other users’ sessions. Once topic overlap was controlled for, style was no longer a significant factor.

3. A piece of experiment code pruned noisy items from the treatment group but not the control group. When the pruning was applied evenly or not at all, the result disappeared.

4. A robustness test for a noisy LLM annotator was run on unvalidated outputs containing a lot of known mistakes. This led to a verdict of “solid” for the annotator. Rerunning the test on validated outputs dropped its score to “unacceptable”.

Would a team of human scientists have caught all these issues? The test results for 1 looked so suspiciously good that we would certainly have dug deeper. The confound in 2 seems very salient and so we might have raised that as well. For 3 and 4, we feel less sure. Both seem easy to overlook, especially since 4’s initial verdict was merely “solid”; an “outstanding” assessment would have seemed fishy but “solid” seems unremarkable enough that it could pass by unscrutinized. 

Overall, if we are honest with ourselves, we feel nervous about our capacity to reliably catch all these errors. The agents are earning their keep. The Adversarial Protocol Review is expensive, but the costs are small compared to the cost of running a pointless experiment, and infinitesimal compared to the cost of publishing a faulty claim.

Still a partnership between humans and AI

There is still a vital role for human scientists in this picture. The human user is critical for catching problems related to how the research questions are framed and what conclusions are licensed by the evidence (for related findings, see Ho et al. 2026). To put it roughly: The Adversarial Protocol Review catches what’s wrong with the implementation; the human catches what’s wrong with the interpretation. Both are absolutely critical to the scientific process.

A new skill.md file

We developed the Adversarial Protocol Review over many research sessions and a lot of proactive management of the agents we use for research. As we noted above, we have now distilled this protocol into an extensive skill.md file. The file is available in this Github repository. We also include the results of a validation experiment showing that the file is used where, and only where, we expect it to be used (at least by Claude Sonnet 5), and that it adds substantial value, especially when it comes to comprehensively reporting on problematic designs.

This version of the protocol removes some information that is highly specific to our own research context. Our advice would be to use it as a starting point and regularly update it based on moments of friction and discovery in your own sessions, so that your Adversarial Protocol Review becomes tightly fit to your goals and practices. We believe that this strategy can lead you (and your agents) to meaningful and lasting results.