Review the analysis with a fresh second agent
In this exercise, you commission an independent code and methodological review by launching a second agent session with a clean context window. Unburdened by the assumptions and conversational momentum of the session that generated the analysis, a fresh agent evaluates artifacts with greater objectivity, catching edge cases and undocumented assumptions.
Before you start: A completed analytical milestone persisted as local files: an analysis script, input data, and a results summary (Setup).
Background
The agent session that authored an analysis is structurally ill-suited to audit its own code: its context window retains earlier conversational turns and reasoning steps, creating confirmation bias that favors its own assumptions.
Launching a new session eliminates this history entirely. The secondary agent evaluates only the persisted files on disk, acting as an independent peer reviewer encountering the codebase for the first time (Context).
Frame the review request as an adversarial error audit rather than a confirmation check. Inquiring "Does this look correct?" induces sycophantic agreement. Furthermore, recognize model limitations: because models from the same architecture family can share similar reasoning biases, a clean secondary review lowers risk but does not replace human domain oversight (Verification).
General methodology
Initialize a clean agent session in the project directory, directing it to audit persisted files without granting conversational background:
You are an independent peer reviewer auditing an empirical analysis. Inspect <protocol file>, <scripts/analyze.py>, and <out/results.md>. Audit the script line by line against the protocol specification. Identify computational errors, unstated assumptions, and claims in the narrative that are not supported by empirical estimates. Do not modify any files. Present your findings as a prioritized list ordered by potential impact on study conclusions, providing line-specific evidence for each issue.
Review the resulting critique carefully. Secondary reviewers can also propose false positives; evaluate whether each flagged issue represents a substantive methodological defect before taking action.
Workshop example · Clinical: clinic network
Start from: the analyzed checkpoint (open a new session in it) (how to restore)
Produces: nothing saved (ask for the review as out/review.md if you want to keep it)
Prompt
You are reviewing someone else's analysis. Read pilot_protocol.docx, scripts/analyze_pilot.py and out/pilot_results.md. Check the cohort rules and the outcome definition in the script against the protocol, line by line. List errors, questionable assumptions, and anything in the results not supported by the data. Do not change any files. Rank the issues and give the evidence for each.
Files used: pilot_protocol.docx, scripts/analyze_pilot.py, out/pilot_results.md.
What a good review finds. For this analysis the reference solution is correct, so a careful reviewer should mostly confirm the definitions and then raise limitations, such as:
- Call assignment was not random, so residual confounding is possible beyond age.
- The outcome counts any measured HbA1c in the window, regardless of which visit type produced it.
- Patients with no follow-up at all are counted as "not rechecked", which is the intended outcome but depends on the clinic being the only place they would be tested.
- The interval is wide (the lower bound is near 1), so the estimate is imprecise.
A better test: plant an error. To see the reviewer work, change one thing in a copy of the script, such as counting a call attempt as exposure (the exposed group becomes 103 instead of 92) or changing the window to 14 to 90 days, then ask a fresh session to review it. A good reviewer should find the mismatch with the protocol. That also shows why the exposure and window definitions were quoted in the cohort step.
What a review cannot do. It cannot tell you whether "reached by a call" is the right exposure for your question. That judgement is yours.
Verification criteria
- Evidence-backed critiques: Every critique cites a specific line of code, formula, or table value.
- Human evaluation: Flagged concerns are reviewed and confirmed by the researcher before altering analytical code.
- Read-only execution: The secondary review session produced an evaluation report without modifying repository files.
Common pitfalls
| Symptom | Underlying cause | Resolution |
|---|---|---|
| Superficial review ("Analysis looks sound") | Prompt requested confirmation rather than an adversarial audit | Direct the agent: "Identify at least three potential sources of bias, unhandled edge cases, or model limitations" |
| Excessive trivial formatting critiques | Prompt failed to require prioritization | Instruct the agent: "Rank the top three issues most likely to invalidate study conclusions" |
| Agent rewrites the analysis autonomously | Prompt lacked explicit write prohibitions | Re-issue command: "Provide review notes only; do not modify any files" |
Concepts behind this
- Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
- Context. Context represents the model's active working memory. Because every turn re-transmits the complete conversation, long sessions increase latency, cost, and attention drift. Go deeper →
- Use AI recursively. Apply the agent recursively to improve your collaboration workflows: drafting plans, refining instructions, reviewing code, and packaging tools. Go deeper →
Builds on: Analyze an exposure and outcome, crude and adjusted