Skip to content

Review the analysis with a fresh second agent

Optional detour  about 6 min

In this exercise, you commission an independent code and methodological review by launching a second agent session with a clean context window. Unburdened by the assumptions and conversational momentum of the session that generated the analysis, a fresh agent evaluates artifacts with greater objectivity, catching edge cases and undocumented assumptions.

Before you start: A completed analytical milestone persisted as local files: an analysis script, input data, and a results summary (Setup).

Background

The agent session that authored an analysis is structurally ill-suited to audit its own code: its context window retains earlier conversational turns and reasoning steps, creating confirmation bias that favors its own assumptions.

Launching a new session eliminates this history entirely. The secondary agent evaluates only the persisted files on disk, acting as an independent peer reviewer encountering the codebase for the first time (Context).

Frame the review request as an adversarial error audit rather than a confirmation check. Inquiring "Does this look correct?" induces sycophantic agreement. Furthermore, recognize model limitations: because models from the same architecture family can share similar reasoning biases, a clean secondary review lowers risk but does not replace human domain oversight (Verification).

General methodology

Initialize a clean agent session in the project directory, directing it to audit persisted files without granting conversational background:

You are an independent peer reviewer auditing an empirical analysis. Inspect <protocol file>, <scripts/analyze.py>, and <out/results.md>. Audit the script line by line against the protocol specification. Identify computational errors, unstated assumptions, and claims in the narrative that are not supported by empirical estimates. Do not modify any files. Present your findings as a prioritized list ordered by potential impact on study conclusions, providing line-specific evidence for each issue.

Review the resulting critique carefully. Secondary reviewers can also propose false positives; evaluate whether each flagged issue represents a substantive methodological defect before taking action.

Workshop example · Imaging: chest radiographs

Start from: the evaluated checkpoint (open a new session in it) (how to restore)
Produces: nothing saved (ask for the review as out/review.md if you want to keep it)

Prompt

You are reviewing someone else's evaluation. Read eval_protocol.docx, scripts/build_cohort.py, scripts/evaluate.py and out/evaluation.md. Check the inclusion rules, the choice of index image, the bootstrap, the model comparison and the thresholds in the code against the protocol, line by line. List errors, questionable assumptions, and anything in the results not supported by the data. Do not change any files. Rank the issues and give the evidence for each.

Files used: eval_protocol.docx, scripts/build_cohort.py, scripts/evaluate.py, out/evaluation.md.

What a good review finds. For this evaluation the reference solution follows the protocol, so a careful reviewer should mostly confirm the rules and then raise limitations, such as:

  • Three subgroup comparisons were made and none corrects for multiple looks; PA vs AP rests on 19 positive AP patients.
  • The published operating points were set by the model developers on other data, and two of the models may have seen these images in training, so the numbers are likely optimistic.
  • The sample has about 29% pneumothorax, far above clinical prevalence, so sensitivity and specificity transfer but predictive values do not.
  • The reference labels came with the public sample; how they were made is not documented here.
  • Burned-in "AP" and "PORTABLE" markers and chest tubes are visible in the images and may help a model without telling it anything about the lung.
  • Follow-up studies were dropped, which is correct for the protocol but means each patient is represented by one image.

A better test: plant an error. In a copy of the folder, change one thing and ask a fresh session to review it. Two good candidates: in scripts/build_cohort.py take the latest study instead of the earliest as the index image (the cohort stays at 203 but some rows change), or change age >= 18 to age > 18. A good reviewer should find the mismatch with the protocol. It should not simply trust the summary table, because the final counts barely move.

What a review cannot do. It cannot tell you whether the pilot bar is the right bar for your department, or whether the sample resembles your patients. Those judgements are yours.

Verification criteria

  • Evidence-backed critiques: Every critique cites a specific line of code, formula, or table value.
  • Human evaluation: Flagged concerns are reviewed and confirmed by the researcher before altering analytical code.
  • Read-only execution: The secondary review session produced an evaluation report without modifying repository files.

Common pitfalls

Symptom Underlying cause Resolution
Superficial review ("Analysis looks sound") Prompt requested confirmation rather than an adversarial audit Direct the agent: "Identify at least three potential sources of bias, unhandled edge cases, or model limitations"
Excessive trivial formatting critiques Prompt failed to require prioritization Instruct the agent: "Rank the top three issues most likely to invalidate study conclusions"
Agent rewrites the analysis autonomously Prompt lacked explicit write prohibitions Re-issue command: "Provide review notes only; do not modify any files"

Concepts behind this

  • Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
  • Context. Context represents the model's active working memory. Because every turn re-transmits the complete conversation, long sessions increase latency, cost, and attention drift. Go deeper →
  • Use AI recursively. Apply the agent recursively to improve your collaboration workflows: drafting plans, refining instructions, reviewing code, and packaging tools. Go deeper →

Builds on: Evaluate an AI tool, with intervals