Skip to content

Build a study cohort with a flow table

Full step  about 8 min

Empirical research requires defining an unambiguous study population. In this exercise, you direct the agent to translate written clinical eligibility criteria into a reproducible cohort dataset (out/cohort.csv) alongside a formal cohort attrition diagram (out/cohort_flow.md) tracking sample sizes across each selection rule.

Before you start: Cleaned data and a written study protocol or cohort specification (Setup).

Background

In observational study design, an unambiguous cohort definition specifies four distinct parameters:

  • Target population: Broad demographic or clinical eligibility criteria (e.g., adults receiving care at designated clinic sites).
  • Index event: The precise anchor timestamp that initiates follow-up for each individual (e.g., the first qualifying encounter within the study window).
  • Exposure classification: The operational definition distinguishing exposed individuals from controls.
  • Outcome observation window: The defined temporal interval following the index event during which endpoints are ascertained.

Ambiguity in eligibility criteria is a primary source of silent bias. When rules reside in study protocols or memos, require the agent to read the document and quote the exact eligibility sentences before writing any code (Inputs).

An attrition flow table (or PRISMA/STROBE cohort flow diagram) documents sample counts retained and excluded at each stage. Any discrepancy between step counts indicates unintended record drops (Verification).

General methodology

Instruct the agent to audit protocol definitions, quote the source rules, and generate an executable cohort creation script:

Read <protocol file>. Quote the exact eligibility criteria, index event definition, exposure criteria, and outcome observation window. Then write an executable script (<scripts/build_cohort.py>) applying these rules to <clean data>. Generate a cohort dataset (<out/cohort.csv>) with strictly one row per individual, and an attrition flow table (<out/cohort_flow.md>) reporting counts remaining after each selection rule. Classify an individual as exposed if <exposure condition>; categorize all remaining eligible individuals as unexposed controls so no eligible subjects are dropped. Explicitly document any areas where protocol phrasing required interpretation.

Requiring the agent to highlight interpretive assumptions ensures domain choices remain visible for researcher confirmation.

Workshop example · Imaging: chest radiographs

Start from: the cleaned checkpoint (how to restore)
Produces: scripts/build_cohort.py, out/cohort.csv, out/cohort_flow.md

Prompt

Read eval_protocol.docx (or text/eval_protocol.md if you converted it earlier). Quote the exact inclusion rules and the unit of analysis. Then write scripts/build_cohort.py that applies them to data/clean/ and writes out/cohort.csv (one row per patient: patient ID, index image, study date, label, view, sex, age and the four scores) and out/cohort_flow.md (what remains after each step). Tell me every place you had to interpret the protocol, and report the count with each step.

Files used: eval_protocol.docx, data/clean/ (images, labels_clean.csv, scores_clean.csv).

Rules to be quoted from the protocol. Adults (18 or older, age recorded and plausible); a frontal PA or AP radiograph at 1024 x 1024; a reference label and all four scores; the patient as the unit, using the earliest study as the index image.

Reference flow table

Step Images / patients
Cleaned images 258
With a reference label and all four scores 247
Age recorded and plausible 237
Adults (age >= 18) 220
One index image per patient (earliest study) 203

Reference cohort. 203 patients, 59 with pneumothorax; 120 PA and 83 AP.

Interpretation points to look for. A blank age cannot show that someone is an adult, so those images are excluded (10 here). The last step removes 17 images: later studies of patients who are already in the cohort. Patient identity depends on the normalized ID: if the agent skips that, a patient with a prefixed ID appears twice. Dropping images with missing scores is a missingness filter, not a performance filter, but a careful agent says so and reports how many images it removed (11).

A reference version is in scenarios/imaging/checkpoints/cohort/out/.

Verification criteria

  • Protocol alignment: The quoted eligibility rules match the protocol source text verbatim.
  • Unit of observation integrity: The final cohort dataset contains strictly one record per patient ID.
  • Monotonic sample reduction: Each row in the attrition flow table is less than or equal to the prior count, and the final flow count matches out/cohort.csv exactly.
  • Transparent assumptions: The agent explicitly documents edge-case assumptions (e.g., handling ties in same-day visits) for your review.

Common pitfalls

Symptom Underlying cause Resolution
Multiple rows per patient ID Index event logic failed to restrict records to the initial qualifying encounter Instruct the agent to group records by patient ID and slice the earliest qualifying event
Unexposed comparison group is unexpectedly small Records lacking explicit exposure logs were coded as null and dropped Explicitly define the control group as all eligible cohort members who do not meet exposure criteria
Sample size diverges from historical benchmarks Discordant temporal window boundary interpretations (e.g., < 30 vs. <= 30 days) Reconcile window boundary expressions against protocol definitions

Concepts behind this

  • Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
  • Tools. Turning interactive prompts into parameterized scripts creates deterministic, testable tools that run identically on every execution. Go deeper →
  • Inputs. Language models process information as text. Selecting appropriate ingestion pathways and source formats preserves data structure and prevents extraction errors. Go deeper →

Builds on: Clean the data with a script, and show the flow