Build a study cohort with a flow table
Empirical research requires defining an unambiguous study population. In this exercise, you direct the agent to translate written clinical eligibility criteria into a reproducible cohort dataset (out/cohort.csv) alongside a formal cohort attrition diagram (out/cohort_flow.md) tracking sample sizes across each selection rule.
Before you start: Cleaned data and a written study protocol or cohort specification (Setup).
Background
In observational study design, an unambiguous cohort definition specifies four distinct parameters:
- Target population: Broad demographic or clinical eligibility criteria (e.g., adults receiving care at designated clinic sites).
- Index event: The precise anchor timestamp that initiates follow-up for each individual (e.g., the first qualifying encounter within the study window).
- Exposure classification: The operational definition distinguishing exposed individuals from controls.
- Outcome observation window: The defined temporal interval following the index event during which endpoints are ascertained.
Ambiguity in eligibility criteria is a primary source of silent bias. When rules reside in study protocols or memos, require the agent to read the document and quote the exact eligibility sentences before writing any code (Inputs).
An attrition flow table (or PRISMA/STROBE cohort flow diagram) documents sample counts retained and excluded at each stage. Any discrepancy between step counts indicates unintended record drops (Verification).
General methodology
Instruct the agent to audit protocol definitions, quote the source rules, and generate an executable cohort creation script:
Read <protocol file>. Quote the exact eligibility criteria, index event definition, exposure criteria, and outcome observation window. Then write an executable script (<scripts/build_cohort.py>) applying these rules to <clean data>. Generate a cohort dataset (<out/cohort.csv>) with strictly one row per individual, and an attrition flow table (<out/cohort_flow.md>) reporting counts remaining after each selection rule. Classify an individual as exposed if <exposure condition>; categorize all remaining eligible individuals as unexposed controls so no eligible subjects are dropped. Explicitly document any areas where protocol phrasing required interpretation.
Requiring the agent to highlight interpretive assumptions ensures domain choices remain visible for researcher confirmation.
Workshop example · Clinical: clinic network
Start from: the cleaned checkpoint (how to restore)
Produces: scripts/build_cohort.py, out/cohort.csv, out/cohort_flow.md
Prompt
Read pilot_protocol.docx (or text/pilot_protocol.md if you converted it earlier). Quote the exact eligibility rules, index visit, exposure and outcome window. Then write scripts/build_cohort.py that applies them to data/clean/visits_clean.csv and pilot_calls.csv and writes out/cohort.csv (one row per patient, with columns for age, sex, index HbA1c, whether the patient was reached by a call, and whether HbA1c was rechecked) and out/cohort_flow.md (patients remaining after each step). Mark a patient as reached (1) only if pilot_calls.csv has reached = Y for them; every other patient in the cohort, including those with reached = N and those with no call record, is 0, so nobody is left blank. Tell me every place you had to interpret the protocol. Report the count with each step.
Files used: pilot_protocol.docx, data/clean/visits_clean.csv, pilot_calls.csv.
Rules to be quoted from the protocol. Adults (18 or older) at Hillcrest. Index visit: the first visit from 2025-04-01 to 2025-08-31 with
a measured HbA1c of 7.0% or higher. Exposure: reached by a pilot call (reached = Y). Outcome: any measured HbA1c from 14 to 120 days after
the index visit.
Reference flow table
| Step | Patients |
|---|---|
| Hillcrest patients with any HbA1c | 820 |
| Adults (age >= 18) | 819 |
| Index visit: first HbA1c >= 7.0% from 2025-04-01 to 2025-08-31 | 251 |
Reference cohort. 251 patients; 92 reached by a call; 92 rechecked.
Interpretation points to look for. "Reached" means reached = Y, not merely a call attempt: 11 further calls were not completed, and
those patients belong in the comparison group. Window boundaries (14 and 120 days) are inclusive. A patient with no measured HbA1c
cannot have an index visit. If the agent counts attempted calls as exposed, the exposed group will be 103, not 92, which is a good example
of why the definition has to be checked. Patients with no call record (148 of the 251) are not reached too. If they are left blank and dropped,
the comparison group shrinks from 159 to 11 and the odds ratio changes completely.
A reference version is in scenarios/toy/checkpoints/cohort/out/.
Verification criteria
- Protocol alignment: The quoted eligibility rules match the protocol source text verbatim.
- Unit of observation integrity: The final cohort dataset contains strictly one record per patient ID.
- Monotonic sample reduction: Each row in the attrition flow table is less than or equal to the prior count, and the final flow count matches
out/cohort.csvexactly. - Transparent assumptions: The agent explicitly documents edge-case assumptions (e.g., handling ties in same-day visits) for your review.
Common pitfalls
| Symptom | Underlying cause | Resolution |
|---|---|---|
| Multiple rows per patient ID | Index event logic failed to restrict records to the initial qualifying encounter | Instruct the agent to group records by patient ID and slice the earliest qualifying event |
| Unexposed comparison group is unexpectedly small | Records lacking explicit exposure logs were coded as null and dropped | Explicitly define the control group as all eligible cohort members who do not meet exposure criteria |
| Sample size diverges from historical benchmarks | Discordant temporal window boundary interpretations (e.g., < 30 vs. <= 30 days) |
Reconcile window boundary expressions against protocol definitions |
Concepts behind this
- Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
- Tools. Turning interactive prompts into parameterized scripts creates deterministic, testable tools that run identically on every execution. Go deeper →
- Inputs. Language models process information as text. Selecting appropriate ingestion pathways and source formats preserves data structure and prevents extraction errors. Go deeper →