Skip to content

Analyze an exposure and outcome, crude and adjusted

Full step  about 8 min

In this exercise, you direct the agent to conduct a statistical analysis contrasting unadjusted (crude) comparisons against multivariable regression models adjusted for confounding covariates. Observing how point estimates shift after controlling for baseline imbalances provides a foundational lesson in empirical analysis.

Before you start: A cohort file containing one record per individual, an exposure indicator, an outcome endpoint, and candidate covariates (Setup).

Background

A crude statistical comparison evaluates raw differences in outcome rates between exposed and unexposed groups. In observational research, crude comparisons are frequently biased by confounding: baseline imbalances across characteristics that influence both exposure likelihood and outcome risk. For example, if older patients are more likely to receive clinical outreach and independently experience higher re-encounter rates, age confounds the observed association.

A multivariable logistic regression models the relationship between exposure and outcome while holding candidate covariates (such as age, sex, and baseline severity) constant, estimating an adjusted odds ratio (aOR):

  • OR = 1.0: Indicates no association.
  • OR > 1.0: Indicates higher odds of the outcome among the exposed.
  • 95% confidence interval: Quantifies estimation uncertainty; intervals spanning 1.0 indicate associations consistent with chance variation.

In observational studies, estimated odds ratios reflect statistical association rather than definitive causal effect, and summary write-ups must adhere to cautious scientific phrasing.

Covariate selection represents a substantive methodological decision that must be determined a priori based on clinical reasoning rather than automated stepwise significance hunting.

General methodology

Direct the agent to compute both unadjusted and multivariable adjusted estimates, mandating explicit sample accounting across both models:

Using <cohort file>, analyze the association between <exposure> and <outcome>. Report raw sample sizes, event counts, and incidence rates for both exposed and unexposed cohorts. Calculate the unadjusted crude odds ratio, followed by an adjusted odds ratio from a multivariable logistic regression controlling for <covariates>, each with 95% confidence intervals. Save the analysis script to <scripts/analyze.py> and formatted results to <out/results.md>. Specify model formulas, regression diagnostics, and total sample counts utilized in each model (which should match <cohort size>). Provide a concise narrative interpreting why the adjusted point estimate shifts relative to the crude estimate.

Workshop example · Clinical: clinic network

Start from: the cohort checkpoint (how to restore)
Produces: scripts/analyze_pilot.py, out/pilot_results.md

Prompt

Using out/cohort.csv, compare HbA1c recheck between patients reached by a call and patients not reached. Report counts and rates for each group, then the unadjusted odds ratio, then an odds ratio from a logistic regression adjusted for age, sex and index HbA1c, each with a 95% confidence interval. Write the script to scripts/analyze_pilot.py and the results to out/pilot_results.md. State the model, the number of patients in each model (it should be 251), and why the adjusted estimate differs from the crude one.

Files used: out/cohort.csv (251 patients).

Reference results

Group Patients Rechecked Rate
Not reached 159 47 30%
Reached 92 45 49%
Model Odds ratio 95% CI
Unadjusted 2.28 1.34 to 3.88
Adjusted for age, sex, index HbA1c 1.80 1.03 to 3.15

What this shows. The unadjusted estimate is larger because older patients were more likely both to be called and to have a recheck. The adjusted estimate is closer to the effect built into the synthetic data (an odds ratio of about 1.7), and its interval is wider and just excludes 1. A good write-up says the adjusted estimate is the one to report, and notes that the interval is wide and the design is observational.

Hand check. The unadjusted odds ratio is (45/47) / (47/112) = 0.957 / 0.420 ≈ 2.28. If your agent's number does not match this, find out why before going on.

Your adjusted value may differ slightly if the agent codes sex or age differently; it should still be near 1.8 with an interval close to 1.0 to 3.2.

Verification criteria

  • Sample size consistency: Exposed and unexposed subgroup totals strictly sum to the total cohort sample size.
  • Crude estimate reconciliation: The reported crude odds ratio matches a manual 2x2 contingency table cross-product calculation (\([A \times D] / [B \times C]\)).
  • Model sample retention: Both unadjusted and adjusted models reflect identical sample sizes, confirming no records were dropped due to missing covariate entries.
  • Cautious scientific interpretation: Narrative summaries discuss observed statistical associations without making unverified causal claims.

Common pitfalls

Symptom Underlying cause Resolution
Sample size drops in adjusted model Missing values in covariate columns triggered automatic listwise deletion Impute covariates or verify why records contain missing covariate fields
Highly divergent estimates between runs Non-deterministic sampling or shifting input files Fix input paths; standard logistic regression estimation is completely deterministic
Extreme odds ratios or vast confidence intervals Quasi-complete separation: zero events within a specific covariate sub-stratum Re-evaluate model parameterization, collapse sparse categorical bins, or report model instability

Concepts behind this

  • Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
  • Scoping work. Scope requests to sizes that can be independently verified. Require structured plans before executing complex multi-step pipelines. Go deeper →

Builds on: Build a study cohort with a flow table