Skip to content

A first answer from raw data

Full step  about 6 min

An agent can calculate preliminary metrics across a raw dataset in minutes. This exercise demonstrates how to request an initial exploratory estimate while enforcing strict verification standards: requiring explicit variable definitions, reporting exact denominators, and documenting baseline anomalies before cleaning begins.

Before you start: A raw dataset, an accompanying data dictionary, and any relevant performance target files (Setup).

Background

Generating an initial estimate from uncleaned data provides useful orientation, but represents exploratory profiling rather than definitive findings. Uncurated observational datasets almost always contain latent anomalies (such as placeholder test entries, date format discrepancies, or unit mismatches) that silently distort calculations.

A rigorous first pass must explicitly declare the analytical definitions used and report the active sample sizes (denominators) supporting each metric, establishing an empirical baseline against which subsequent cleaning impacts can be audited (Verification).

General methodology

Include the exact operational definition within the prompt to prevent the agent from assuming informal definitions, and mandate explicit sample size reporting:

Using <data file> and <data dictionary>, evaluate: <research question>. Apply this operational definition: <definition>. Report every calculated metric alongside the exact count of underlying rows or individuals (denominator). Do not filter, clean, or modify any data points yet; report the raw numbers and note any data anomalies observed.

Explicitly instructing the agent not to filter data keeps the initial baseline honest: you observe what the uncurated records state, leaving data cleaning decisions for subsequent planned stages (Scoping work).

Workshop example · Clinical: clinic network

Start from: the data kit (how to restore)
Produces: nothing saved; this is a quick exploratory answer (ask the agent to save its script if you want to keep it)

Prompt

Using clinic_visits.csv, data_dictionary.md and the targets in clinic_targets.xlsx, tell me which clinics meet their HbA1c target. Definition: adult patients (age 18 or older) with diabetes (any HbA1c of 6.5 or higher in 2025), whose most recent HbA1c is below 8.0. Report each clinic's share with the number of patients it is based on. Do not change or exclude any data; tell me what you notice about the data.

Files used: clinic_visits.csv, data_dictionary.md, clinic_targets.xlsx (targets: Northside 60%, Riverside 60%, Hillcrest 55%).

What you should see (reference results on the raw data, with this definition)

Clinic Patients Share with most recent HbA1c < 8.0 Target Meets target?
Hillcrest 478 57.7% 55% Yes
Northside 295 65.1% 60% Yes
Riverside 229 53.3% 60% No

Your numbers may differ slightly if the agent reads a rule differently (for example how it treats ties or missing values); the conclusion for each clinic should match.

What to notice. The raw data also produces a few extra "clinics" with tiny counts (northside, hillcrest, and so on). These are spelling variants of the three real clinics, and they are a hint that the data needs cleaning. The next step looks for such problems systematically.

Optional follow-up. Riverside misses its target. The directory page (clinic_directory.html) and staffing.db say a nurse practitioner position has been vacant since June. Ask: "Does staffing explain Riverside's result?" A careful agent will say that staffing is context, not evidence of cause, and will check whether HbA1c actually changed after June (it did not change meaningfully in this data).

Verification criteria

  • Operational definition fidelity: The agent adhered strictly to the provided definition rather than substituting a default metric.
  • Unambiguous denominators: Every summary percentage or mean displays both the numerator and the active denominator.
  • Anomaly transparency: The agent flags irregular records or impossible values observed during calculation without silently omitting them.

Common pitfalls

Symptom Underlying cause Resolution
Calculation uses an unexpected definition Operational definition was omitted or ambiguous Restate the exact formula and prompt: "Which mathematical definition was applied here?"
Missing denominators Prompt did not explicitly mandate sample counts Instruct the agent: "Report the exact row and patient counts supporting each metric"
Silent record exclusions Scripting library omitted nulls or outliers automatically Direct the agent to report the total count of unmapped or dropped rows

Concepts behind this

  • Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
  • Inputs. Language models process information as text. Selecting appropriate ingestion pathways and source formats preserves data structure and prevents extraction errors. Go deeper →
  • Scoping work. Scope requests to sizes that can be independently verified. Require structured plans before executing complex multi-step pipelines. Go deeper →

Builds on: Read and summarize text files