A first answer from raw data
An agent can calculate preliminary metrics across a raw dataset in minutes. This exercise demonstrates how to request an initial exploratory estimate while enforcing strict verification standards: requiring explicit variable definitions, reporting exact denominators, and documenting baseline anomalies before cleaning begins.
Before you start: A raw dataset, an accompanying data dictionary, and any relevant performance target files (Setup).
Background
Generating an initial estimate from uncleaned data provides useful orientation, but represents exploratory profiling rather than definitive findings. Uncurated observational datasets almost always contain latent anomalies (such as placeholder test entries, date format discrepancies, or unit mismatches) that silently distort calculations.
A rigorous first pass must explicitly declare the analytical definitions used and report the active sample sizes (denominators) supporting each metric, establishing an empirical baseline against which subsequent cleaning impacts can be audited (Verification).
General methodology
Include the exact operational definition within the prompt to prevent the agent from assuming informal definitions, and mandate explicit sample size reporting:
Using <data file> and <data dictionary>, evaluate: <research question>. Apply this operational definition: <definition>. Report every calculated metric alongside the exact count of underlying rows or individuals (denominator). Do not filter, clean, or modify any data points yet; report the raw numbers and note any data anomalies observed.
Explicitly instructing the agent not to filter data keeps the initial baseline honest: you observe what the uncurated records state, leaving data cleaning decisions for subsequent planned stages (Scoping work).
Workshop example · Imaging: chest radiographs
Start from: the imaging kit (how to restore)
Produces: nothing saved; this is a quick exploratory answer (ask the agent to save its script if you want to keep it)
Prompt
Using labels.csv, model_scores.csv and data_dictionary.md, tell me how well score_all separates images with a pneumothorax from images without one: report the AUC and the number of images it is based on. Do not change or exclude any data; tell me what you notice about the data.
Files used: labels.csv, model_scores.csv, data_dictionary.md.
What you should see. A naive AUC of roughly 0.72 on about 240 to 260 images. Your number will vary with how the agent joins the files and treats unusual values. The clean-data reference (a later step) gives a different answer, which is the point.
What to notice (all findable from these two files).
- The label column has more than two spellings of the same two classes (
PTX,pneumothorax,Pneumothoraxwith a trailing space,no pneumothorax,Normal,NO PNEUMOTHORAX). - 25 score rows hold values below 0 (the file mixes logits with probabilities).
- Some scores are blank, some images have no score at all, and some file names differ only in letter case, so an exact-match join drops them.
labels.csvhas more rows thanmodel_scores.csv, and a couple of labels refer to image files that are not in the folder.
If the agent quietly drops unmatched rows, ask how many it dropped and why. The next step audits all of this systematically.
Verification criteria
- Operational definition fidelity: The agent adhered strictly to the provided definition rather than substituting a default metric.
- Unambiguous denominators: Every summary percentage or mean displays both the numerator and the active denominator.
- Anomaly transparency: The agent flags irregular records or impossible values observed during calculation without silently omitting them.
Common pitfalls
| Symptom | Underlying cause | Resolution |
|---|---|---|
| Calculation uses an unexpected definition | Operational definition was omitted or ambiguous | Restate the exact formula and prompt: "Which mathematical definition was applied here?" |
| Missing denominators | Prompt did not explicitly mandate sample counts | Instruct the agent: "Report the exact row and patient counts supporting each metric" |
| Silent record exclusions | Scripting library omitted nulls or outliers automatically | Direct the agent to report the total count of unmapped or dropped rows |
Concepts behind this
- Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
- Inputs. Language models process information as text. Selecting appropriate ingestion pathways and source formats preserves data structure and prevents extraction errors. Go deeper →
- Scoping work. Scope requests to sizes that can be independently verified. Require structured plans before executing complex multi-step pipelines. Go deeper →
Builds on: Read and summarize text files