Profile the data and find problems
Observational datasets invariably contain anomalies that can alter downstream statistical estimates. In this exercise, you direct the agent to execute a systematic data-quality audit, quantifying affected record counts and providing concrete examples while keeping raw files strictly untouched.
Before you start: The raw dataset and its accompanying data dictionary (Setup).
Background
Data profiling characterizes dataset distributions and integrity issues without altering records: auditing column types, missingness structures, duplicate keys, physiologically implausible values, and unit discordance.
Profiling is intentionally decoupled from data cleaning. Many apparent anomalies are not data errors; for instance, missing laboratory values often indicate that a test was clinically unnecessary rather than omitted by accident. Furthermore, determining how to address anomalies involves substantive research decisions that belong to the investigator, not the agent (Steering).
| Anomaly class | Illustrative manifestation | Methodological risk |
|---|---|---|
| Entity duplicates | Multiple rows sharing identical patient IDs and timestamps | Distorts variance estimates and inflates sample size |
| Categorical discordance | Inconsistent string casing (e.g., Site A, site a, SITE A) |
Fractures categorical strata during regression modeling |
| Implausible values | Negative ages, blood pressure of 0, heart rate of 999 | Biases distribution means and regression coefficients |
| Unit discordance | Laboratory values recorded in both mg/dL and mmol/L | Introduces bimodal artifactual distributions |
| Mixed date formats | Interleaved YYYY-MM-DD and MM/DD/YYYY strings |
Triggers parsing failures or incorrect temporal sorting |
| Structured missingness | Unpopulated fields across specific visit types | Violates Missing Completely at Random (MCAR) assumptions |
| Content that contradicts its metadata | A file whose header says one thing and whose contents show another (e.g. an image stored upside down or mirrored) | Silently corrupts any measurement taken from the content |
Not all data is a table. For files such as images, the same audit applies to two layers: the metadata (headers and tags, which code can check exhaustively) and the content itself (which code can check only partly, so a person may need to look). Say which layer each finding comes from.
General methodology
Direct the agent to profile the dataset against the data dictionary, quantifying affected rows and isolating real errors from expected structural missingness:
Profile <data file> using <data dictionary>. Document every data-quality anomaly found, reporting the total count of affected rows and 2-3 illustrative row identifiers. Distinguish definitive data corruption from expected structural nulls. Do not modify or clean any records yet.
Follow up by evaluating potential remediation strategies: "For each identified issue, summarize plausible remediation approaches and describe how each option could alter downstream analytical findings." This analysis establishes the decision space for the subsequent planning stage.
Workshop example · Imaging: chest radiographs
Start from: the imaging kit (how to restore)
Produces: a profile report (ask the agent to save it as out/profile.md)
Prompt
Profile the files in images/ (DICOM headers, plus any pixel checks that need no human viewing), labels.csv and model_scores.csv, using data_dictionary.md. List every data-quality problem you find, with the number of affected files and a few example file names. Separate real errors from things that are probably fine, and say which kinds of problem you could not check without looking at the images. Do not fix or change anything. Save the report as out/profile.md.
Files used: images/*.dcm, labels.csv, model_scores.csv, data_dictionary.md.
Reference results. The kit has 283 image files. The planted problems, with counts:
| Layer | Problem | Count | Notes |
|---|---|---|---|
| Header | Patient ID written in upper case, with a leading space, or with a SITE1- prefix |
12 | The same patient would not match across files |
| Header | Sex as MALE/FEMALE, blank, or lower case | 15 | |
| Header | Age blank (8) or implausible, 212Y or 150Y (2) | 10 | 062Y style ages (20) are the standard format, not errors |
| Header | ViewPosition blank | 10 | Recoverable from SeriesDescription ("view: AP") |
| Header | PixelSpacing missing | 6 | |
| Header | Patient name, birth date or MRN in the header | 5 | Synthetic, but treat as PHI |
| Header | Modality DX instead of CR |
10 | Both can describe a chest radiograph; the mix is an inconsistency |
| File | Unreadable (truncated) file | 2 | |
| File | Not a chest radiograph (blank, gradient, checkerboard) | 3 | Header still says CHEST |
| File | Matrix 512 x 512 instead of 1024 x 1024 | 4 | |
| Pixels | Inverted and tagged MONOCHROME1 | 6 | Honest tag; fine once the tag is honored |
| Pixels | Inverted but tagged MONOCHROME2 | 4 | Detectable by a brightness check; confirm by eye |
| Pixels | Mirrored left-right | 4 | Not findable by code |
| Pixels | Patient text burned into the picture | 5 | Found by a check of the top-left corner |
| Duplicates | Identical files (same SOP UID) | 6 | |
| Duplicates | Same pixels, new IDs (2 with the opposite label) | 4 | |
| Duplicates | Second study of the same patient (same patient ID, later date) | 20 | Not an error, but patients appear twice |
| Labels | Spelling variants | 18 | |
| Labels | Conflicting labels for identical pixels | 2 | |
| Labels | Labels for images that are not in the folder | 2 | |
| Scores | Logits instead of probabilities | 25 | All four columns in the row |
| Scores | Blank (5), missing row (6), file name in different case (10) | 21 |
Not errors. Lossy JPEG compression; pixel spacing that varies between scanners; a mix of AP and PA; 8-bit pixels; ages written as bare numbers (62); real burned-in letters such as "AP" and
"PORTABLE" (not patient information); and about 19 images of patients under 18 (relevant to eligibility later, not a defect).
A good report says clearly that mirrored images cannot be found by code, and that it only suspects inversion until someone looks. If your agent finds a problem not on this list, check it: it may be a real quirk of the public images (a few are rotated, shifted or have black margins), or a misreading.
Verification criteria
- Explicit row accounting: Every reported anomaly includes an exact count of affected records and inspectable row identifiers.
- Contextual null assessment: Missing values are evaluated against data dictionary definitions rather than flagged generically as corruption.
- Strictly read-only execution: The raw dataset remains completely unmodified (verified via file checksums or modification timestamps).
Common pitfalls
| Symptom | Underlying cause | Resolution |
|---|---|---|
| Agent modifies files during profiling | Prompt allowed ambiguous write permissions | Intervene immediately: "Profile the data only; do not write or modify any files" |
| Unit discrepancies overlooked | Script validated basic datatypes (float vs string) without evaluating numerical range plausibility | Instruct the agent to compare empirical minimum and maximum ranges against expected physiological boundaries in the data dictionary |
Concepts behind this
- Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
- Steering. Balance supervisory approval modes with task risk, intervene early when execution drifts, and codify recurring corrections into standing rules. Go deeper →
Builds on: A first answer from raw data