Skip to content

Profile the data and find problems

Full step  about 7 min

Observational datasets invariably contain anomalies that can alter downstream statistical estimates. In this exercise, you direct the agent to execute a systematic data-quality audit, quantifying affected record counts and providing concrete examples while keeping raw files strictly untouched.

Before you start: The raw dataset and its accompanying data dictionary (Setup).

Background

Data profiling characterizes dataset distributions and integrity issues without altering records: auditing column types, missingness structures, duplicate keys, physiologically implausible values, and unit discordance.

Profiling is intentionally decoupled from data cleaning. Many apparent anomalies are not data errors; for instance, missing laboratory values often indicate that a test was clinically unnecessary rather than omitted by accident. Furthermore, determining how to address anomalies involves substantive research decisions that belong to the investigator, not the agent (Steering).

Anomaly class Illustrative manifestation Methodological risk
Entity duplicates Multiple rows sharing identical patient IDs and timestamps Distorts variance estimates and inflates sample size
Categorical discordance Inconsistent string casing (e.g., Site A, site a, SITE A) Fractures categorical strata during regression modeling
Implausible values Negative ages, blood pressure of 0, heart rate of 999 Biases distribution means and regression coefficients
Unit discordance Laboratory values recorded in both mg/dL and mmol/L Introduces bimodal artifactual distributions
Mixed date formats Interleaved YYYY-MM-DD and MM/DD/YYYY strings Triggers parsing failures or incorrect temporal sorting
Structured missingness Unpopulated fields across specific visit types Violates Missing Completely at Random (MCAR) assumptions
Content that contradicts its metadata A file whose header says one thing and whose contents show another (e.g. an image stored upside down or mirrored) Silently corrupts any measurement taken from the content

Not all data is a table. For files such as images, the same audit applies to two layers: the metadata (headers and tags, which code can check exhaustively) and the content itself (which code can check only partly, so a person may need to look). Say which layer each finding comes from.

General methodology

Direct the agent to profile the dataset against the data dictionary, quantifying affected rows and isolating real errors from expected structural missingness:

Profile <data file> using <data dictionary>. Document every data-quality anomaly found, reporting the total count of affected rows and 2-3 illustrative row identifiers. Distinguish definitive data corruption from expected structural nulls. Do not modify or clean any records yet.

Follow up by evaluating potential remediation strategies: "For each identified issue, summarize plausible remediation approaches and describe how each option could alter downstream analytical findings." This analysis establishes the decision space for the subsequent planning stage.

Workshop example · Imaging: chest radiographs

Start from: the imaging kit (how to restore)
Produces: a profile report (ask the agent to save it as out/profile.md)

Prompt

Profile the files in images/ (DICOM headers, plus any pixel checks that need no human viewing), labels.csv and model_scores.csv, using data_dictionary.md. List every data-quality problem you find, with the number of affected files and a few example file names. Separate real errors from things that are probably fine, and say which kinds of problem you could not check without looking at the images. Do not fix or change anything. Save the report as out/profile.md.

Files used: images/*.dcm, labels.csv, model_scores.csv, data_dictionary.md.

Reference results. The kit has 283 image files. The planted problems, with counts:

Layer Problem Count Notes
Header Patient ID written in upper case, with a leading space, or with a SITE1- prefix 12 The same patient would not match across files
Header Sex as MALE/FEMALE, blank, or lower case 15
Header Age blank (8) or implausible, 212Y or 150Y (2) 10 062Y style ages (20) are the standard format, not errors
Header ViewPosition blank 10 Recoverable from SeriesDescription ("view: AP")
Header PixelSpacing missing 6
Header Patient name, birth date or MRN in the header 5 Synthetic, but treat as PHI
Header Modality DX instead of CR 10 Both can describe a chest radiograph; the mix is an inconsistency
File Unreadable (truncated) file 2
File Not a chest radiograph (blank, gradient, checkerboard) 3 Header still says CHEST
File Matrix 512 x 512 instead of 1024 x 1024 4
Pixels Inverted and tagged MONOCHROME1 6 Honest tag; fine once the tag is honored
Pixels Inverted but tagged MONOCHROME2 4 Detectable by a brightness check; confirm by eye
Pixels Mirrored left-right 4 Not findable by code
Pixels Patient text burned into the picture 5 Found by a check of the top-left corner
Duplicates Identical files (same SOP UID) 6
Duplicates Same pixels, new IDs (2 with the opposite label) 4
Duplicates Second study of the same patient (same patient ID, later date) 20 Not an error, but patients appear twice
Labels Spelling variants 18
Labels Conflicting labels for identical pixels 2
Labels Labels for images that are not in the folder 2
Scores Logits instead of probabilities 25 All four columns in the row
Scores Blank (5), missing row (6), file name in different case (10) 21

Not errors. Lossy JPEG compression; pixel spacing that varies between scanners; a mix of AP and PA; 8-bit pixels; ages written as bare numbers (62); real burned-in letters such as "AP" and "PORTABLE" (not patient information); and about 19 images of patients under 18 (relevant to eligibility later, not a defect).

A good report says clearly that mirrored images cannot be found by code, and that it only suspects inversion until someone looks. If your agent finds a problem not on this list, check it: it may be a real quirk of the public images (a few are rotated, shifted or have black margins), or a misreading.

Verification criteria

  • Explicit row accounting: Every reported anomaly includes an exact count of affected records and inspectable row identifiers.
  • Contextual null assessment: Missing values are evaluated against data dictionary definitions rather than flagged generically as corruption.
  • Strictly read-only execution: The raw dataset remains completely unmodified (verified via file checksums or modification timestamps).

Common pitfalls

Symptom Underlying cause Resolution
Agent modifies files during profiling Prompt allowed ambiguous write permissions Intervene immediately: "Profile the data only; do not write or modify any files"
Unit discrepancies overlooked Script validated basic datatypes (float vs string) without evaluating numerical range plausibility Instruct the agent to compare empirical minimum and maximum ranges against expected physiological boundaries in the data dictionary

Concepts behind this

  • Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
  • Steering. Balance supervisory approval modes with task risk, intervene early when execution drifts, and codify recurring corrections into standing rules. Go deeper →

Builds on: A first answer from raw data