Classify many records with a language-model API
In this advanced exercise, you build a batch-processing script that queries an external language-model API to classify hundreds of text records, incorporating cost estimation, incremental checkpointing, and empirical accuracy benchmarking against human-annotated ground truth.
Before you start: An authorized API key configured via environment variables (Use a secret safely), and a dataset of unclassified records alongside a small annotated validation sample. Note that this exercise incurs API costs and transmits text records across external networks (Costs, Privacy and data handling).
Background
Interactive chat interfaces are unsuited for processing hundreds or thousands of empirical records. Developing a dedicated script automates inference record-by-record, records structured responses, and allows interrupted runs to resume.
A production-grade batch classification script requires five engineering guardrails:
- Strict output schemas: Require predictions as structured JSON matching a predefined enum list to eliminate parsing ambiguity.
- Dry-run cost estimation: Test prompt formatting on a minimal sample and compute total projected cost before running full-scale processing.
- Resilient checkpointing and rate limiting: Persist results incrementally to disk after each batch, incorporating exponential backoff to handle API rate limits.
- Empirical accuracy measurement: Evaluate model classifications against a human-annotated ground truth sample to quantify precision, recall, or accuracy.
- Reproducible provenance tracking: Record the model identifier, temperature, prompt text, and execution timestamp alongside output predictions.
General methodology
Direct the agent to write a batch classification script supporting dry-run estimation, incremental checkpointing, and rate limiting:
Write a batch classification script, scripts/classify_notes.py, that reads <records file> and prompts <API endpoint> to assign exactly one label from: <label categories>. Ensure responses return as valid JSON. Write classifications incrementally to <out/labels.csv> so interrupted runs can resume without reprocessing existing records. Incorporate exponential backoff retries and polite delays between requests. Support a --limit flag and a --dry-run flag that prints the initial prompt and calculates total projected token cost without executing API calls. Record the model identifier and prompt template within the output metadata.
Execute the evaluation workflow:
- Run
--dry-runto inspect the assembled prompt and estimated token cost. - Run with
--limit 5to inspect real API response formatting. - Establish a hard spending cap in your provider console and execute the full run.
- Calculate classification accuracy against the annotated validation benchmark.
Workshop example · Clinical: clinic network
Start from: the committed project with .env set up (how to restore)
Produces: scripts/classify_notes.py, out/note_labels.csv
Prompt
Write scripts/classify_notes.py that reads visit_notes.csv and, for each note, asks the model for exactly one label from [medication_refill, new_symptom, chronic_followup, preventive, administrative], returned as JSON. Save results incrementally to out/note_labels.csv so it can resume, with retries and a short delay. Add --limit and --dry-run (print the first prompt and an estimated cost; call nothing). Record the model name and prompt in the output. Then run the dry run and --limit 5, and show me both.
Files used: visit_notes.csv (200 short synthetic notes), visit_notes_labeled_sample.csv (correct labels for the first 30).
Steps. Dry run, then --limit 5, check the five by eye, estimate the total from the first batch, set a spending cap, then run all 200. Compare to the 30 labelled notes:
Compare out/note_labels.csv with visit_notes_labeled_sample.csv for the 30 labelled notes. Report accuracy, and list the notes it got wrong.
What you should see. A table of predicted vs. correct labels for the 30 notes and a list of mismatches. Expect high accuracy, since these synthetic notes are formulaic: a simple keyword rule gets about 97% on them. A real set of clinical notes would be harder, so treat the exercise as a test of the workflow (dry run, cap, resume, accuracy check), not as evidence of how well a model classifies real notes.
Cached result. If you cannot use an API, the labelled sample is enough to practise the comparison step with any set of labels.
Verification criteria
- Verified dry-run output: The
--dry-runcommand printed a valid prompt structure and an accurate token cost calculation. - Incremental state persistence: Output records save incrementally, and restarting an interrupted run resumes from the last processed index.
- Empirical classification metrics: Accuracy or F1 scores are calculated and reported against the annotated validation sample.
- Credential security: The API key is injected via environment variables and never printed to logs or output tables.
Common pitfalls
| Symptom | Underlying cause | Resolution |
|---|---|---|
| Unparseable response strings | Model generated conversational commentary alongside JSON | Direct the agent to use structured output mode or parse JSON using regex extractors |
| Cumulative cost exceeds projections | Prompt length increased or retries consumed additional tokens | Recompute projections from the initial 5-record pilot and enforce hard budget caps |
| HTTP 429 Rate Limit errors | Concurrency or request frequency exceeded provider quotas | Introduce exponential backoff retries (time.sleep(2 ** attempt)) |
Concepts behind this
- Tools. Turning interactive prompts into parameterized scripts creates deterministic, testable tools that run identically on every execution. Go deeper →
- Costs. Language model pricing is determined by token volume. Because conversation history re-transmits on every turn, session length compounds costs quadratically. Go deeper →
- Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
Builds on: Use a secret safely