Skip to content

Classify many records with a language-model API

Advanced  about 12 min

In this advanced exercise, you build a batch-processing script that queries an external language-model API to classify hundreds of text records, incorporating cost estimation, incremental checkpointing, and empirical accuracy benchmarking against human-annotated ground truth.

Before you start: An authorized API key configured via environment variables (Use a secret safely), and a dataset of unclassified records alongside a small annotated validation sample. Note that this exercise incurs API costs and transmits text records across external networks (Costs, Privacy and data handling).

Background

Interactive chat interfaces are unsuited for processing hundreds or thousands of empirical records. Developing a dedicated script automates inference record-by-record, records structured responses, and allows interrupted runs to resume.

A production-grade batch classification script requires five engineering guardrails:

  • Strict output schemas: Require predictions as structured JSON matching a predefined enum list to eliminate parsing ambiguity.
  • Dry-run cost estimation: Test prompt formatting on a minimal sample and compute total projected cost before running full-scale processing.
  • Resilient checkpointing and rate limiting: Persist results incrementally to disk after each batch, incorporating exponential backoff to handle API rate limits.
  • Empirical accuracy measurement: Evaluate model classifications against a human-annotated ground truth sample to quantify precision, recall, or accuracy.
  • Reproducible provenance tracking: Record the model identifier, temperature, prompt text, and execution timestamp alongside output predictions.

General methodology

Direct the agent to write a batch classification script supporting dry-run estimation, incremental checkpointing, and rate limiting:

Write a batch classification script, scripts/classify_notes.py, that reads <records file> and prompts <API endpoint> to assign exactly one label from: <label categories>. Ensure responses return as valid JSON. Write classifications incrementally to <out/labels.csv> so interrupted runs can resume without reprocessing existing records. Incorporate exponential backoff retries and polite delays between requests. Support a --limit flag and a --dry-run flag that prints the initial prompt and calculates total projected token cost without executing API calls. Record the model identifier and prompt template within the output metadata.

Execute the evaluation workflow:

  1. Run --dry-run to inspect the assembled prompt and estimated token cost.
  2. Run with --limit 5 to inspect real API response formatting.
  3. Establish a hard spending cap in your provider console and execute the full run.
  4. Calculate classification accuracy against the annotated validation benchmark.

Workshop example · Imaging: chest radiographs

Start from: the committed project with .env set up (how to restore)
Produces: scripts/classify_reports.py, out/report_labels.csv

Prompt

Write scripts/classify_reports.py that reads radiology_reports.csv and, for each report, asks the model whether the report says a pneumothorax is present now, using exactly one label from [present, absent, uncertain], returned as JSON. A pneumothorax that has resolved is absent, and a report that says it cannot be excluded or is equivocal is uncertain. Save results incrementally to out/report_labels.csv so it can resume, with retries and a short delay. Add --limit and --dry-run (print the first prompt and an estimated cost; call nothing). Record the model name and prompt in the output. Then run the dry run and --limit 5, and show me both.

Files used: radiology_reports.csv (200 short synthetic reports), radiology_reports_labeled_sample.csv (correct categories for the first 30).

Steps. Dry run, then --limit 5, check the five by eye, estimate the total from the first batch, set a spending cap, then run all 200. Compare to the 30 labelled reports:

Compare out/report_labels.csv with radiology_reports_labeled_sample.csv for the 30 labelled reports. Report accuracy, show the table of predicted vs. correct categories, and list the reports it got wrong.

What you should see. A table of predicted vs. correct categories for the 30 reports and a list of mismatches. Expect high accuracy, since these synthetic reports are formulaic; a real set of reports would be harder (negation, hedged wording, comparison with older studies). The difficult cases to look at first are the resolved pneumothoraces ("previously seen... has resolved", which are absent) and the hedged ones ("difficult to exclude", "may represent a skin fold", which are uncertain). In the full set of 200, 114 are absent, 48 present and 38 uncertain.

Not a link to the images. The reports were written from templates and do not describe the specific images they are attached to, so do not use them to check a model's score on that image. Treat the exercise as a test of the workflow (dry run, cap, resume, accuracy check), not as evidence of how well a model reads real reports.

Cached result. If you cannot use an API, the labelled sample is enough to practise the comparison step with any set of labels.

Verification criteria

  • Verified dry-run output: The --dry-run command printed a valid prompt structure and an accurate token cost calculation.
  • Incremental state persistence: Output records save incrementally, and restarting an interrupted run resumes from the last processed index.
  • Empirical classification metrics: Accuracy or F1 scores are calculated and reported against the annotated validation sample.
  • Credential security: The API key is injected via environment variables and never printed to logs or output tables.

Common pitfalls

Symptom Underlying cause Resolution
Unparseable response strings Model generated conversational commentary alongside JSON Direct the agent to use structured output mode or parse JSON using regex extractors
Cumulative cost exceeds projections Prompt length increased or retries consumed additional tokens Recompute projections from the initial 5-record pilot and enforce hard budget caps
HTTP 429 Rate Limit errors Concurrency or request frequency exceeded provider quotas Introduce exponential backoff retries (time.sleep(2 ** attempt))

Concepts behind this

  • Tools. Turning interactive prompts into parameterized scripts creates deterministic, testable tools that run identically on every execution. Go deeper →
  • Costs. Language model pricing is determined by token volume. Because conversation history re-transmits on every turn, session length compounds costs quadratically. Go deeper →
  • Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →

Builds on: Use a secret safely