Skip to content

Run an AI model yourself, on a CPU

Optional detour  about 10 min

A table of model scores is only as trustworthy as the process that made it. In this exercise you direct the agent to run the model itself, on an ordinary computer with no GPU, and to compare the result with the scores you were given. A match is good evidence that you understand the pipeline. A mismatch is a lead worth explaining.

Before you start: A cohort or table of items to score, the matching input files, and the scores you were given (Build a cohort). Allow a few minutes for installing the software and downloading the model weights.

Background

A pretrained model is a set of weights plus the code that applies them. The same weights give different numbers if the input is prepared differently: how pixel values are scaled, how the image is resized, whether the colours are inverted, and how the raw output is turned into a probability. So "the score" is a property of the whole pipeline, not of the weights alone.

Running a model also has practical costs (Costs, Environment): the software can be hundreds of megabytes (the default build of the deep-learning library bundles GPU code you do not need), the weights are downloaded the first time, and the run takes time that scales with the number of items. Models of this size run on a CPU in well under a second per image, which is plenty for a few hundred images.

Two cautions apply. First, only run on data you are allowed to process locally: the model runs on your machine and nothing is uploaded, but the software and weights are fetched from the internet (Privacy). Second, a reproduced score tells you the pipeline is the same, not that the model is good: that is what the evaluation is for (Verification).

General methodology

Ask for a plan first, because installing large packages is a decision you should make:

I want to reproduce the scores in <table> by running <model> on <inputs> on the CPU. First tell me which packages and weights you would install, roughly how large they are, and where they would go, and wait for my approval. Then write <scripts/run_model.py> that prepares the inputs as the library documents, runs the model, and writes <out/my_scores.csv> with my score next to the given score. Report the correlation and the largest difference, and explain every item that differs by more than <tolerance>.

Ask the agent to explain mismatches rather than to tune the pipeline until they disappear.

Workshop example · Imaging: chest radiographs

Start from: the cohort checkpoint (how to restore)
Produces: scripts/run_model.py, out/my_scores.csv

Prompt

I want to reproduce score_all in out/cohort.csv by running the primary model myself on the CPU: the TorchXRayVision DenseNet121 with weights densenet121-res224-all, output "Pneumothorax", on the cleaned images in data/clean/images/. First tell me which packages and weights you would install, roughly how large they are, and where they would go (use CPU-only PyTorch, inside this project's environment), and wait for my approval. Then write scripts/run_model.py that prepares each image as the library documents (normalize to its expected range, one channel, 224 x 224), switches off the library's rescaling around each model's operating point, turns the raw output into a probability with the sigmoid, and writes out/my_scores.csv with my score next to the score in out/cohort.csv. Report the correlation and the largest difference, list every image that differs by more than 0.001, and explain each one.

Files used: out/cohort.csv (203 patients), data/clean/images/.

Reference result (primary model, all 203 images, on a CPU): the correlation between your scores and the kit's is about 0.993, the median difference is zero, and 198 of 203 scores agree to within 0.001. The run takes from under a minute on a many-core machine to a few minutes on a laptop. The five that differ, and why:

Image Mine vs kit Why
img_0201, img_0113, img_0273 0.006 vs 0.009, 0.053 vs 0.070, 0.027 vs 0.030 The cleaning step blacked out burned-in patient text, so the pixels differ from the images the kit's scores came from
img_0107, img_0279 0.032 vs 0.063, 0.0025 vs 0.001 These are follow-up images added to the kit; their kit scores are the original image's score plus noise, not real model output (the data dictionary says follow-up images are synthetic)

A good agent finds the explanation by looking at the cleaning manifest and the image headers. A poor one says "rounding" or adjusts the pipeline until the numbers agree. If you ask for the other three models too (--models all,chex,nih,pc), the same kind of five or fewer differ, and the run takes about four times as long.

Check it. Recompute the AUC of your scores against the labels: it should be about 0.78, the same as the evaluation. The weights are downloaded from the library's release page on first use (about 30 MB per model), so the first run needs a network connection.

Things to try

  • Run on the original images/ instead of data/clean/images/ and see which images change by a large amount. The ones that were inverted or tagged MONOCHROME1 are the clearest case of why cleaning comes before scoring.
  • Ask: Which preprocessing choices would change a score by more than 0.01?

A reference version is scenarios/imaging/reference/run_model.py.

Verification criteria

  • Approval before installing. The agent listed the packages and sizes and waited.
  • Scores line up item by item. The comparison is by item (file name), not by row position.
  • Every mismatch has a stated cause, verified by looking at that item, not guessed.
  • Same recipe stated. The preparation steps (scaling, size, output conversion) are written down, so someone else could repeat the run.

Common pitfalls

Symptom Underlying cause Resolution
A multi-gigabyte install The default build of the library includes GPU support Ask for the CPU-only build
Scores are all compressed into a narrow band, or all negative The library's own output rescaling or raw logits were used Ask the agent what the raw output means and how it was converted
Scores match for most items but not for a few The few items were changed or are synthetic Look at each; do not average the problem away
The run is slow Every item was processed one at a time, with every model Start with a small sample and one model, then scale up

Concepts behind this

  • Environment. Computational research requires isolated execution environments to prevent dependency conflicts and guarantee that findings can be reproduced on other systems. Go deeper →
  • Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
  • Costs. Language model pricing is determined by token volume. Because conversation history re-transmits on every turn, session length compounds costs quadratically. Go deeper →

Builds on: Build a study cohort with a flow table