Look at the images
Tables can be summarized with counts. Images cannot: a count of files tells you nothing about whether the pictures are right. In this exercise you direct the agent to read image metadata and to produce a contact sheet that a person can look at, and you check the agent's description of the images against your own.
Before you start: A folder of image files and a Python environment that can read them (Setup).
Background
Many scientific and clinical images are stored in formats that pair pixels with metadata. The metadata describes who or what the image shows, how it was made, and how to display it. Neither layer is reliable on its own: the metadata can be inconsistent, and the pixels can be wrong in ways the metadata does not mention.
Agents differ in whether they can see images at all. Some can open a picture and describe it; others cannot. Either way, an agent's description of an image is a claim to verify, not a measurement. The most dependable pattern is to have the agent produce an image file or web page that you open (Verification).
Images also cost a great deal of the agent's working memory (Context). A single contact sheet of 30 small thumbnails is far cheaper than viewing 30 images one at a time, and is usually enough.
General methodology
Ask for a metadata summary first, then a labelled contact sheet of a small, random sample, and open it yourself:
List the image files in <folder> and read their metadata (not the pixels yet). Summarize the main fields: how many files, which formats, and the distribution of the fields that describe how each image was acquired and displayed. Then save a contact sheet of <N> randomly chosen images as <out/contact_sheet.png> (or a single self-contained HTML page), with each thumbnail labelled by its file name. Tell me how to open it.
Open the contact sheet in any image viewer or browser. If your agent can view images, ask it to describe what it sees, then compare with your own reading.
Workshop example · Imaging: chest radiographs
Start from: the imaging kit and the environment from the previous step (how to restore)
Produces: out/contact_sheet.png (or .html) and a metadata summary
Prompt
Read the DICOM headers (not the pixels yet) of the files in images/ and summarize them: how many files, which modality, the view positions, the sex and age fields, the image size, and how the pixel values are meant to be interpreted. Then save a contact sheet of 24 randomly chosen images, each labelled with its file name, as out/contact_sheet.png, and tell me how to open it. Do not modify anything in images/. Tell me which of your statements came from the headers and which from the pixels.
Files used: images/*.dcm.
What you should see in the headers. About 283 files; almost all CR (computed radiography) with a few DX; PA and AP views with some blank; 1024 x 1024 images
for nearly all; sex, age and patient ID fields that are mostly clean but not entirely. A good agent will say that some headers are inconsistent and that some files cannot be read.
It should say that it has not yet checked the pixels (it has not looked at the pictures).
What you should see in the contact sheet. Frontal chest radiographs, mostly dark lungs on a lighter body. Several show burned-in letters such as "AP" or "PORTABLE", and some show tubes or drains. With a random sample of 24 you may or may not see a problem image. Whether you do is part of the lesson: a quick look does not replace a systematic audit.
Check it yourself. Open the sheet and compare with the agent's description. If your agent can view images, ask it what it sees and compare. Note that the contact sheet labels come from file names, and the file names say nothing about the diagnosis.
If your environment cannot show an image directly, ask for a self-contained HTML page instead and open it in a browser. In a headless or remote setting, serve the folder with
python3 -m http.server and forward the port.
Verification criteria
- You looked at it. The contact sheet was opened by a person, not only summarized by the agent.
- Thumbnails are labelled. Every thumbnail can be traced back to a file name.
- Metadata and pixels are kept separate. The agent says which findings came from metadata and which from pixels, and does not claim to have checked pixels it did not open.
- Originals untouched. Nothing was written into the image folder.
Common pitfalls
| Symptom | Underlying cause | Resolution |
|---|---|---|
| The agent confidently describes images it never opened | It inferred from file names or metadata | Ask: "Which of these statements came from the pixels, and which from the metadata?" |
| The context fills up quickly | Many full-size images were loaded one by one | Ask for one downsampled contact sheet instead |
| Images look washed out, black, or inverted | The display step ignored how the file says its pixel values should be interpreted | Note it; this is exactly the kind of problem the next steps look for |
Concepts behind this
- Inputs. Language models process information as text. Selecting appropriate ingestion pathways and source formats preserves data structure and prevents extraction errors. Go deeper →
- Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
Builds on: Prepare your environment