Setup
Before beginning the exercises, configure a dedicated project directory with an active coding agent. You only need to perform this setup once; subsequent tutorial modules assume this environment is in place.
Each demo step specifies its required starting state, allowing you to jump directly into any exercise (Restore a starting point).
Quickstart
An automated setup script installs the sample data and configures a coding agent (Kiro CLI). Use the command for your platform.
macOS, Linux, or Windows via WSL (bash):
curl -fsSL https://peterchang77.github.io/agent-ai-skills/install.sh | bash -s -- --scenario imaging
Windows (PowerShell, no WSL needed):
& ([scriptblock]::Create((irm https://peterchang77.github.io/agent-ai-skills/install.ps1))) -Scenario imaging
The script performs the following actions:
- Prompts for configuration preferences: Confirms the destination directory (defaulting to
~/workshop-imaging), asks whether to set up intermediate checkpoints, and checks whether to install the agent. - Retrieves the benchmark dataset: Offers to install the Python tool
uvif it is missing, downloads the small build scripts (checksum-verified) and builds the image kit on your computer from the public sample (about 33 MB downloaded, a minute or two, the same files every time); optionally builds the checkpoints (about 0.5 GB). The scripts that some steps mention, such asscenarios/imaging/..., are placed in a separate-toolsfolder next to your project folder. - Configures the agent: Downloads and runs the official agent installer after displaying the source URL. This step is skipped if an agent is already detected.
- Ensures idempotent execution: Re-running the script updates missing files without overwriting local modifications unless you explicitly pass the
--forceflag (-Forcein PowerShell) (which creates a timestamped backup before replacing files).
Once the script completes, change into the project directory and launch your agent:
cd ~/workshop-imaging
On Windows PowerShell the default folder is $HOME\workshop-imaging.
If you chose a custom folder, use that path wherever this guide references ~/workshop-imaging. Advanced flags can be passed after bash -s -- (for example, bash -s -- --scenario imaging --dir ~/my-folder --yes --checkpoints --no-agent); pass --help to inspect all options. In PowerShell, run the script as a script block to pass options: & ([scriptblock]::Create((irm https://peterchang77.github.io/agent-ai-skills/install.ps1))) -Scenario imaging -Dir C:\work -Yes -Checkpoints -NoAgent.
[!NOTE] Review network scripts before execution: you can inspect the installer source at install.sh or install.ps1. The workshop dataset is completely synthetic; before using institutional or clinical data, consult Privacy and data handling.
Custom setup
If you prefer to configure your workspace manually, use an alternative agent, or operate within a restricted computing environment, follow these steps:
1. Install an agent
Install your preferred agent harness following the vendor's official documentation. Most tools provide straightforward command-line installation instructions. Always review third-party scripts before execution. For architectural trade-offs between different harnesses, see Harnesses and models. Verify institutional approval before analyzing sensitive or regulated datasets (Privacy and data handling).
2. Create a project directory
An agent scopes its filesystem context to the directory where it is launched. Always initialize a dedicated folder for your project:
mkdir -p ~/workshop-imaging # create project directory
cd ~/workshop-imaging # enter directory
pwd # verify current path
On Windows PowerShell:
mkdir $HOME\workshop-imaging # create project directory
cd $HOME\workshop-imaging # enter directory
pwd # verify current path
Use ls (also available in PowerShell) to audit local contents. For additional guidance on permissions, directory navigation, and shell safety, refer to the Linux command line guide.
3. Verify local software dependencies
Check whether essential system utilities and compilers are available:
uv --version # Python environment and package manager
pdflatex --version # required for PDF report compilation
pandoc --version # required for Word document conversion
These commands work the same in PowerShell. On Windows, pdflatex is provided by MiKTeX or TeX Live.
During Prepare your environment, the agent will audit these dependencies and propose installation commands for missing tools (Environment). Always inspect and approve system-level installations deliberately.
4. Populate scenario data
Follow Your scenario's data below to unpack or build the dataset files. (The automated quickstart performs this step for you.)
Restore a starting point
Each demo exercise defines the prerequisite state it expects, such as the raw data kit or an intermediate checkpoint representing earlier analytical progress. A checkpoint is a self-contained snapshot of the project directory.
To reset your workspace to a clean checkpoint:
rm -rf ~/workshop-imaging && mkdir ~/workshop-imaging # ensure the target folder is disposable before removing
cp -r <scenario folder>/checkpoints/<name>/. ~/workshop-imaging/
cd ~/workshop-imaging
On Windows PowerShell:
Remove-Item -Recurse -Force $HOME\workshop-imaging -ErrorAction SilentlyContinue # ensure the target folder is disposable first
Copy-Item -Recurse <scenario folder>\checkpoints\<name> $HOME\workshop-imaging
cd $HOME\workshop-imaging
If you ran the quickstart with --checkpoints, snapshots are pre-downloaded to ~/workshop-imaging-checkpoints/<name>/ ($HOME\workshop-imaging-checkpoints\<name> on Windows), which replaces <scenario folder>/checkpoints/<name> in the snippet above.
After copying the files, start the agent inside that folder. Maintaining an untouched copy of the data archive allows quick resets between exercises, while version control provides the same safety for your ongoing work.
Expected variance in agent outputs
Large language models generate non-deterministic outputs. Two runs will frequently follow slightly different execution paths, generate different helper filenames, or format code uniquely. This variance is expected. Each demo module provides explicit validation criteria and reference benchmarks so you can verify that the final analytical result is correct regardless of the specific route taken.
Your scenario's data
Scenario: Imaging: chest radiographs
A kit built around real, public chest radiographs (a 250-image sample from the SIIM-ACR pneumothorax challenge, de-identified before release) with a set of synthetic problems added so there are known answers to find. It is a toy demonstration of how to direct an agent through an AI-evaluation workflow, not a validation study. Everything that is not a public image (the data problems, patient details that look real, radiology reports, follow-up images) is synthetic.
The story. You are the analyst. Radiology informatics is considering a pilot of an open-source AI tool that scores chest radiographs for pneumothorax (air that has leaked into the space around the lung). Before agreeing, they want to know how well it works on a sample of images, whether the primary model really is better than three alternatives, and whether it works equally well for different groups of patients. The sample was assembled quickly and has not been checked.
Get the kit
The quickstart installer builds the kit for you. The kit (about 280 files, 74 MB) is generated from the public images by a script, so it is not stored in Git. To build it by hand instead, from the repository's
agent-workshop/ folder, with uv installed:
uv run scenarios/imaging/build_kit.py
mkdir -p ~/workshop-imaging && cp -r scenarios/imaging/kit/. ~/workshop-imaging/
cd ~/workshop-imaging && ls
On Windows PowerShell:
uv run scenarios\imaging\build_kit.py
mkdir $HOME\workshop-imaging -Force
Copy-Item -Recurse scenarios\imaging\kit\* $HOME\workshop-imaging
cd $HOME\workshop-imaging; ls
The first run downloads the 33 MB public image sample and caches it; later runs are offline. The result is the same every time.
| File | Format | What it is |
|---|---|---|
images/img_NNNN.dcm |
DICOM | About 280 chest radiographs (some with problems) |
labels.csv |
CSV | Reference label for each image: pneumothorax or not |
model_scores.csv |
CSV | Scores from four open-source AI models (score_all is the primary model) |
radiology_reports.csv |
CSV | 200 short synthetic radiology reports |
radiology_reports_labeled_sample.csv |
CSV | Correct category for the first 30 reports |
eval_protocol.docx |
DOCX | The evaluation protocol: scope, metrics, pilot bar |
data_dictionary.md |
MD | What each file and field means, and the models' published operating points |
radiology_notes.txt |
TXT | The request from radiology informatics, with five questions |
New to DICOM or to AI evaluation? Read this first.
DICOM is the standard file format for medical images. Each file holds the picture (pixel data) and a header of named fields (tags) such as patient ID, view position, and how to interpret the pixel values. Problems can hide in either part: a header can be inconsistent, and a picture can be inverted, mirrored, or contain burned-in text.
AUC (area under the ROC curve) summarizes how well a score separates positive from negative cases: 0.5 is chance and 1.0 is perfect. Sensitivity is the share of true pneumothoraces the tool flags; specificity is the share of normal images it leaves unflagged. With a small sample, every one of these numbers has a wide margin of error, so we always report an interval.
A note on the models (read once)
The four models come from the open-source TorchXRayVision library and run on an ordinary CPU. These images are believed to come
from the NIH ChestX-ray14 collection, which is part of the training data of two of the models (score_all and score_nih), so their
numbers are likely optimistic. That is one more reason to treat this kit as a demonstration of the workflow, not as evidence about
any tool.
Checkpoints
To start a step without doing the earlier ones, build the checkpoints once (they are copies of the kit plus the work done so far) and restore one (how):
uv run scenarios/imaging/make_checkpoints.py
(The installer does this when you choose checkpoints. It takes a few minutes and about 0.5 GB.)
Checkpoint (in scenarios/imaging/checkpoints/) |
Contains |
|---|---|
| (the kit itself) | The original files only |
cleaned |
Kit + scripts/clean_images.py, data/clean/ (cleaned images, manifest, labels, scores), out/cleaning_flow.md, out/review.csv, notes/cleaning-decisions.md |
cohort |
cleaned + AGENTS.md, scripts/build_cohort.py, out/cohort.csv, out/cohort_flow.md |
evaluated |
cohort + scripts/evaluate.py, out/evaluation.md |
referenced |
evaluated + scripts/pubmed_refs.py, references.bib (four verified PubMed entries), refs/search-log.md, and the citation rule in AGENTS.md |
reported |
referenced + scripts/make_dashboard.py, out/dashboard.html, out/abstract.tex, out/abstract.pdf |
The reference results quoted in each step come from these checkpoints. Your agent's numbers should match closely, or you should be able to explain the difference.