Skip to content

Setup

Before beginning the exercises, configure a dedicated project directory with an active coding agent. You only need to perform this setup once; subsequent tutorial modules assume this environment is in place.

Each demo step specifies its required starting state, allowing you to jump directly into any exercise (Restore a starting point).

Quickstart

An automated setup script installs the sample data and configures a coding agent (Kiro CLI). Use the command for your platform.

macOS, Linux, or Windows via WSL (bash):

curl -fsSL https://peterchang77.github.io/agent-ai-skills/install.sh | bash -s -- --scenario imaging

Windows (PowerShell, no WSL needed):

& ([scriptblock]::Create((irm https://peterchang77.github.io/agent-ai-skills/install.ps1))) -Scenario imaging

The script performs the following actions:

  • Prompts for configuration preferences: Confirms the destination directory (defaulting to ~/workshop-imaging), asks whether to set up intermediate checkpoints, and checks whether to install the agent.
  • Retrieves the benchmark dataset: Offers to install the Python tool uv if it is missing, downloads the small build scripts (checksum-verified) and builds the image kit on your computer from the public sample (about 33 MB downloaded, a minute or two, the same files every time); optionally builds the checkpoints (about 0.5 GB). The scripts that some steps mention, such as scenarios/imaging/..., are placed in a separate -tools folder next to your project folder.
  • Configures the agent: Downloads and runs the official agent installer after displaying the source URL. This step is skipped if an agent is already detected.
  • Ensures idempotent execution: Re-running the script updates missing files without overwriting local modifications unless you explicitly pass the --force flag (-Force in PowerShell) (which creates a timestamped backup before replacing files).

Once the script completes, change into the project directory and launch your agent:

cd ~/workshop-imaging

On Windows PowerShell the default folder is $HOME\workshop-imaging.

If you chose a custom folder, use that path wherever this guide references ~/workshop-imaging. Advanced flags can be passed after bash -s -- (for example, bash -s -- --scenario imaging --dir ~/my-folder --yes --checkpoints --no-agent); pass --help to inspect all options. In PowerShell, run the script as a script block to pass options: & ([scriptblock]::Create((irm https://peterchang77.github.io/agent-ai-skills/install.ps1))) -Scenario imaging -Dir C:\work -Yes -Checkpoints -NoAgent.

[!NOTE] Review network scripts before execution: you can inspect the installer source at install.sh or install.ps1. The workshop dataset is completely synthetic; before using institutional or clinical data, consult Privacy and data handling.

Custom setup

If you prefer to configure your workspace manually, use an alternative agent, or operate within a restricted computing environment, follow these steps:

1. Install an agent

Install your preferred agent harness following the vendor's official documentation. Most tools provide straightforward command-line installation instructions. Always review third-party scripts before execution. For architectural trade-offs between different harnesses, see Harnesses and models. Verify institutional approval before analyzing sensitive or regulated datasets (Privacy and data handling).

2. Create a project directory

An agent scopes its filesystem context to the directory where it is launched. Always initialize a dedicated folder for your project:

mkdir -p ~/workshop-imaging     # create project directory
cd ~/workshop-imaging           # enter directory
pwd                          # verify current path

On Windows PowerShell:

mkdir $HOME\workshop-imaging    # create project directory
cd $HOME\workshop-imaging       # enter directory
pwd                          # verify current path

Use ls (also available in PowerShell) to audit local contents. For additional guidance on permissions, directory navigation, and shell safety, refer to the Linux command line guide.

3. Verify local software dependencies

Check whether essential system utilities and compilers are available:

uv --version         # Python environment and package manager
pdflatex --version   # required for PDF report compilation
pandoc --version     # required for Word document conversion

These commands work the same in PowerShell. On Windows, pdflatex is provided by MiKTeX or TeX Live.

During Prepare your environment, the agent will audit these dependencies and propose installation commands for missing tools (Environment). Always inspect and approve system-level installations deliberately.

4. Populate scenario data

Follow Your scenario's data below to unpack or build the dataset files. (The automated quickstart performs this step for you.)

Restore a starting point

Each demo exercise defines the prerequisite state it expects, such as the raw data kit or an intermediate checkpoint representing earlier analytical progress. A checkpoint is a self-contained snapshot of the project directory.

To reset your workspace to a clean checkpoint:

rm -rf ~/workshop-imaging && mkdir ~/workshop-imaging      # ensure the target folder is disposable before removing
cp -r <scenario folder>/checkpoints/<name>/. ~/workshop-imaging/
cd ~/workshop-imaging

On Windows PowerShell:

Remove-Item -Recurse -Force $HOME\workshop-imaging -ErrorAction SilentlyContinue   # ensure the target folder is disposable first
Copy-Item -Recurse <scenario folder>\checkpoints\<name> $HOME\workshop-imaging
cd $HOME\workshop-imaging

If you ran the quickstart with --checkpoints, snapshots are pre-downloaded to ~/workshop-imaging-checkpoints/<name>/ ($HOME\workshop-imaging-checkpoints\<name> on Windows), which replaces <scenario folder>/checkpoints/<name> in the snippet above.

After copying the files, start the agent inside that folder. Maintaining an untouched copy of the data archive allows quick resets between exercises, while version control provides the same safety for your ongoing work.

Expected variance in agent outputs

Large language models generate non-deterministic outputs. Two runs will frequently follow slightly different execution paths, generate different helper filenames, or format code uniquely. This variance is expected. Each demo module provides explicit validation criteria and reference benchmarks so you can verify that the final analytical result is correct regardless of the specific route taken.

Your scenario's data

Scenario: Imaging: chest radiographs

A kit built around real, public chest radiographs (a 250-image sample from the SIIM-ACR pneumothorax challenge, de-identified before release) with a set of synthetic problems added so there are known answers to find. It is a toy demonstration of how to direct an agent through an AI-evaluation workflow, not a validation study. Everything that is not a public image (the data problems, patient details that look real, radiology reports, follow-up images) is synthetic.

The story. You are the analyst. Radiology informatics is considering a pilot of an open-source AI tool that scores chest radiographs for pneumothorax (air that has leaked into the space around the lung). Before agreeing, they want to know how well it works on a sample of images, whether the primary model really is better than three alternatives, and whether it works equally well for different groups of patients. The sample was assembled quickly and has not been checked.

Get the kit

The quickstart installer builds the kit for you. The kit (about 280 files, 74 MB) is generated from the public images by a script, so it is not stored in Git. To build it by hand instead, from the repository's agent-workshop/ folder, with uv installed:

uv run scenarios/imaging/build_kit.py
mkdir -p ~/workshop-imaging && cp -r scenarios/imaging/kit/. ~/workshop-imaging/
cd ~/workshop-imaging && ls

On Windows PowerShell:

uv run scenarios\imaging\build_kit.py
mkdir $HOME\workshop-imaging -Force
Copy-Item -Recurse scenarios\imaging\kit\* $HOME\workshop-imaging
cd $HOME\workshop-imaging; ls

The first run downloads the 33 MB public image sample and caches it; later runs are offline. The result is the same every time.

File Format What it is
images/img_NNNN.dcm DICOM About 280 chest radiographs (some with problems)
labels.csv CSV Reference label for each image: pneumothorax or not
model_scores.csv CSV Scores from four open-source AI models (score_all is the primary model)
radiology_reports.csv CSV 200 short synthetic radiology reports
radiology_reports_labeled_sample.csv CSV Correct category for the first 30 reports
eval_protocol.docx DOCX The evaluation protocol: scope, metrics, pilot bar
data_dictionary.md MD What each file and field means, and the models' published operating points
radiology_notes.txt TXT The request from radiology informatics, with five questions
New to DICOM or to AI evaluation? Read this first.

DICOM is the standard file format for medical images. Each file holds the picture (pixel data) and a header of named fields (tags) such as patient ID, view position, and how to interpret the pixel values. Problems can hide in either part: a header can be inconsistent, and a picture can be inverted, mirrored, or contain burned-in text.

AUC (area under the ROC curve) summarizes how well a score separates positive from negative cases: 0.5 is chance and 1.0 is perfect. Sensitivity is the share of true pneumothoraces the tool flags; specificity is the share of normal images it leaves unflagged. With a small sample, every one of these numbers has a wide margin of error, so we always report an interval.

A note on the models (read once)

The four models come from the open-source TorchXRayVision library and run on an ordinary CPU. These images are believed to come from the NIH ChestX-ray14 collection, which is part of the training data of two of the models (score_all and score_nih), so their numbers are likely optimistic. That is one more reason to treat this kit as a demonstration of the workflow, not as evidence about any tool.

Checkpoints

To start a step without doing the earlier ones, build the checkpoints once (they are copies of the kit plus the work done so far) and restore one (how):

uv run scenarios/imaging/make_checkpoints.py

(The installer does this when you choose checkpoints. It takes a few minutes and about 0.5 GB.)

Checkpoint (in scenarios/imaging/checkpoints/) Contains
(the kit itself) The original files only
cleaned Kit + scripts/clean_images.py, data/clean/ (cleaned images, manifest, labels, scores), out/cleaning_flow.md, out/review.csv, notes/cleaning-decisions.md
cohort cleaned + AGENTS.md, scripts/build_cohort.py, out/cohort.csv, out/cohort_flow.md
evaluated cohort + scripts/evaluate.py, out/evaluation.md
referenced evaluated + scripts/pubmed_refs.py, references.bib (four verified PubMed entries), refs/search-log.md, and the citation rule in AGENTS.md
reported referenced + scripts/make_dashboard.py, out/dashboard.html, out/abstract.tex, out/abstract.pdf

The reference results quoted in each step come from these checkpoints. Your agent's numbers should match closely, or you should be able to explain the difference.