Turn the analysis into a tool and a skill
In this exercise, you package an interactive multi-step analysis into a tested command-line script and author an agent skill that instructs future sessions when and how to invoke it. Codifying workflows into standalone tools represents the most durable form of project memory: executing future analyses requires a concise instruction rather than an extended exploratory conversation.
Before you start: A verified analysis script, input datasets, and output tables (Setup).
Background
Prompt-driven analytical sessions remain probabilistic: repeating an analysis via conversational prompts can introduce subtle discrepancies in column naming, statistical packages, or output formatting.
In contrast, a deterministic tool (a Python script with explicit arguments and --help documentation) executes the exact same mathematical logic every time, runs independently of the language model, and can be validated with regression tests (Tools).
An agent skill is a modular instruction runbook housed in its own directory that specifies when the agent should invoke the tool and how to evaluate its output, eliminating the need to re-explain procedures in future sessions (Memory).
| Component | Responsibility | Technical properties |
|---|---|---|
| Executable tool | Analytical execution logic | Deterministic, parameterized, tested against reference data |
Tool documentation (README.md) |
Usage instructions, arguments, sample calls | Human-readable specification |
Agent skill (SKILL.md) |
Invocation criteria, exact CLI syntax, validation rules | Machine-readable runbook ingested dynamically by the harness |
Standing instructions (AGENTS.md) |
Repository-wide guardrails and directory paths | Ingested at session initialization across all tasks |
General methodology
- Package into a parameterized script:
Consolidate data cleaning, cohort definition, and statistical modeling into an executable CLI tool: scripts/pilot_report.py. Provide structured arguments for input datasets and output destination directories, with full --help documentation. Ensure the tool does not modify input datasets. Provide a concise README.md detailing usage.
-
Implement regression testing: Execute the script on baseline data and confirm that generated metrics match previously verified results. Implement an automated validation test (via
pytestor an assertion script) that verifies core statistical estimates against known benchmarks. -
Author the modular skill:
Create a custom skill in .agents/skills/pilot-report/SKILL.md (under 40 lines) specifying:
- Trigger criteria: when to invoke this tool.
- Exact CLI execution syntax and required arguments.
- Verification checks to execute on generated output files.
Do not duplicate tool README documentation; reference the script directly.
- Verify skill discovery: Launch a clean agent session, request an updated report on the data, and verify that the agent autonomously discovers and executes the skill.
Workshop example · Imaging: chest radiographs
Start from: the evaluated checkpoint (how to restore)
Produces: scripts/eval_report.py, README.md, a test, .agents/skills/cxr-eval/SKILL.md
Prompt
Combine scripts/clean_images.py, scripts/build_cohort.py and scripts/evaluate.py into one command-line tool, scripts/eval_report.py, that takes the path to the kit folder (images, labels.csv, model_scores.csv), my out/review.csv and an output folder, and writes the cleaning flow, cohort, cohort flow and evaluation. It must not modify its inputs and must have --help. Add a test that checks the key numbers (258 cleaned images, 203 patients, 59 with pneumothorax, AUC of score_all 0.78). Write README.md. Then create the skill .agents/skills/cxr-eval/SKILL.md (under 40 lines) saying when to use the tool, the exact command, and what to check.
Files used: scripts/clean_images.py, scripts/build_cohort.py, scripts/evaluate.py, notes/cleaning-decisions.md, out/review.csv.
Known answers for the test. 258 cleaned images; 247, 237, 220 and 203 patients through the cohort steps; 59 with pneumothorax; AUC of score_all 0.78 (0.715 to 0.843); sensitivity 0.92 and specificity 0.47 at the published threshold (rounded). The bootstrap is random, so the test should allow a
small tolerance on the interval, and a fixed seed makes it repeatable.
A skill sketch (yours will differ):
name: cxr-eval
description: Clean a chest-radiograph kit and evaluate pneumothorax detector scores (cleaning flow, cohort, AUC with intervals).
Use when asked to rerun or update the detector evaluation on a new batch of images and scores.
Run: uv run scripts/eval_report.py <kit folder> out/review.csv out/
Check afterwards: out/cleaning_flow.md counts add up; out/cohort_flow.md final count equals the rows in out/cohort.csv;
every estimate has an interval and patient counts. Human review (out/review.csv) must be redone for new images. See README.md.
Prove it. In a new session, ask: "Rerun the detector evaluation on the current files." The agent should find and run the tool without being told the command.
New batch. With a new batch the same command produces a new report, but the review file does not carry over: mirrored or inverted images must be found by a person again. The tool should say so rather than silently reuse the old file.
Verification criteria
- Numerical reproducibility: The packaged tool reproduces earlier empirical point estimates and sample counts exactly.
- Automated test assertions: The test suite passes on valid data and fails if key numerical values or column schemas are altered.
- Skill conciseness: The
SKILL.mdfile remains under 40 lines, providing targeted invocation guidelines without redundant documentation. - Credential hygiene: The skill, tool, and documentation contain zero hardcoded secrets.
Common pitfalls
| Symptom | Underlying cause | Resolution |
|---|---|---|
| Agent fails to invoke the skill in fresh sessions | Skill file path does not match harness discovery conventions, or trigger description is vague | Verify your harness's expected skill directory path; clarify the description in SKILL.md |
| Tool outputs diverge from prior results | Data filtering or modeling logic was altered during script consolidation | Conduct step-by-step diff comparisons between the original exploratory script and the consolidated CLI script |
Concepts behind this
- Tools. Turning interactive prompts into parameterized scripts creates deterministic, testable tools that run identically on every execution. Go deeper →
- Memory. Language models do not retain memory across sessions. Preserving knowledge requires externalizing guidelines, documentation, and tools into persistent files. Go deeper →
- Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
Builds on: Make the agent remember with AGENTS.md, Evaluate an AI tool, with intervals