Skip to content

Turn the analysis into a tool and a skill

Full step  about 10 min

In this exercise, you package an interactive multi-step analysis into a tested command-line script and author an agent skill that instructs future sessions when and how to invoke it. Codifying workflows into standalone tools represents the most durable form of project memory: executing future analyses requires a concise instruction rather than an extended exploratory conversation.

Before you start: A verified analysis script, input datasets, and output tables (Setup).

Background

Prompt-driven analytical sessions remain probabilistic: repeating an analysis via conversational prompts can introduce subtle discrepancies in column naming, statistical packages, or output formatting.

In contrast, a deterministic tool (a Python script with explicit arguments and --help documentation) executes the exact same mathematical logic every time, runs independently of the language model, and can be validated with regression tests (Tools).

An agent skill is a modular instruction runbook housed in its own directory that specifies when the agent should invoke the tool and how to evaluate its output, eliminating the need to re-explain procedures in future sessions (Memory).

Component Responsibility Technical properties
Executable tool Analytical execution logic Deterministic, parameterized, tested against reference data
Tool documentation (README.md) Usage instructions, arguments, sample calls Human-readable specification
Agent skill (SKILL.md) Invocation criteria, exact CLI syntax, validation rules Machine-readable runbook ingested dynamically by the harness
Standing instructions (AGENTS.md) Repository-wide guardrails and directory paths Ingested at session initialization across all tasks

General methodology

  1. Package into a parameterized script:
Consolidate data cleaning, cohort definition, and statistical modeling into an executable CLI tool: scripts/pilot_report.py. Provide structured arguments for input datasets and output destination directories, with full --help documentation. Ensure the tool does not modify input datasets. Provide a concise README.md detailing usage.
  1. Implement regression testing: Execute the script on baseline data and confirm that generated metrics match previously verified results. Implement an automated validation test (via pytest or an assertion script) that verifies core statistical estimates against known benchmarks.

  2. Author the modular skill:

Create a custom skill in .agents/skills/pilot-report/SKILL.md (under 40 lines) specifying:
- Trigger criteria: when to invoke this tool.
- Exact CLI execution syntax and required arguments.
- Verification checks to execute on generated output files.
Do not duplicate tool README documentation; reference the script directly.
  1. Verify skill discovery: Launch a clean agent session, request an updated report on the data, and verify that the agent autonomously discovers and executes the skill.

Workshop example · Clinical: clinic network

Start from: the analyzed checkpoint (how to restore)
Produces: scripts/pilot_report.py, README.md, a test, .agents/skills/pilot-report/SKILL.md

Prompt

Combine scripts/clean_visits.py and scripts/analyze_pilot.py into one command-line tool, scripts/pilot_report.py, that takes the paths to clinic_visits.csv and pilot_calls.csv and an output folder, and writes the cleaning flow, cohort, and results. It must not modify its inputs and must have --help. Add a test that checks the key numbers (251 patients, 92 reached, adjusted odds ratio 1.80). Write README.md. Then create the skill .agents/skills/pilot-report/SKILL.md (under 40 lines) saying when to use the tool, the exact command, and what to check.

Files used: scripts/clean_visits.py, scripts/analyze_pilot.py, notes/cleaning-decisions.md.

Known answers for the test. 251 patients in the cohort; 92 reached; 92 rechecked; unadjusted odds ratio 2.28; adjusted odds ratio 1.80 (rounded to two decimals).

A skill sketch (yours will differ):

---
name: pilot-report
description: Produce the Hillcrest follow-up pilot report (cleaning flow, cohort, odds ratios) from the visits and calls CSV files.
---
Use when asked to rerun or update the pilot analysis.
Run: uv run scripts/pilot_report.py clinic_visits.csv pilot_calls.csv out/
Check afterwards: out/cleaning_flow.md counts add up; out/cohort_flow.md final count equals the rows in out/cohort.csv;
report every number with its patient count. See README.md for details.

Prove it. In a new session, ask: "Rerun the pilot report on the current files." The agent should find and run the tool without being told the command.

Next month. If you replace the CSV with next month's extract, the same command produces the new report. That repeatability is the point of the step.

Verification criteria

  • Numerical reproducibility: The packaged tool reproduces earlier empirical point estimates and sample counts exactly.
  • Automated test assertions: The test suite passes on valid data and fails if key numerical values or column schemas are altered.
  • Skill conciseness: The SKILL.md file remains under 40 lines, providing targeted invocation guidelines without redundant documentation.
  • Credential hygiene: The skill, tool, and documentation contain zero hardcoded secrets.

Common pitfalls

Symptom Underlying cause Resolution
Agent fails to invoke the skill in fresh sessions Skill file path does not match harness discovery conventions, or trigger description is vague Verify your harness's expected skill directory path; clarify the description in SKILL.md
Tool outputs diverge from prior results Data filtering or modeling logic was altered during script consolidation Conduct step-by-step diff comparisons between the original exploratory script and the consolidated CLI script

Concepts behind this

  • Tools. Turning interactive prompts into parameterized scripts creates deterministic, testable tools that run identically on every execution. Go deeper →
  • Memory. Language models do not retain memory across sessions. Preserving knowledge requires externalizing guidelines, documentation, and tools into persistent files. Go deeper →
  • Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →

Builds on: Make the agent remember with AGENTS.md, Analyze an exposure and outcome, crude and adjusted