Skip to content

Make the agent remember with AGENTS.md

Full step  about 8 min

Because language models maintain no persistent internal state across independent sessions, launching a new conversation typically requires re-explaining directory structures, cleaning conventions, and project constraints. In this exercise, you observe this loss of context firsthand and resolve it by establishing an AGENTS.md configuration file.

Before you start: A project directory where key analytical decisions have already been made (Setup).

Background

When an interactive session terminates, the active conversational context is discarded (Context). The only way to preserve state across sessions is through persistent files.

An AGENTS.md file located in the project root is automatically ingested by most modern agent harnesses at session initialization, providing a standardized, portable mechanism for project memory (Memory).

Effective standing instructions must be concise and specific: documenting directory locations, active cleaning definitions, output paths, and strict negative constraints (such as never overwriting raw datasets). Because every line in AGENTS.md consumes context tokens on every turn, keep the file concise and focused.

General methodology

  1. Observe context loss: Launch a fresh agent session in the project directory and prompt it with a question that depends on prior decisions. Note how the agent either makes incorrect assumptions or asks for clarification on previously resolved conventions.
  2. Draft standing instructions: Instruct the agent to synthesize the established project conventions into AGENTS.md:
Create an AGENTS.md file in the project root (under 25 lines) documenting:
- Relative paths to raw data, cleaned datasets, and the data dictionary.
- Key data cleaning rules and cohort definitions established so far.
- Designated output directories for generated tables and figures.
- Explicit operational constraints (e.g., raw data is read-only; all numbers must report denominators).
Keep instructions concise and specific. Omit trivial details that can be derived directly from the code.
  1. Verify memory persistence: Launch another new session and re-issue the initial query from step 1. The agent should now execute the task adhering to the documented conventions without asking redundant questions.
  2. Refine instructions recursively: Direct the agent: "Review AGENTS.md and remove any vague generalities or redundant rules." Using the agent to audit its own instructions is a high-leverage maintenance practice (Steering).

Workshop example · Imaging: chest radiographs

Start from: the cleaned checkpoint (it has no AGENTS.md yet) (how to restore)
Produces: AGENTS.md

Prompt

Create AGENTS.md in this folder, under 25 lines, capturing: where the cleaned images, labels and scores live; the cleaning decisions in notes/cleaning-decisions.md; that the original files are never modified and outputs go in out/; that no exclusion may depend on a model score; that the unit of analysis is the patient (earliest study); and that every estimate is reported with a 95% interval and the number of patients and positives. Be specific. Leave out anything you could work out from the code.

Files used: notes/cleaning-decisions.md, data/clean/, scripts/clean_images.py, eval_protocol.docx.

Step 1: see the problem. In a new session in this folder, before creating the file, ask:

How well does score_all separate pneumothorax from no pneumothorax in adult patients? Report the AUC with a 95% interval and the number of patients.

You should see the agent do one of two things: read the raw labels.csv and model_scores.csv and ignore your cleaning, or stop to ask which files and rules to use. Either shows what is lost between sessions.

Step 3: test it. After creating AGENTS.md, start another new session and ask the same question. The agent should use data/clean/, count patients, and mention the rules without being told.

Reference answer on the cleaned data: AUC about 0.78 (0.72 to 0.84) on 203 adult patients (59 with pneumothorax), if the agent builds the cohort as the protocol describes (the next step does this explicitly). On the raw files the number is different and has no clear basis.

A reference file is at scenarios/imaging/checkpoints/cohort/AGENTS.md.

Verification criteria

  • Concise and specific: The file remains under 25 lines and provides concrete rules rather than vague generalities.
  • Security compliance: The file contains zero API keys, passwords, or confidential data (Secrets).
  • Effective steering: In a clean test session, the agent correctly cites and adheres to the rules defined in AGENTS.md.

Common pitfalls

Symptom Underlying cause Resolution
Agent ignores instructions File name or directory location does not match harness configuration conventions Verify the expected instruction file convention for your specific agent harness
Bloated, generic instructions Prompt failed to enforce brevity constraints Edit the file to retain only high-impact rules that prevent genuine operational mistakes
Contradictory rules Instructions reflect outdated draft decisions rather than finalized rules Reconcile AGENTS.md directly against notes/cleaning-decisions.md

Concepts behind this

  • Memory. Language models do not retain memory across sessions. Preserving knowledge requires externalizing guidelines, documentation, and tools into persistent files. Go deeper →
  • Context. Context represents the model's active working memory. Because every turn re-transmits the complete conversation, long sessions increase latency, cost, and attention drift. Go deeper →
  • Steering. Balance supervisory approval modes with task risk, intervene early when execution drifts, and codify recurring corrections into standing rules. Go deeper →

Builds on: Clean the data with a script, and show the flow