Find references from a bibliographic database
Large language models frequently fabricate plausible-sounding academic citations when prompted from training memory. In this exercise, you implement a programmatic reference retrieval workflow that queries PubMed via external APIs, saves verified records to a BibTeX bibliography, documents search parameters, and validates metadata against upstream registries.
Before you start: A project directory and active network access. (A local fallback archive is available if network access is restricted; see Setup).
Background
A dependable scientific bibliographic pipeline adheres to four core methodological principles:
- Programmatic retrieval over memory recall: Every citation entry must be ingested directly from an authoritative bibliographic database, never generated from model parameters.
- Persistent identifier tracking: Every entry must store a persistent digital identifier (PMID or DOI) enabling automated re-verification.
- Reproducible query auditing: The search string, execution date, total hits returned, and explicit inclusion/exclusion decisions are preserved in an audit log (
refs/search-log.md). - Upstream metadata validation: Metadata fields (authors, title, publication year) are systematically validated against registry records before citing.
This exercise uses the National Library of Medicine's PubMed E-utilities API: an open programmatic interface where an esearch query retrieves matching PMIDs and an esummary call retrieves structured metadata.
The generated references.bib file acts as durable project memory (Memory), readable by both the agent and LaTeX compilation engines. Establish a standing rule in AGENTS.md: "All academic citations must be programmatically retrieved and include a valid PMID or DOI; never generate citations from memory."
General methodology
Direct the agent to develop a lightweight, standard-library Python utility to search, append, and verify records:
Create a Python CLI utility, scripts/pubmed_refs.py, using only the Python standard library, supporting three subcommands:
1. `search "<query>"`: Queries PubMed E-utilities, prints candidate PMIDs with publication year, title, first author, and journal, and appends the query and result counts to refs/search-log.md.
2. `add <PMID...>`: Fetches official metadata for the specified PMIDs and appends formatted BibTeX entries to references.bib, preventing duplicates.
3. `verify`: Re-queries PubMed for each PMID in references.bib and asserts that local titles and publication years match official records.
Incorporate polite rate limiting between API requests. Test the utility and display its --help documentation.
Execute the workflow interactively:
Search PubMed for <research topic>. Display candidate papers with titles and abstracts. Pause so I can select which records to include. Then add the selected PMIDs to references.bib and run the verification audit.
The human researcher must make the final selection: relevance to your research question requires scientific judgement. Always read candidate abstracts before citing papers in your manuscripts.
Workshop example · Clinical: clinic network
Start from: the analyzed checkpoint (or any folder) (how to restore)
Produces: scripts/pubmed_refs.py, references.bib, refs/search-log.md
Prompt 1: Add the rule
Add this rule to AGENTS.md: Citations come only from retrieved PubMed records. Each entry in references.bib must contain a PMID. Never write a citation from memory.
Prompt 2: Build the tool
Write a small command-line script, scripts/pubmed_refs.py, using only the Python standard library, with three commands. `search "query"` prints the PMID, year, title, first author and journal of each hit, and appends the query, the date and the results to refs/search-log.md. `add PMID...` fetches those records from PubMed and appends BibTeX entries that include the PMID to references.bib, skipping duplicates. `verify` re-fetches every entry in references.bib and reports any whose title or year differs. Use PubMed E-utilities and wait between requests. Test it, and show me the --help.
Prompt 3: Search
Using scripts/pubmed_refs.py, search PubMed for telephone follow-up diabetes HbA1c. Show me the top 8 candidates with PMID, year and title, and wait for me to choose which to add.
Prompt 4: Add and verify
Add PMIDs 20950378, 21092015, 15842446 and 26871245 to references.bib, then run verify and show me the result.
Run the prompts in order. Check that the tool works (the --help output and a passing test) before you search. The first prompt is the rule from the general page, saved where every future session will read it.
What you should see. A query such as telephone follow-up diabetes HbA1c randomized returned about 200 hits when this material was
prepared (the count will differ over time). Among the top results were these, which suit the background of the abstract:
| PMID | Year | Title (shortened) |
|---|---|---|
| 20950378 | 2010 | Telephone follow-up to improve glycaemic control in type 2 diabetes: systematic review and meta-analysis |
| 21092015 | 2010 | Effect of telephone follow-up on adherence to a diabetes therapeutic regimen |
| 15842446 | 2005 | Nurse follow-up of patients with diabetes: randomized controlled trial |
| 26871245 | 2016 | Does telephone follow-up and education affect self-care and metabolic control in diabetic patients? |
The same search also returns papers that are not relevant (a transition-of-care review, a text-message programme, postpartum follow-up). Choosing is your job, and rejecting is part of the log. The last prompt names the four PMIDs from the table so that your result matches; with your own search, pick the candidates that suit your abstract instead.
Fallback. scenarios/toy/extras/pubmed_refs.py is a reference version of the script and extras/references.bib holds these four verified
entries. Running python3 pubmed_refs.py verify --bib references.bib should report ok for each.
Your search will not match exactly. Search results change as new papers appear, and the agent may phrase its query differently. The workflow is what matters: retrieve, choose, record, verify.
Verification criteria
- Persistent identifier presence: Every entry in
references.bibcontains a valid PMID or DOI, and runningscripts/pubmed_refs.py verifyexits without error. - Auditable search log: The file
refs/search-log.mdrecords the exact search query, execution timestamp, and rationale for selected records. - Abstract verification: The researcher has inspected the abstract for every cited paper to verify that findings support the intended manuscript claims.
Common pitfalls
| Symptom | Underlying cause | Resolution |
|---|---|---|
| HTTP 429 Too Many Requests | Script exceeded PubMed's unauthenticated request rate limits (typically 3 requests/second) | Add explicit sleep pauses (time.sleep(0.4)) between API calls |
| Hallucinated reference fields | The agent supplemented missing metadata fields from internal memory | Mandate that all BibTeX attributes originate strictly from parsed API response payloads |
| Duplicate bibliography keys | The same paper was added under multiple query variations | Enforce uniqueness checks on PMIDs and DOIs prior to appending entries |
Concepts behind this
- Tools. Turning interactive prompts into parameterized scripts creates deterministic, testable tools that run identically on every execution. Go deeper →
- Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
- Memory. Language models do not retain memory across sessions. Preserving knowledge requires externalizing guidelines, documentation, and tools into persistent files. Go deeper →
Builds on: Orient the agent in a folder