Skip to content

Find references from a bibliographic database

Full step  about 9 min

Large language models frequently fabricate plausible-sounding academic citations when prompted from training memory. In this exercise, you implement a programmatic reference retrieval workflow that queries PubMed via external APIs, saves verified records to a BibTeX bibliography, documents search parameters, and validates metadata against upstream registries.

Before you start: A project directory and active network access. (A local fallback archive is available if network access is restricted; see Setup).

Background

A dependable scientific bibliographic pipeline adheres to four core methodological principles:

  1. Programmatic retrieval over memory recall: Every citation entry must be ingested directly from an authoritative bibliographic database, never generated from model parameters.
  2. Persistent identifier tracking: Every entry must store a persistent digital identifier (PMID or DOI) enabling automated re-verification.
  3. Reproducible query auditing: The search string, execution date, total hits returned, and explicit inclusion/exclusion decisions are preserved in an audit log (refs/search-log.md).
  4. Upstream metadata validation: Metadata fields (authors, title, publication year) are systematically validated against registry records before citing.

This exercise uses the National Library of Medicine's PubMed E-utilities API: an open programmatic interface where an esearch query retrieves matching PMIDs and an esummary call retrieves structured metadata.

The generated references.bib file acts as durable project memory (Memory), readable by both the agent and LaTeX compilation engines. Establish a standing rule in AGENTS.md: "All academic citations must be programmatically retrieved and include a valid PMID or DOI; never generate citations from memory."

General methodology

Direct the agent to develop a lightweight, standard-library Python utility to search, append, and verify records:

Create a Python CLI utility, scripts/pubmed_refs.py, using only the Python standard library, supporting three subcommands:
1. `search "<query>"`: Queries PubMed E-utilities, prints candidate PMIDs with publication year, title, first author, and journal, and appends the query and result counts to refs/search-log.md.
2. `add <PMID...>`: Fetches official metadata for the specified PMIDs and appends formatted BibTeX entries to references.bib, preventing duplicates.
3. `verify`: Re-queries PubMed for each PMID in references.bib and asserts that local titles and publication years match official records.
Incorporate polite rate limiting between API requests. Test the utility and display its --help documentation.

Execute the workflow interactively:

Search PubMed for <research topic>. Display candidate papers with titles and abstracts. Pause so I can select which records to include. Then add the selected PMIDs to references.bib and run the verification audit.

The human researcher must make the final selection: relevance to your research question requires scientific judgement. Always read candidate abstracts before citing papers in your manuscripts.

Workshop example · Clinical: clinic network

Start from: the analyzed checkpoint (or any folder) (how to restore)
Produces: scripts/pubmed_refs.py, references.bib, refs/search-log.md

Prompt 1: Add the rule

Add this rule to AGENTS.md: Citations come only from retrieved PubMed records. Each entry in references.bib must contain a PMID. Never write a citation from memory.

Prompt 2: Build the tool

Write a small command-line script, scripts/pubmed_refs.py, using only the Python standard library, with three commands. `search "query"` prints the PMID, year, title, first author and journal of each hit, and appends the query, the date and the results to refs/search-log.md. `add PMID...` fetches those records from PubMed and appends BibTeX entries that include the PMID to references.bib, skipping duplicates. `verify` re-fetches every entry in references.bib and reports any whose title or year differs. Use PubMed E-utilities and wait between requests. Test it, and show me the --help.

Prompt 3: Search

Using scripts/pubmed_refs.py, search PubMed for telephone follow-up diabetes HbA1c. Show me the top 8 candidates with PMID, year and title, and wait for me to choose which to add.

Prompt 4: Add and verify

Add PMIDs 20950378, 21092015, 15842446 and 26871245 to references.bib, then run verify and show me the result.

Run the prompts in order. Check that the tool works (the --help output and a passing test) before you search. The first prompt is the rule from the general page, saved where every future session will read it.

What you should see. A query such as telephone follow-up diabetes HbA1c randomized returned about 200 hits when this material was prepared (the count will differ over time). Among the top results were these, which suit the background of the abstract:

PMID Year Title (shortened)
20950378 2010 Telephone follow-up to improve glycaemic control in type 2 diabetes: systematic review and meta-analysis
21092015 2010 Effect of telephone follow-up on adherence to a diabetes therapeutic regimen
15842446 2005 Nurse follow-up of patients with diabetes: randomized controlled trial
26871245 2016 Does telephone follow-up and education affect self-care and metabolic control in diabetic patients?

The same search also returns papers that are not relevant (a transition-of-care review, a text-message programme, postpartum follow-up). Choosing is your job, and rejecting is part of the log. The last prompt names the four PMIDs from the table so that your result matches; with your own search, pick the candidates that suit your abstract instead.

Fallback. scenarios/toy/extras/pubmed_refs.py is a reference version of the script and extras/references.bib holds these four verified entries. Running python3 pubmed_refs.py verify --bib references.bib should report ok for each.

Your search will not match exactly. Search results change as new papers appear, and the agent may phrase its query differently. The workflow is what matters: retrieve, choose, record, verify.

Verification criteria

  • Persistent identifier presence: Every entry in references.bib contains a valid PMID or DOI, and running scripts/pubmed_refs.py verify exits without error.
  • Auditable search log: The file refs/search-log.md records the exact search query, execution timestamp, and rationale for selected records.
  • Abstract verification: The researcher has inspected the abstract for every cited paper to verify that findings support the intended manuscript claims.

Common pitfalls

Symptom Underlying cause Resolution
HTTP 429 Too Many Requests Script exceeded PubMed's unauthenticated request rate limits (typically 3 requests/second) Add explicit sleep pauses (time.sleep(0.4)) between API calls
Hallucinated reference fields The agent supplemented missing metadata fields from internal memory Mandate that all BibTeX attributes originate strictly from parsed API response payloads
Duplicate bibliography keys The same paper was added under multiple query variations Enforce uniqueness checks on PMIDs and DOIs prior to appending entries

Concepts behind this

  • Tools. Turning interactive prompts into parameterized scripts creates deterministic, testable tools that run identically on every execution. Go deeper →
  • Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
  • Memory. Language models do not retain memory across sessions. Preserving knowledge requires externalizing guidelines, documentation, and tools into persistent files. Go deeper →

Builds on: Orient the agent in a folder