Inside paper-retrieval

Offline, once per pipeline run: paper-retrieval rebuild --run-id <id>. Everything a search uses is built here from one pipeline run, read-only. How papers get into that run (crawling, GROBID, masterset-toolkit) is in the lab-systems map, not here.

No counts on purpose: they change with every pipeline run. The current ones are in each step's manifest.json and in the server's /health.

One paper through the build

The lab-systems map traces NAACL 2018 “Content-Based Citation Recommendation” (N18-1022) from the venue website to paper-retrieval's snapshot. Inside paper-retrieval it continues like this (snapshot 20261103-152800; a worked example, not a statistic).

Passage units → chunks

91	passage units from the TEI
−18	appendix paragraphs
−12	footnotes (mostly URLs: GitHub, gensim, …)
−9	figure / table captions
−1	back matter (acknowledgements)
−1	template text: “Our main contributions are:” occurs in hundreds of papers
50	abstract + body paragraphs kept
43	chunks: 6 of them merge 2–3 short paragraphs of the same section

One chunk

chunk ff8eec7a-…:c37  section "6 Related Work"
"Citation recommendation systems can be divided into two categories -
 local and global. A local citation recommendation system takes a few
 sentences … as input …"

Indexed with the prefix “title | section”, so a chunk still says which paper and section it is from.

Indexed

In both builds (it is ≤ 2024): title + abstract → 1 paper vector (Qwen3-Embedding-4B); 43 chunks → 43 chunk vectors (0.6B) + 43 BM25 documents; a node in the citation graph; P(baseline) on each of its outgoing citation edges.

Found by a search

The search in “Inside a search” (“local citation recommendation using the full text context of the citing paper”) returns it at rank 5 (paper dense 4, passage dense 3, BM25 30, reranker 8, graph 28), with chunk c37 as evidence. Show row 5 →

A real find_papers call against the running full build (2026-11-12, warm GPU, s end to end)

“”

Results

Where the paper vectors, chunks, BM25 index and citation graph come from: “How the index is built”. Row 5 is the paper traced there.

The layers between you and the index

You

Ask in plain words, in Claude Code or Codex on the laptop.

→

Agent + guide Claude Code connected

lei's laptop: the guide is part of the paperlevelup skill, which uses these tools as its library-search channel (since 2026-11-13).

Elsewhere: agent/SKILL.md as a Claude Code skill, or appended to AGENTS.md for Codex.

The guide only says how to call the tools.

→

MCP tools (7)

Registered once: claude mcp add --scope user paper-retrieval -- ssh -T -o BatchMode=yes <server> '<repo>/.venv/bin/paper-retrieval-mcp'

Adapter speaks MCP on stdio over SSH; no port is opened.

→

paper-retrieval-server

rider, 127.0.0.1:8790, tmux pr-server. Holds all indexes in memory and does the actual retrieval.

Skill vs. tools. The skill is a text file of instructions: “call find_papers once with the user’s question verbatim, report every result in order, re-search only if clearly off-target.” The retrieval itself (BM25, dense vectors, reranker, citation graph) all happens inside one find_papers call on the server. The agent never chooses BM25 vs. dense; those are internal channels, not separate tools.

The seven tools

Tool	Use it for	What runs inside	Role in the guide
`find_papers(query, original_query?, relation_type?, filters?, k=20)`	“Papers on X”, the main entry	Full pipeline: paper dense (Qwen3-Emb-4B) + passage BM25 + passage dense (Qwen3-Emb-0.6B) → RRF → Qwen3-Reranker-4B → citation-graph expansion (PPR + co-citation) → evidence passage per result. See “Inside a search”.	Discover: call once, verbatim
`find_related_papers(paper_id \| url \| text, relation_type?, filters?, k=20)`	“Work related to this paper”; out-of-corpus papers as `text` (title + abstract)	Same pipeline, with the paper’s title + abstract as the query; the paper itself is excluded	Discover
`search_passages(query, paper_ids?, k=20)`	Find where something is said, optionally within given papers	Passage level only: dense (0.6B) top 500 + BM25 top 500, fused by RRF. No reranker, no graph	Verify
`exact_search(term, paper_ids?, k=50)`	Method / dataset names, notation	Case-insensitive literal regex over titles and abstracts (full scan) and passages (BM25 top 20,000 as candidates, then literal check)	Verify
`get_paper(paper_id, section?)`	Metadata; a section’s full text	Table lookup in the snapshot	Verify
`get_citation_neighbors(paper_id, direction="refs"\|"cited_by", intent?)`	Who it cites / who cites it; `intent=baseline` filter	Citation graph + citation-intent probabilities (TF-IDF + logistic regression trained on MasterSet’s type_1 labels)	Verify
`get_research_profile(paper_id)`	Understand one paper in depth	Returns ~14k characters of key passages (abstract, intro start, method, conclusion) with passage ids. The agent writes the profile; no local LLM	Verify

relation_type: any (default) and baseline (graph step uses intent-weighted co-citation) are evaluated. same_problem, same_method, lineage, contradicts are accepted but ranked exactly like any. filters: venues, year_min, year_max, exclude_ids (a mask applied to every channel).

Typical requests → calls

“Papers on local citation recommendation with full-text context”	`find_papers(query=<verbatim>, original_query=<verbatim>)`
“A broad survey of graph drawing with neural nets”	same, with `k=60..100`, not several narrower calls
“Baselines I should compare against for X”	`find_papers(..., relation_type="baseline")`
“Related work for this arXiv paper”	`find_related_papers(text=<title + abstract>)`
“Only ICLR / NeurIPS since 2023”	`filters={"venues":["iclr","neurips"],"year_min":2023}`
“More, but not the ones you already showed”	`filters={"exclude_ids":[…]}`
“Does paper P use dataset D?”	`exact_search("D", paper_ids=[P])` or `search_passages`
“Who builds on paper P?”	`get_citation_neighbors(P, "cited_by")`

Why “call once”: in T4 (reports/V3.md) multi-round follow-ups found no more gold papers than one call with a larger k, and agents lost recall by filtering results. The multi-round guide is kept as agent/SKILL-multiround.md not shipped.

Where things are written down

agent/SKILL.md: the agent guide (what the agent reads)
docs/setup-agent.md: connecting Claude Code / Codex on the laptop
docs/operations.md: start / stop, logs, rebuild after a new pipeline run, the library mirror and changes.json
DESIGN.md §2 architecture, §5 tool contracts
src/paper_retrieval/serving.py: the exact configuration that is served
src/paper_retrieval/engine.py: the search flow; tools.py: tool definitions
docs/decisions.md: every choice with its numbers
Lab systems map: how the projects and machines fit together

Operating it (rider)

curl -s 127.0.0.1:8790/health             # snapshot, counts, gpu_loaded
tmux attach -t pr-server                   # server console
tmux kill-session -t =pr-server            # stop
# calls log: <data root>/runs/server-logs/full/<date>.jsonl
# new pipeline run: paper-retrieval rebuild --run-id <id>
#   snapshot → passages → full build → library mirror + changes.json → POST /admin/reload
#   (the library step needs PR_MUSTCITE_REPO in .env)
# library mirror only: paper-retrieval library --run-id <id>   → libraries/<id>/, libraries/current
# what a run changed:  paper-retrieval changes                 → libraries/<current run>/changes.json

Cold start: the GPU part (~18 GB) is released after 15 idle minutes; the next call reloads it (~10 s). Warm calls take ~2 s.

Consumers and their status

live	Claude Code on the laptop	Connected 2026-11-13 (`docs/setup-agent.md`); first real use: an awesome-list refresh. Codex: same setup, not yet connected.
live	paperlevelup	Since 2026-11-13 its library-search channel (`find_papers`, `find_related_papers`), carrying `agent/SKILL.md`’s rules. Since 2026-11-14 it reads rider only: title matching and keyword search use the FTS5 index of rider’s library mirror (`libraries/current`, in place of archer’s Meilisearch), which stays as cross-check and fallback; metadata, citation gaps and PDFs come from the same mirror and rider’s `raw/`; OpenAlex covers papers outside the library. Results are fetched as `archer:<paper_id>` (the source keeps its old name so stored ids stay valid).
pending	Awesome-list topic refresh	SYSTEMS.md §11: reads rider’s `libraries/<run_id>/changes.json` (the papers each run added and removed, with ids), then calls `find_papers` with venue/year filters for the changed venue-years and keeps results whose `paper_id` was added. First run by hand 2026-11-13; not yet run from `topics.yaml`.
offline	LLM judge (vLLM on lancer)	`llm_judge=true` needs the server started without `--no-llm` and vLLM up. Off: at query time the agent judges from evidence passages.
offline	Comparison services	`pr-meili` (Meilisearch 1.7.6, :7791) and `pr-baseline` (:8792): a reproduction of paperlevelup’s keyword search for the T4 Claude comparison; stopped 2026-11-13, start again only for a new comparison.