Multi-hop answers through a path you can check
A benchmark and method for answering questions about AI papers, where every hop is tied to an exact source quote.
The research question
When a model answers a multi-step question about a set of research papers, does retrieval from a provenance-linked knowledge graph, with a verified reasoning path, make it more correct and less likely to hallucinate than text-only retrieval? In particular, does it reject false premises and flag conflicting evidence instead of inventing an answer?
Pre-registered hypothesis: compared with text-only RAG under the same base model, prompt and 2,048-token budget, GraphTrace raises the false-premise rejection rate and the share of answers whose reasoning path is fully valid, while answer accuracy on single-hop and multi-hop items drops by no more than 5 points (lower bound of the paired 95% bootstrap CI).
How it works
Every fact in the graph keeps the verbatim quote and character offsets it came from. Anything whose quote cannot be found in the source chunk is rejected, so no edge exists without a source span.
The graph is built once from the corpus. Benchmark items and the graph-based methods both read it; text RAG reads the same chunks directly. Path validity is therefore a measure of traceability against recorded, sourced facts, not of truth on its own.
What the benchmark tests
Items come from fixed templates over the graph. Each stores its required path and gold source chunks, and only items a person accepts against the source text are released.
| Category | Templates | What a correct method does | Target items |
|---|---|---|---|
| Single hop | S1-S5 | Answers from one recorded fact: datasets, method, task, metrics or a reported result of a paper. | 70 |
| Multi hop | M1-M4 | Chains two or three facts across papers, for example a paper's method, the datasets it is evaluated on, and other papers using those datasets. | 70 |
| False premise | F1-F2 | Refuses a question that presupposes something the paper never reports, such as its results on a dataset it does not use, or its use of a method it never mentions. | 40 |
| Conflict or insufficient | C1-C2 | Flags results that disagree across papers, or abstains when no value is reported anywhere. Reported separately from the headline. | 40 |
Methods compared
BM25 passages from the whole corpus. It must still state its reasoning as triples, so path validity is comparable.
Facts around entities linked from the question, written as triples.
Graph facts plus the source passages they came from, filled with BM25 to the budget.
Hybrid context, a required cited path, a deterministic path verifier, NLI support checks and conflict detection. A path that fails verification gets one retry, then the method abstains.
Where the project stands
Corpus built from real data. 120 papers drawn from 697 candidates, 43 benchmark datasets, one connected component.
Pipeline, methods and evaluation built and tested with scripted models; 157 tests run without network or API calls.
Live demo of the model-free stages: BM25 retrieval into the method's context and the NLI claim check on real corpus text.
Graph extraction, benchmark review and the method comparison are pending. They need an API key and a human reviewer. No result is reported until those runs exist.
Limitations
Circularity
Questions come from the same graph the graph-based methods read, which favours them. Gold is checked against source text and results are reported per template.
Graph-relative path validity
A valid path means every hop is a recorded, sourced fact. Truth is measured separately, by accuracy against human-verified gold and grounding precision.
Extraction recall
Missing edges can make a correct text-RAG answer look unsupported. Reviewers reject affected items and the rejection rate is reported.
Parametric knowledge
The base model may already know these 2015-2020 NLP papers. False-premise items test whether it answers from memory.
Lexicon-driven selection
Paper selection favours well-known benchmarks, and short names can be false matches. Selection affects coverage, not gold.
Scope
English NLP papers only, one main base model, and an LLM judge audited by hand on a sample.