Your retriever found the topic. Did it satisfy the question?
We tested a question asking for research from a particular university, on trapped electrons, within Atomic Physics.
Text search put the correct paper seventh. A basic graph walk put it fifth. Ranking candidates by which relationships actually satisfied the question put it first.
One convenient example proves very little, so we tested 500 questions from the public STaRK-MAG benchmark.
Across two splits, our graph-constraint ranking put a correct paper first on 62.4% and 66.0% of questions. Lexical text search scored 36.4% and 40.4%. A basic graph approach scored 51.2% and 52.4%.
The useful takeaway for builders is that matching a topic and satisfying a question are different retrieval tasks. A paper can sound perfect and still come from the wrong institution.
Some boundaries matter here. This is our methodology benchmark using STaRK’s existing graph, with roughly 26,000 candidate papers per split. It does not test yggy’s document extraction or hosted product, and it is not a comparison against modern vector or hybrid retrieval.
We published the setup, limitations and per-question results so you can inspect where it works and where it loses.
If you’re building an agent, try this on your own evals: keep the topic identical, change one required relationship, and see whether your retriever notices.
Replies
A useful companion test would be a question where each requirement appears somewhere in the graph but no single paper satisfies all of them. Does the retriever return no match, or still promote the nearest paper?