+1 (415) 649-9454
GraphGuru

GraphRAG vs. Plain RAG: A Benchmark on Multi-Hop Questions

Every GraphRAG pitch includes the claim that a knowledge graph answers multi-hop questions better than a vector store. We believe it — we sell it — but a claim you cannot reproduce on your own data is marketing, not engineering. So this post does something different: it gives you a benchmark you can run in an afternoon, on your corpus, that builds the same documents twice (vector-only and Neo4j knowledge graph), asks both the same questions, and scores them. We describe what we consistently see when we run it for clients, and the cases where the graph does not help.

What "multi-hop" means, precisely

A single-hop question is answered by one passage: "What is the notice period in the supplier agreement?" A multi-hop question requires combining facts from passages that do not mention each other: "Which suppliers of the product line we discontinued in 2025 are still under contract?" — the discontinuation is in a product memo, the supplier relationship in a contract, and the contract status in a renewal log.

Vector retrieval finds passages similar to the question. The question mentions discontinuation, suppliers, and contracts, so it retrieves passages about each — but it has no mechanism to ensure the supplier passages are about the right suppliers. The graph does: the discontinued product is a node, SUPPLIED_BY points at the suppliers, UNDER_CONTRACT points at the contracts, and the traversal returns exactly the set. That structural difference is the whole thesis; the benchmark tests whether it shows up in answer quality.

The setup

One corpus, two builds, one question set, one judge.

Corpus

Use your own documents — 50 to 500 is plenty. If you need a public stand-in, any set of interlinked documents works: a company's annual reports plus press releases, a product's changelogs plus support articles. The corpus must contain connected facts or there is nothing to measure.

Build A — plain RAG

Chunk, embed, store in Neo4j as plain Chunk nodes with a vector index (a vector database would do equally well; using Neo4j for both keeps the comparison honest and the infrastructure single).

from neo4j_graphrag.embeddings import OpenAIEmbeddings
from neo4j_graphrag.retrievers import VectorRetriever
from neo4j_graphrag.generation import GraphRAG
from neo4j_graphrag.llm import OpenAILLM

embedder = OpenAIEmbeddings(model="text-embedding-3-small")
llm = OpenAILLM(model_name="gpt-4o", model_params={"temperature": 0})

plain = GraphRAG(
    retriever=VectorRetriever(driver, index_name="chunk_embeddings",
                              embedder=embedder, return_properties=["text"]),
    llm=llm,
)

Build B — knowledge graph

Same chunks, plus entity and relationship extraction with a schema, using SimpleKGPipeline exactly as in our GraphRAG pipeline tutorial. The retriever is a VectorCypherRetriever: vector search for the entry chunks, then a traversal that brings in the entities those chunks mention and their one-hop neighbourhood.

from neo4j_graphrag.retrievers import VectorCypherRetriever

RETRIEVAL = """
MATCH (e:__Entity__)-[:FROM_CHUNK]->(node)
OPTIONAL MATCH (e)-[r]-(nbr:__Entity__)
OPTIONAL MATCH (nbr)-[:FROM_CHUNK]->(ctx:Chunk)
WITH node, e, nbr, r, collect(DISTINCT ctx.text)[0..2] AS nbrText
RETURN node.text AS text,
       collect(DISTINCT e.name + ' -' + type(r) + '-> ' + coalesce(nbr.name, '')) AS facts,
       collect(DISTINCT nbrText) AS context
"""

graph = GraphRAG(
    retriever=VectorCypherRetriever(driver, index_name="chunk_embeddings",
                                    embedder=embedder, retrieval_query=RETRIEVAL),
    llm=llm,
)

Same embedding model, same LLM, same top_k, same prompt. The only variable is what the retriever returns.

Question set

Write 40 questions: 20 single-hop, 20 multi-hop, each with a reference answer a human has verified against the documents. This is the expensive part and the part teams skip. Do not skip it. Tag each question with its hop count and, for multi-hop, the number of documents the answer spans.

questions = [
    {"id": "s01", "hops": 1, "q": "...", "ref": "..."},
    {"id": "m01", "hops": 2, "q": "...", "ref": "..."},
    # ...
]

Judge

An LLM judge scores each answer against the reference on a three-point scale — correct, partially correct, wrong — with the reasoning logged; a human then reviews every "partially correct" and a random 20% of the rest. The judge prompt should say explicitly that an answer which admits it cannot tell is wrong, not partial, because "I don't know" is the failure mode vector-only systems produce most on multi-hop questions.

import json

def judge(question, reference, answer) -> dict:
    prompt = f"""Score the ANSWER against the REFERENCE for the QUESTION.
Return JSON {{"score": 2|1|0, "reason": "..."}} where 2 = correct and complete,
1 = partially correct, 0 = wrong, missing, or declines to answer.
QUESTION: {question}
REFERENCE: {reference}
ANSWER: {answer}"""
    out = llm.invoke(prompt)
    return json.loads(out.content)

results = []
for item in questions:
    for name, system in (("plain", plain), ("graph", graph)):
        resp = system.search(query_text=item["q"], retriever_config={"top_k": 8},
                             return_context=True)
        score = judge(item["q"], item["ref"], resp.answer)
        results.append({**item, "system": name, "answer": resp.answer, **score,
                        "context": [i.content for i in resp.retriever_result.items]})

Aggregate by hops and system. Report mean score and the count of zeros.

What we see when we run this

We are not going to print a percentage from a client engagement as if it were a universal constant — results depend on the corpus, the schema, and how well the extraction step worked. What is consistent across the runs we have done:

  • Single-hop questions are a draw. Both systems retrieve the right passage; the graph's extra context is noise the LLM mostly ignores. Sometimes plain RAG is marginally better here, because the graph retriever's larger context dilutes the answer passage.
  • Multi-hop questions separate the systems. The plain build fails on a substantial fraction — either wrong or "the documents do not say" — whenever the hops cross documents. The graph build answers most of them, and its failures trace to a specific, fixable cause: an entity the extractor missed, two entities it failed to resolve as the same thing, or a relationship type not in the schema.
  • The gap grows with hop count. Three-document questions are close to hopeless for vector-only retrieval and merely hard for the graph.
  • Graph failures are debuggable. Because return_context=True shows the facts retrieved, a wrong graph answer points at the missing edge. A wrong vector answer points at nothing.

That last property is, in our experience, the one that matters most in production. It turns retrieval quality into an engineering loop.

The honest limits

  • Extraction quality caps everything. A sloppy schema or a cheap extraction model produces a graph with missing or wrong edges, and the graph build can then underperform plain RAG. Budget for schema design and for checking a sample of extracted triples by hand.
  • Entity resolution is unsolved in general. "Acme", "Acme Corp", and "ACME Corporation" must become one node or the traversal breaks. Fuzzy resolvers help; a per-domain rule set helps more.
  • Cost and latency are higher. Extraction is an LLM call per chunk at build time; retrieval is a vector search plus a traversal. For a corpus of standalone FAQs that only ever gets single-hop questions, plain RAG is the right choice and the graph is over-engineering.
  • The judge has bias. Use the same judge for both systems, review its partials by hand, and do not trust any single-run difference smaller than your human-review disagreement rate.
  • Schema-constrained extraction can miss the interesting edge. If the question set probes relationships your schema did not anticipate, neither system will find them — but only the graph makes that gap visible.

Run it on your data

The code above, a day of question writing, and an API budget in the tens of dollars gives you a real answer to "would GraphRAG help us?" for your documents. If the multi-hop rows separate, you have a business case; if they do not, you have saved a project.

We run this benchmark as the first week of our GraphRAG consulting engagements, precisely because it tells both of us whether the rest of the engagement is worth doing. If you would like us to run it with you, get in touch.