+1 (415) 649-9454

Load Testing Neo4j: Building a Workload Model, a Saturation Curve, and a CI Regression Gate

Most Neo4j performance work starts after the incident: the graph got bigger, a feature shipped, and p99 latency went from 40 ms to four seconds. By then you are tuning under pressure with no baseline to compare against. A load test fixes that — not the vanity kind that fires the same query a million times, but a workload model that resembles your real traffic, run on every release, with a number that fails the build when it regresses.

This tutorial builds that: a parameterised workload, a k6 test driving it through the Neo4j Query API, the server-side metrics worth watching while it runs, and a regression gate wired into CI.

Why not just time a query in the browser?

Single-query timings in Neo4j Browser lie in three ways. They include the browser's own rendering, they are measured with a warm page cache and a warm query cache after the first run, and they never show you what happens when forty of those queries run concurrently against a database that is also writing. Lock contention, page-cache thrash and checkpoint pauses only appear under concurrency.

So the unit of measurement is a workload: a mix of query shapes, in realistic proportions, with realistic parameters, run at a controlled arrival rate.

Step 1 — capture a real workload mix

The query log already knows your traffic. On a system with db.logs.query.enabled=INFO, each entry carries the query text, parameters, runtime and allocated memory. Aggregate a day of it by normalised query text:

# query.log is JSON-formatted on 5.x+ when db.logs.query.format=json
jq -r 'select(.event=="query") | [.query, .elapsedTimeMs] | @tsv' query.log \
  | awk -F'\t' '{count[$1]++; total[$1]+=$2} END {for (q in count) printf "%d\t%.1f\t%s\n", count[q], total[q]/count[q], q}' \
  | sort -rn | head -20

You want two lists from this: the top queries by call count (they define the mix) and the top queries by total time (they define where tuning pays). In practice five to eight query shapes cover 90% of traffic. That is your workload model.

Record the proportions. A read-heavy product graph might come out as: 55% entity lookup by key, 20% one-to-two hop neighbourhood fetch, 15% recommendation/similarity read, 8% write of a single event, 2% analytical aggregation.

Step 2 — generate realistic parameters

The single biggest mistake in Neo4j load testing is reusing one parameter value. Query one node ten thousand times and you are benchmarking the page cache, not the database. Pull a parameter pool out of the graph itself, with the same skew your users have:

MATCH (c:Customer)
WHERE c.id IS NOT NULL
WITH c, rand() AS r ORDER BY r LIMIT 10000
RETURN c.id AS id

Export that to CSV and let the load generator pick rows at random. If your real traffic is skewed — a few hot accounts, a long tail of cold ones — reproduce the skew, because a uniform draw over ten million nodes is a harder test than production and will send you tuning problems you do not have. A simple approach: take 80% of draws from a 1,000-row hot pool and 20% from a 100,000-row cold pool.

Step 3 — drive it with k6 over the Query API

k6 speaks HTTP, and Neo4j's Query API gives it a clean target — no JVM, no driver, trivially containerised. (If you need to measure driver behaviour specifically — connection pooling, bookmarks, cluster routing — use Gatling with the Java driver instead; the workload model is identical.)

// loadtest.js
import http from 'k6/http';
import { check } from 'k6';
import { SharedArray } from 'k6/data';
import { randomSeed } from 'k6';

const hot  = new SharedArray('hot',  () => JSON.parse(open('./hot_ids.json')));
const cold = new SharedArray('cold', () => JSON.parse(open('./cold_ids.json')));

const URL = `${__ENV.NEO4J_HTTP}/db/neo4j/query/v2`;
const PARAMS = {
  headers: {
    'Content-Type': 'application/json',
    'Accept': 'application/json',
    'Authorization': `Basic ${__ENV.NEO4J_AUTH_B64}`,
  },
  tags: {},
};

export const options = {
  scenarios: {
    steady: {
      executor: 'constant-arrival-rate',
      rate: 200, timeUnit: '1s',
      duration: '10m',
      preAllocatedVUs: 50, maxVUs: 300,
    },
  },
  thresholds: {
    'http_req_duration{shape:lookup}':       ['p(95)<25'],
    'http_req_duration{shape:neighbourhood}':['p(95)<120'],
    'http_req_duration{shape:recommend}':    ['p(95)<400'],
    'http_req_failed':                       ['rate<0.001'],
  },
};

function pickId() {
  return Math.random() < 0.8
    ? hot[Math.floor(Math.random() * hot.length)]
    : cold[Math.floor(Math.random() * cold.length)];
}

function run(shape, statement, parameters) {
  const body = JSON.stringify({ statement, parameters });
  const res = http.post(URL, body, { ...PARAMS, tags: { shape } });
  check(res, { 'status 202/200': (r) => r.status < 300 });
  return res;
}

export default function () {
  const r = Math.random();
  const id = pickId();

  if (r < 0.55) {
    run('lookup',
      'MATCH (c:Customer {id: $id}) RETURN c.id, c.name, c.tier',
      { id });
  } else if (r < 0.75) {
    run('neighbourhood',
      `MATCH (c:Customer {id: $id})-[:PLACED]->(o:Order)-[:CONTAINS]->(p:Product)
       RETURN p.sku AS sku, count(*) AS n ORDER BY n DESC LIMIT 20`,
      { id });
  } else if (r < 0.90) {
    run('recommend',
      `MATCH (c:Customer {id: $id})-[:PLACED]->(:Order)-[:CONTAINS]->(p:Product)
             <-[:CONTAINS]-(:Order)<-[:PLACED]-(peer:Customer)
       WHERE peer <> c
       MATCH (peer)-[:PLACED]->(:Order)-[:CONTAINS]->(rec:Product)
       WHERE NOT EXISTS { (c)-[:PLACED]->(:Order)-[:CONTAINS]->(rec) }
       RETURN rec.sku AS sku, count(DISTINCT peer) AS score
       ORDER BY score DESC LIMIT 10`,
      { id });
  } else if (r < 0.98) {
    run('write',
      `MATCH (c:Customer {id: $id})
       CREATE (c)-[:VIEWED {at: datetime()}]->(:PageView {path: $path})`,
      { id, path: '/p/' + Math.floor(Math.random() * 5000) });
  } else {
    run('analytics',
      `MATCH (o:Order) WHERE o.placedAt > datetime() - duration('P1D')
       RETURN o.channel AS channel, count(*) AS n, sum(o.total) AS revenue`,
      {});
  }
}

Two details that matter more than they look:

  • constant-arrival-rate, not constant-vus. A fixed pool of virtual users slows down when the server slows down, which hides the problem — the test paces itself to whatever the database can manage. A fixed arrival rate keeps pushing, so queueing shows up as latency, which is what production does.
  • Tag per query shape. A blended p95 across lookups and analytics is meaningless. Per-shape thresholds are what you can actually act on.

Step 4 — warm up, then measure

Every run needs a discard phase. Neo4j's page cache, the query plan cache and the JVM JIT all need a few minutes before numbers mean anything. Either run a short warm-up scenario with startTime and exclude it from thresholds, or prewarm explicitly:

CALL db.prepareForReplanning();  -- optional: clears plan cache for a clean plan-compile measurement
CALL apoc.warmup.run(true, true, true);  -- if APOC Extended is available

Also decide what you are measuring against. A load test on a laptop with a 2 GB page cache and 500 GB of data tells you about disk, not about your query. Match the test environment to production page cache and heap settings, or at least match the ratio of page cache to graph size, and say so in the report.

Step 5 — watch the server while the test runs

Client-side latency tells you something is wrong; server metrics tell you what. Keep these on a dashboard (see our observability setup) and screenshot them with each run:

SignalMetric / commandWhat a bad number means
Page cache hit rationeo4j_page_cache_hit_ratioBelow ~0.98 under read load: cache too small, or the query touches too much
Heap usage / GC pausesneo4j_vm_heap_used, GC logSawtooth to the ceiling: oversized result sets or too many concurrent transactions
Active transactionsneo4j_transaction_activeClimbing steadily: the server is queueing, you have passed capacity
Lock wait timedbms.listTransactions()Writes contending on the same nodes — see our bulk write patterns
Checkpoint durationneo4j_check_point_durationLatency spikes correlating with checkpoints: IO-bound writes
Store size / IOPShost metricsCloud disks throttle; a burst-credit cliff looks exactly like a database problem

The useful artefact from a run is not "we did 200 req/s". It is a saturation curve: repeat the run at 100, 200, 400, 800 req/s and plot p95 against rate. The knee in that curve is your capacity number, and it is the only honest input to a scaling decision.

Step 6 — make it a regression gate in CI

Once the test exists, run it on every release candidate against a seeded database — Testcontainers makes this reproducible, exactly as in our CI/CD tutorial.

# .github/workflows/perf.yml
name: perf
on: [workflow_dispatch, push]
jobs:
  loadtest:
    runs-on: ubuntu-latest
    services:
      neo4j:
        image: neo4j:2025.10-enterprise
        env:
          NEO4J_AUTH: neo4j/testtesttest
          NEO4J_ACCEPT_LICENSE_AGREEMENT: "yes"
          NEO4J_server_memory_pagecache_size: 2G
        ports: ['7474:7474', '7687:7687']
    steps:
      - uses: actions/checkout@v4
      - name: Seed graph
        run: ./scripts/seed.sh          # deterministic dataset + indexes
      - name: Warm up
        run: k6 run --quiet --duration 90s --vus 10 loadtest.js
      - name: Load test
        env:
          NEO4J_HTTP: http://localhost:7474
          NEO4J_AUTH_B64: ${{ secrets.NEO4J_AUTH_B64 }}
        run: k6 run --out json=results.json loadtest.js
      - uses: actions/upload-artifact@v4
        with: { name: k6-results, path: results.json }

k6 exits non-zero when a threshold fails, so the job fails the build on its own. Two warnings about CI perf gates:

  1. Shared runners are noisy. Absolute millisecond thresholds will flap. Either run the gate on a dedicated runner, or gate on a relative change — compare p95 against the last 10 runs on main and fail at, say, +25%.
  2. Gate on plan shape too. Cheaper and far more stable than timing: run EXPLAIN on each critical query and fail if the plan contains NodeByLabelScan or AllNodesScan where you expect an index seek. A plan regression is the cause; latency is the symptom.
# plan_guard.py — fails CI if a critical query loses its index seek
FORBIDDEN = {"AllNodesScan", "NodeByLabelScan", "CartesianProduct"}

def check(driver, name, cypher, params):
    rec = driver.execute_query("EXPLAIN " + cypher, params).summary.plan
    ops, stack = set(), [rec]
    while stack:
        n = stack.pop()
        ops.add(n["operatorType"].split("@")[0])
        stack.extend(n.get("children", []))
    bad = ops & FORBIDDEN
    assert not bad, f"{name}: forbidden operators {bad}"

What to write down

A load test that does not produce a one-page record gets re-argued every quarter. Capture: dataset size (nodes, relationships, store size on disk), server spec and memory config, Neo4j version, the workload mix, the arrival rates tested, the p50/p95/p99 per shape at each rate, and the saturation point. That page is what you compare against after the next upgrade, the next schema change, or the next ten million nodes.

Need a baseline before your next release?

GraphGuru builds workload models from real query logs, runs the saturation curves, and hands back a capacity number you can plan against — plus the CI gate that keeps it true. If you are sizing a cluster, preparing for a traffic event, or trying to settle an argument about whether the graph or the application is slow, get in touch and we will scope a short performance engagement.