+1 (415) 649-9454
GraphGuru

Serverless Graph Algorithms: Running GDS on Warehouse Data with Aura Graph Analytics

Most teams that want graph algorithms do not want a graph database. They want to know which accounts sit at the centre of a payment network, which customers cluster together, or which suppliers are single points of failure — and their data already lives in Snowflake, BigQuery, Databricks, or Postgres. Historically the answer was: stand up a Neo4j instance with the Graph Data Science plugin, build an ingest pipeline, keep it patched forever.

Aura Graph Analytics changes that trade-off. It is a serverless graph compute layer: you start a session, project a graph into memory from whatever source you have, run the GDS algorithm library, write the results back to your source, and tear the session down. No database, no plugin, no cluster to own. This tutorial walks the whole loop from Python.

When this is the right tool

Use a serverless session when:

  • The graph is analytical, not operational — you need centrality, communities, similarity, or embeddings as features, not sub-second traversals in an app.
  • Your system of record is a warehouse or relational database, and you do not want a second copy of the truth.
  • The workload is bursty: a nightly or weekly scoring job, or an exploratory project where nobody has approved a permanent cluster yet.

Stick with a Neo4j database (self-managed or AuraDB) when you need persistent storage, Cypher queries from an application, GraphRAG retrievers, or real-time lookups. Our earlier tutorials on GDS recommendation pipelines and real-time graph features for fraud models assume that persistent case; this one is the opposite end of the spectrum.

Prerequisites

  • An Aura account with API credentials (client ID and secret, created in the Aura console under API Keys).
  • Python 3.10+ and the GDS Python client:
pip install "graphdatascience>=1.13"
  • Read access to your source data. The examples below use a plain relational extract via pandas, then show the warehouse-native variant.

Set the credentials in the environment so nothing sensitive ends up in your notebook:

export AURA_API_CLIENT_ID=...
export AURA_API_CLIENT_SECRET=...

Step 1 — start a session

Sessions are sized by memory, and you pick the size (or let the client estimate it) based on node and relationship counts. Always set a time-to-live: an idle session that nobody remembers is the main way people overspend here.

import os
from datetime import timedelta
from graphdatascience.session import GdsSessions, AuraAPICredentials, AlgorithmCategory

sessions = GdsSessions(
    api_credentials=AuraAPICredentials(
        client_id=os.environ["AURA_API_CLIENT_ID"],
        client_secret=os.environ["AURA_API_CLIENT_SECRET"],
    )
)

memory = sessions.estimate(
    node_count=2_000_000,
    relationship_count=18_000_000,
    algorithm_categories=[AlgorithmCategory.CENTRALITY, AlgorithmCategory.COMMUNITY_DETECTION],
)
print("estimated session size:", memory)

gds = sessions.get_or_create(
    session_name="supplier-risk-weekly",
    memory=memory,
    ttl=timedelta(hours=2),
)

get_or_create is idempotent by name, which makes it safe to re-run from a scheduler: a retried job attaches to the existing session instead of paying for a second one. sessions.list() shows what is running — put that call in your ops runbook.

Step 2 — project a graph from tables

A graph projection is just nodes plus relationships. If your source is a warehouse table or a SQL query, load two dataframes and let the client construct the in-memory graph.

import pandas as pd

nodes = pd.DataFrame({
    "nodeId": suppliers["supplier_id"],
    "labels": "Supplier",
    "spend": suppliers["annual_spend"],
})

rels = pd.DataFrame({
    "sourceNodeId": edges["buyer_id"],
    "targetNodeId": edges["seller_id"],
    "relationshipType": "SUPPLIES",
    "weight": edges["order_value"],
})

G = gds.graph.construct("supplier-network", nodes, rels)
print(G.node_count(), G.relationship_count())

Two rules save a lot of debugging:

  1. nodeId must be an integer and globally unique across all node types. If you have string keys ("SUP-8891"), build a dense integer index first and keep the mapping — you will need it to join results back.
  2. Every endpoint in the relationship frame must exist in the node frame. Filter your edge extract with a semi-join against the node keys before constructing, or the projection fails on the first dangling reference.

If your data sits in a warehouse that Aura can read directly, the same session can pull it server-side instead of round-tripping through your laptop — the projection API takes a source query rather than dataframes. That is the variant you want for anything above a few million relationships, because the client-side path is bounded by your own network.

Step 3 — run the algorithms

From here the API is the GDS you already know. Estimate first, run in stats or stream mode while you are exploring, and only mutate the in-memory graph once you like the numbers.

# Who is structurally central in the supply network?
pr = gds.pageRank.mutate(G, mutateProperty="pagerank", relationshipWeightProperty="weight")
print(pr["nodePropertiesWritten"], pr["ranIterations"])

# Which suppliers cluster into the same commercial community?
comm = gds.louvain.mutate(G, mutateProperty="communityId", relationshipWeightProperty="weight")
print("communities:", comm["communityCount"], "modularity:", round(comm["modularity"], 3))

# Feature vectors for a downstream model
gds.fastRP.mutate(G, mutateProperty="embedding", embeddingDimension=128, randomSeed=7)

results = gds.graph.nodeProperties.stream(
    G,
    node_properties=["pagerank", "communityId", "embedding"],
    separate_property_columns=True,
)

A few things worth knowing before you interpret any of this:

  • Weighted PageRank needs a sane weight. Order value spans several orders of magnitude; log-scale it in the projection unless you want your top-ten list to be the ten largest invoices.
  • Louvain is non-deterministic in its tie-breaking. For reproducible community IDs across runs, set randomSeed and concurrency=1, or accept that IDs are labels, not identities, and compare community membership overlap between runs instead.
  • Check modularity before you present clusters. Below roughly 0.3 you are usually looking at a graph without real community structure, and the pretty picture will not survive a business review.

Step 4 — write results back where they belong

The point of the exercise is a column in your warehouse, not a dataframe in a notebook. Stream the properties out and load them like any other model output:

out = results[["nodeId", "pagerank", "communityId"]].rename(columns={"nodeId": "supplier_key"})
out["scored_at"] = pd.Timestamp.utcnow()
out.to_sql("supplier_graph_features", warehouse_engine, if_exists="replace", index=False)

Join back on the integer index you built in step 2, and keep scored_at — graph features drift, and the first question a data scientist will ask is how old the score is.

Step 5 — always tear the session down

gds.delete()          # or: sessions.delete(session_name="supplier-risk-weekly")
print(sessions.list())

Wrap the whole job in a try/finally so a failed algorithm cannot leave a paid session running until its TTL expires. Belt and braces: keep the TTL short (an hour or two more than the job's worst observed runtime) and delete explicitly.

Putting it in a schedule

A production version of this job is about forty lines: estimate, get_or_create, project from a source query, run two or three algorithms, stream, load, delete. Points to harden before you call it done:

  • Idempotency. Name sessions after the job, not the run, and make the warehouse load a replace-or-merge on a run key.
  • Cost visibility. Log session size and wall-clock duration for every run; serverless bills for time, so a query that quietly doubles in cost is otherwise invisible.
  • Data contract. Pin the source query behind a view. When someone renames a column upstream, you want a failed view, not a silently smaller graph.
  • Validation. Assert on node and relationship counts after projection, and fail the run if either moves more than a few percent from the last run. Most "the algorithm broke" incidents are actually broken extracts.

Where this leaves you

Serverless graph analytics removes the one objection that used to kill graph projects before they started: nobody had to adopt a new database to find out whether graph features were worth anything. You can prove the value on your existing warehouse in an afternoon, and only take on a persistent graph once an application needs traversals rather than scores.

If you are weighing a serverless session against a managed AuraDB instance, or you want a second pair of eyes on a projection that keeps running out of memory, our team does this work weekly — see Neo4j Aura & managed cloud consulting or get in touch with a note about your data volumes and the questions you are trying to answer.