Documentation

Clustering & High Availability

JetGraph supports primary–standby clustering for high availability and read scale-out. The primary serves reads and writes; one or more standbys keep a live in-memory replica of the graph and serve reads only. Replication is delta-based and lag is typically under 35 seconds under normal ingest.

Architecture

A cluster is built from three small components running alongside each graphengine process:

🖥️

graphengine (primary)

Accepts ingest. Writes a full snapshot nightly and a delta file every 30 seconds.

📤

delta-replicator

Sidecar on the primary. Ships each new snapshot and delta to the standby via rsync over SSH within a few seconds.

🧩

delta-compactor

Sidecar on both nodes. Merges long chains of raw deltas into checkpoints so restart replay stays fast.

📥

graphengine (standby)

Runs with STANDBY_MODE=true. Polls for new delta files every 5 s and applies them to the live graph. Hot-reloads on new nightly snapshots without restarting.

┌────────────── PRIMARY ──────────────┐        ┌────────────── STANDBY ──────────────┐
│ graphengine (read + write)          │        │ graphengine   STANDBY_MODE=true     │
│   writes snapshot-*.bin  delta-*.bin│        │   applies deltas, hot-reloads       │
│ delta-replicator ───── rsync/ssh ────────────▶ /opt/graphengine/data/snapshots    │
│ delta-compactor                     │        │ delta-compactor                     │
└─────────────────────────────────────┘        └─────────────────────────────────────┘
             ≤ 30 s produce + ≈ 5 s ship   =   ≤ 35 s replication lag

What Gets Replicated

Schema — snapshots include the finalized schema; the standby reloads automatically when a new snapshot arrives.
Nodes and edges — every CREATE, MERGE, SET, DELETE, and graph.upsertEdge write produces dirty pairs that flow in the next delta file.
Edge state — velocity counters, activity bitmaps, and histograms are snapshot-bound and travel with the data.
Fraud cases — graph.createFraudCase / graph.removeFraudCaseNode propagate through the normal delta channel.

Segment Evaluator and Pattern Miner sidecars maintain their own config stores; they should run on both hosts but do not participate in graph replication.

Replication Lag

Lag = delta_interval_secs (30 s default) + POLL_INTERVAL_SECS (5 s default) = ~35 seconds upper bound under normal load. For tighter RPO, lower both intervals at the cost of more but smaller delta files. Data loss on an unplanned primary failure is bounded by this window.

Minimum Compose Setup

Clusters use two Compose files. The primary runs the engine plus the replicator sidecar; the standby runs the engine with STANDBY_MODE enabled.

yaml — primary (key services)

services:
  graphengine:
    image: ghcr.io/fraudmanagement/graphengine:main
    environment:
      ENABLE_ADMIN_RESET: "true"
      CONFIG_PATH: /config/config.toml
      MALLOC_CONF: "background_thread:true,dirty_decay_ms:1000,muzzy_decay_ms:10000,narenas:16"
    volumes:
      - /opt/graphengine/data:/data
      - ./config.toml:/config/config.toml:ro

  delta-replicator:
    image: ghcr.io/fraudmanagement/graphengine:main
    command: ["/usr/local/bin/delta-replicator"]
    environment:
      SNAPSHOT_DIR: /data/snapshots
      REPLICA_REMOTE: "${STANDBY_USER}@${STANDBY_HOST}:/opt/graphengine/data/snapshots/"
      REPLICA_SSH_KEY: /keys/replication.key
    volumes:
      - /opt/graphengine/data:/data
      - ./replication.key:/keys/replication.key:ro

yaml — standby (key service)

services:
  graphengine:
    image: ghcr.io/fraudmanagement/graphengine:main
    environment:
      STANDBY_MODE: "true"
      POLL_INTERVAL_SECS: "5"
      CONFIG_PATH: /config/config.toml
    volumes:
      - /opt/graphengine/data:/data
      - ./config.toml:/config/config.toml:ro

ℹ️

Port 22 (SSH) on the standby host must be reachable from the primary host — delta-replicator rsyncs over SSH directly, not through the Docker network.

Failover

1

Drain and stop the primary

Stop new writes at the load-balancer level, then docker compose stop graphengine. On SIGTERM the engine writes a final delta and full snapshot — up to stop_grace_period (default 300 s).
2

Wait for the standby to drain the delta queue

Tail the standby logs until no more applying delta lines appear. At that point the two engines are byte-for-byte equivalent.
3

Promote the standby

Restart the engine with STANDBY_MODE removed (or swap in the primary compose file on that host). The data directory is already a full mirror — no migration step.
4

Cut clients over

Update DNS or load-balancer config. Optionally reconfigure the old primary as the new standby — point its STANDBY_HOST at the new primary and install the new replication key.

For an unplanned failover (primary host lost), promote the standby immediately. Data loss is bounded by the last delta the standby applied — at most ≈ 35 seconds.

Verifying Replication

bash

# Compare node counts on both endpoints — should match or differ by one delta
curl -sS -X POST http://PRIMARY:8080/cypher \
  -H 'Content-Type: application/json' \
  -d '{"query":"CALL db.nodeStats() YIELD type, count RETURN type, count"}'

curl -sS -X POST http://STANDBY:8080/cypher \
  -H 'Content-Type: application/json' \
  -d '{"query":"CALL db.nodeStats() YIELD type, count RETURN type, count"}'

# Watch the replicator ship files and the standby apply them
docker logs -f delta-replicator | grep shipping
docker logs -f graphengine     | grep "applying delta"

Operational Limits

Standbys are read-only. All writes must go to the primary; any write attempted on a standby returns an error.
Standbys must not evict independently. Set pressure_eviction_batch_size = 0 on standbys so the replica only deletes edges that arrive in the primary's delta stream.
Size the standby like the primary. It holds the same in-memory graph — size mem_limit identically and use the same (or slightly lower) memory_limit_bytes.
No cascading replication. Topology is primary-to-standby. Chained standbys are not supported.

📖

For step-by-step production setup — SSH key generation, full memory sizing tables up to 300 GiB, and every troubleshooting playbook — see the Clustering Administrator Guide shipped with the repository at docs/clustering-admin-guide.md.

Clustering & High Availability