ReliaNet: Distributed Truth-Seeking Network
A 5-node distributed key-value store built for disaster response - staying consistent and available even through a 40% simultaneous node loss.
Why it exists
The problem
Most databases live on one server, or one data center. Destroy, flood, or cut the connection to that one place, and the whole application goes dark - taking every emergency alert and news update with it, right when they matter most.
The fix
Spread the same data across 5 independent servers with no single "master." Losing any one of them - or even two at once - never takes the whole system down.
The five moving parts
API GatewayFastAPI
The single entry point for every read and write. Instead of a hardcoded server address, it asks the Registry which nodes are healthy right now and routes there.
RegistryService discovery
The cluster's heartbeat monitor - sweeps every node over TCP, and evicts anything that stops responding before the gateway can route traffic to it.
5 Peer NodesgRPC
The actual database. Nodes talk to each other directly over gRPC to replicate writes and vote on reads, using strictly defined Protocol Buffers.
Anti-EntropyRabbitMQ
The background repair crew. A recovering node broadcasts a hash of its data over a message queue so healthy peers can detect drift and resync it - without touching live traffic.
DashboardStreamlit
A real-time control room for the cluster - write data, run quorum reads, and watch replication, latency, and failures happen live.
By the numbers
5
Nodes in the cluster
40%
Node loss survived live
1.5s
Before a slow node gets skipped
~12s
For a recovered node to catch up
How it works
1. A write comes in
The gateway checks which nodes are currently healthy and routes the request to one of them - never a hardcoded server.
2. It's copied everywhere, instantly
That node immediately pushes the data to the other four - like 5 people writing the same note in 5 separate notebooks at once.
3. Reads need a majority
Asking one notebook isn't enough - a read only counts once most of the 5 nodes agree on the answer.
4. A dead node gets dropped
If a node stops responding, it's evicted from the cluster and "majority" recalculates on the fly - 5 becomes 3, then 2.
5. It heals itself
A node that comes back online compares notes with the others and automatically catches itself back up.
Verified by breaking it on purpose
chaos-tests.log
$ docker stop node-1 # kills the entry node mid-write
→ traffic rerouted automatically — zero downtime
$ make stop4and5 # kills 2 of 5 nodes at once (40% of the cluster)
→ quorum recalculated 3 → 2 — reads/writes kept working
# inject 3s of lag on node-3
→ the 1.5s gRPC timeout skipped it — rest of the cluster unaffected
# restart a node holding stale data
→ self-healed and rejoined the quorum in ~12s
The team & what's next
Built from scratch by a team of four for COE 892 - Distributed Cloud Computing, prioritizing consistency and partition tolerance under the CAP theorem. Next up: consistent-hashing-based sharding so large datasets can be split across nodes instead of fully replicated to each one, and a move from Docker Compose to Kubernetes for automated recovery and multi-region deployment.