Cloudflare Meerkat: A New Approach to Global Consensus Without Leaders

Cloudflare Research published details today on Meerkat, an experimental distributed consensus service built to manage control-plane state across their 330+ data centers. The interesting part isn't just another consensus implementation - it's that Meerkat uses QuePaxa, a 2023 algorithm that takes a fundamentally different approach to the timeout and leader problems that plague Raft deployments in wide-area networks.

The core problem Cloudflare is solving: "many internal services need to read and modify the same control-plane state from across our 330+ global data centers" while ensuring "different readers never see inconsistent state." Traditional consensus algorithms struggle here because of how they handle network partitions and leader failures.

Why Not Just Use Raft?

Raft works well in controlled environments, but it has structural limitations for global deployments. The leader-based design means:

1. Single point of failure: All writes route through one node. If the leader fails, the cluster halts until a new leader is elected.
2. Election storms: Under adverse network conditions, repeated leader elections can cause availability degradation.
3. Timeout sensitivity: The algorithm relies on timeouts to detect failures and trigger elections. Tuning these for global networks is notoriously difficult.

Anyone who has fought a Raft cluster on a bad network knows the pain. Leaders flapping, elections storming, latency spiking - these aren't edge cases at Cloudflare's scale.

QuePaxa: Consensus Without Timeouts

QuePaxa, published at SOSP 2023 by researchers at EPFL, takes a different approach. Rather than electing a leader and having all writes flow through it, QuePaxa allows any replica to drive consensus at any time.

The key insight: when multiple replicas propose concurrently, their proposals "constructively interfere" rather than clash. There's no leader election because there's no leader. The system makes progress even under wild fluctuations in message delay.

Cloudflare claims this gives them "approximately 10x higher throughput than Raft and Multi-Paxos during" challenging network conditions. That's a significant difference for a global control plane.

What HN Is Saying

The Hacker News discussion surfaced several thoughtful critiques of both the article and the algorithm:

On the article structure: Multiple commenters found the post confusing because it spent too much time comparing to Raft when the real comparison should be to leaderless Paxos variants. "Given Cloudflare's requirements (e.g. no strong leaders), it immediately seems like they should be comparing to Paxos-class algorithms," one commenter noted. "Comparing to Raft and saying it's better because Meerkat is leaderless is confusing."

On implementation complexity: One commenter observed that QuePaxa "looks even trickier to implement than Paxos (already a notoriously tricky algorithm)" and noted a subtle failure mode: "very long tail latencies. In Paxos/Raft the latencies are more likely bounded by the timeouts... but in this case, you may write something, wait for an ack, then abandon and retry, then realize the old write succeeded."

On the scope of the announcement: Several commenters noted the service isn't in production yet. "Maybe the blog post is just premature. It would be much more valuable if they posted it after actually having run it in production and validated the strengths and weaknesses with real world data."

On when you actually need this: A pragmatic take emerged: "Most 'we need distributed coordination' turns out to be 'we need one writer and a lock,' which a single Postgres hands you for free: advisory locks, SELECT ... FOR UPDATE, SKIP LOCKED for work distribution. Linearizability without running Raft."

On the innovation culture question: One comment sparked a philosophical thread about whether there's an "almost anti-innovation attitude" in tech today. "I'm not saying to hand roll a consensus algorithm at your next startup. But there's definitely a vibe these days that any sort of theoretical, creative, or innovative thinking is suspect. Get back to selling ads!"

The Technical Architecture

Meerkat maintains a distributed log of events across replicas. Each slot in the log can contain a decided value, with the critical invariant that "no two replicas will ever disagree on the value of a decided slot."

The log-based design provides linearizability - "all reads after a write will see that write" - even when requests target different replicas globally distributed. Consensus ensures slot agreement across a majority of replicas before proceeding.

The latency fundamentals are honest: proposal decision requires 1-3+ round trips between proposers and replica majorities, with costs proportional to geographical distances. Meerkat doesn't eliminate physics.

Optimization strategies include:

• Batching multiple writes into single proposals
• Reading stale (but consistent) data from local replicas
• Transactional operations like compare-and-swap
• Allowing developers to optimize replica placement for reduced latency

Where It Fits

Cloudflare is explicit about the use cases: "perfect for control plane information that is written infrequently but must remain consistent." Examples include leadership information for replicated databases and placement information for resources like AI model instances.

Just as explicitly, they note it's unsuitable for general-purpose databases requiring high-frequency operations. This is control-plane infrastructure, not a replacement for your application database.

The current status: "not deployed to production, but we have run multiple proofs-of-concept with up to 50 replicas distributed around the world, to great success. Leaders in our proof-of-concept clusters constantly fail, and the cluster keeps operating with no increase in error-rate."

The Asynchronous Advantage

What makes this academically interesting: Meerkat would be the first production implementation of an asynchronous consensus algorithm.

Paxos, Raft, and most deployed consensus systems are partially synchronous - they rely on timeouts and only make progress if message delay is sufficiently small compared to timeout durations. QuePaxa doesn't rely on timeouts and makes progress even under extreme delay variations.

Historically, asynchronous protocols weren't used because performance wasn't competitive in the normal case when message delays are small and predictable. The question is whether QuePaxa has finally crossed that threshold.

Why This Matters

If Meerkat succeeds in production, it could influence how we think about consensus for globally distributed systems. The problems with Raft in wide-area deployments are well-documented, and most solutions involve carefully tuning timeouts and hoping for the best.

A timeout-free approach that maintains competitive performance would be genuinely useful for:

• Kubernetes control planes (etcd uses Raft and is often a scaling bottleneck)
• Global configuration distribution
• Multi-region database leadership coordination
• Any control plane that needs consistency across disparate network conditions

Several commenters noted they'd love to see this open-sourced as a building block for other globally distributed services. Cloudflare hasn't announced plans on that front.

For now, this is a research project that shows promise. The proof-of-concept results are encouraging, but production validation will tell the real story. Consensus algorithms have a long history of working beautifully in papers and failing in surprising ways under real load.

Sources

• Cloudflare Meerkat Introduction
• Hacker News Discussion
• QuePaxa Paper (SOSP 2023)