Spanner, CockroachDB, and FoundationDB — extended reference

This reference preserves the earlier detailed chapter and its citations. Start with the guided explanation, then use these sections for additional mechanisms and qualifications. References to the original lesson sequence are historical.

Part 8: Architectural case studies and synthesis

Prerequisites: Lesson 45 and the preceding foundations.

Learning outcomes

• Compare Spanner, CockroachDB, and FoundationDB through distributed transaction responsibilities.
• Explain timestamp uncertainty, replicated ranges, and separated transaction/storage roles within their documented scope.
• Identify which locality and coordination costs remain after local execution improves.

A correct transfer travels farther than its data lookup

A company has customers in three regions. Each customer normally changes records owned by one region, but a transfer can involve two customers with different owners. The requirement is precise: a successful transfer must not lose or create money, a completed transfer must be visible to later appropriately consistent reads, and the result must survive the stated regional failure policy.

A local index lookup may take microseconds while the commit path crosses continents. Choosing a faster index can improve local work without materially changing the network-bound critical path. This lesson compares architectural mechanisms rather than trying to rank distributed databases by one latency number.

Spanner, CockroachDB, and FoundationDB demonstrate different ways to organize ordering, replication, transaction validation, storage, and application interfaces. Their common concern is distributed transactional correctness. Their differences include how responsibilities are bundled, what applications directly manipulate, and which paths can remain local. Product details evolve; the cited papers are architectural snapshots, while documentation describes the currently published implementation.

Establish the comparison contract

A meaningful comparison starts with the operation and failure model. “Transfer two balances” is incomplete unless it states whether the balances share a partition, whether stale reads are allowed, which replicas must persist the result, and what happens when the client loses its reply.

For this lesson’s hypothetical operation, the business invariant is conservation of funds. The application atomically debits A, credits B, and records operation identifier X. A later retry of X must retrieve its original result rather than transfer twice. The deployment requires durable replicated commit across the configured failure domains. External payment networks are outside this transaction and would need a separate integration contract.

The read contract matters as much as the write contract. A follower read from an earlier timestamp can correctly return old data under a historical-read interface. It would violate an interface promising that a read beginning after a successful write sees that write. Both results can be valid in different APIs; combining their latency measurements without naming the guarantee is misleading.

The physical placement also matters. A three-replica group contained in one metropolitan region does not have the same disaster tolerance or communication path as replicas spread across continents. A benchmark that changes placement while comparing products changes the workload’s failure contract.

Decompose replication from atomic commitment

Replication establishes agreement about the history of a replicated unit. A distributed transaction establishes one atomic outcome across its participants. Those are related but separate jobs. If a transaction modifies three replicated ranges, each range’s consensus history does not automatically guarantee that all three ranges decide the same transaction outcome.

A conceptual design layers them:

business transaction X
    -> participant range A -> replicated log A
    -> participant range B -> replicated log B
    -> durable transaction outcome / coordination

Figure: Replicated participants still need transaction-level coordination. The drawing is a conceptual decomposition, not a literal process diagram for any one product.

Replication can make a participant’s prepared state durable and recoverable. The transaction protocol then determines whether that participant’s changes commit or abort. When the coordinating process fails, another process must discover the authoritative outcome rather than guess based on its own timeout.

The cost depends on dependency structure. Independent participant work can proceed in parallel, making the slowest path important. A later phase that requires all earlier results remains sequential. Multiplying the number of participants by one round-trip time can overestimate parallel work; ignoring the coordination barrier can underestimate it.

Spanner: bounded uncertainty and externally ordered transactions

The 2012 Spanner paper describes a globally distributed database using Paxos replication, multiversion data, distributed transactions, and TrueTime, an interface exposing clock uncertainty. Its external-consistency mechanism includes choosing timestamps and waiting as necessary so the uncertainty interval no longer permits an ordering violation. The paper is a historical design account, not a complete description of every present service feature. Spanner paper

The conceptual insight is to expose uncertainty rather than pretending wall clocks are exact. A clock reading can be modeled as an interval containing real time under specified assumptions. If a transaction claims timestamp T, a protocol may need to wait until it can safely establish that T is in the past before reporting completion under the desired ordering contract.

This waiting is not an arbitrary performance penalty. It discharges a proof obligation. Removing it while retaining the same timestamp rules can invalidate the real-time ordering guarantee. Another architecture can use a different ordering mechanism; it must still establish the equivalent property if it claims the same contract.

Timestamped versions also support consistent reads at an explicit point in history. A read may avoid interfering with current writes if the required version is retained and safely available. However, serving a historical read is different from promising the most recent value. Choosing a timestamp is a semantic decision before it is an optimization.

A timestamp scenario without magical clocks

Consider an illustrative service with a time estimate interval [100, 108] milliseconds on some abstract reference timeline. The interval expresses uncertainty, not elapsed duration of one request. If a commit uses timestamp 106, the system cannot immediately claim that the timestamp is definitely in the past while the lower bound remains 100. Its protocol must establish the necessary ordering condition through waiting or another permitted dependency.

This example is intentionally generic. It does not specify Spanner’s complete timestamp-selection algorithm or an actual uncertainty measurement. The arithmetic only illustrates why uncertainty can become a latency term.

Suppose local execution costs 3 ms, required replication costs 40 ms, and an additional ordering wait costs 2 ms on a sequential hypothetical path. Total is at least 45 ms before other stages. Reducing local execution by 2 ms changes that part to 43 ms. It does not eliminate the 40 ms communication dependency. If waits overlap in a real implementation, summing them would overstate the total; a trace should show actual dependency edges.

The architectural lesson is that clock quality, placement, and transaction topology can affect latency independently of local data structures. An index optimization and a region-placement optimization act on different terms of the budget.

CockroachDB: SQL over replicated ranges and distributed transactions

CockroachDB documents a layered architecture with SQL, transactions, distribution, replication, and storage responsibilities. Data is divided into ranges; range replication uses Raft. The transaction layer coordinates transactional state over distributed keys, with intents and transaction records among the documented mechanisms. Current isolation options and optimizations must be checked for the deployed release rather than inferred from an old summary. Transaction layer, Replication layer

The design demonstrates how a SQL interface can sit above a distributed key space. A logical row and its indexes translate into physical keys, and one SQL statement can touch several ranges. Schema design therefore influences distributed work even when the application never names a range directly.

An index is a useful example. Adding an index can make reads faster while adding transactional writes at another key position. If the new index entries lie in other ranges, one row modification can acquire more distributed participants. The application sees a declarative schema change; the engine sees more maintained state and potentially more coordination.

Range placement and lease ownership affect where requests go. Routing a client to a nearby SQL gateway does not guarantee that authoritative data access is nearby. The gateway can accept the connection locally while the request still travels to the range owner. Connection locality and data locality are separate facts.

Transaction conflicts become application-visible cost

A distributed transaction may need to retry when concurrent activity invalidates its assumptions or ordering. Retrying is compatible with correctness when the database repeats transactional computation safely. It remains a resource cost and can become a latency problem under contention.

Imagine 20 clients repeatedly updating one inventory row. An optimistic attempt reads stock 1, prepares to decrement it, and races with others. Only one can successfully acquire the final item under a correct invariant. The remaining attempts should return sold-out or otherwise resolve according to their transaction logic. Repeatedly retrying a business failure is wasted work.

A second scenario has stock 10,000 but all buyers also update one global statistics row. That unnecessary shared row can introduce conflicts unrelated to stock availability. Moving approximate statistics to an asynchronous derived view can reduce contention while preserving exact stock. Moving the stock itself asynchronously would weaken the invariant. Distinguishing essential coordination from accidental coordination is a major schema optimization.

The correct metric is useful committed business operations, including retries’ resource cost. A system showing high internal attempt throughput can be doing more work without serving more customers. Track attempts per commit, conflict reasons, transaction duration, and the keys or ranges responsible.

FoundationDB: a transactional substrate with layers above

The 2021 FoundationDB paper describes an ordered transactional key-value system with separated transaction-processing, log, and storage roles, optimistic conflict detection, multiversion reads, and extensive deterministic simulation testing. Its small core supports richer data models built as layers. This is a historical architecture account; deployment limits and APIs require current documentation. FoundationDB paper

The separation changes where an application receives convenience. A SQL database directly supplies parsing, planning, relational constraints, and index management. A transactional key-value substrate can provide a strong foundation while a layer implements those higher-level responsibilities. A smaller core does not mean those responsibilities vanish; it changes their owner.

For an application team building a specialized storage service, that can be useful. It can control key layout and maintain custom indexes in the same transaction as primary records. It also becomes responsible for correctly encoding order, preventing key collisions, maintaining index invariants, handling retries, and evolving formats.

The relevant tradeoff is engineering responsibility versus interface specialization. Using a lower-level API solely because it appears faster can create a large correctness and maintenance burden if the application simply needs ordinary relational behavior.

A layered index trace

Consider a hypothetical ordered key-value layer storing players and an email lookup. It uses namespaced keys so record and index ranges do not collide:

("player", player_id) -> serialized player
("email", normalized_email) -> player_id

To change an email, the transaction reads the current player, checks the new email mapping, removes the old index entry, writes the new mapping, and updates the record. The uniqueness check must participate in conflict detection even when the new key is absent. Otherwise two concurrent transactions can both conclude that the email is free.

transaction:
    player = read(player_key(id))
    require read(email_key(new_email)) is absent
    delete email_key(player.old_email)
    write email_key(new_email) = id
    write player_key(id) = player with updated email
commit or retry under the database's transaction rules

This pseudocode omits encoding and API details deliberately, but not the correctness boundary: every related read and write belongs in one protected transaction. A retry must recompute from the new snapshot. It must not send a welcome email each time the transaction body executes.

The same principle holds in SQL through constraints and transactional index maintenance. A layered approach exposes the mechanism to the developer; a relational engine supplies more of it automatically.

Workload locality determines favorable placement

Consider a hypothetical multitenant service with 10,000 tenants. Ninety-eight percent of operations access one tenant, while two percent perform global reporting. Placing each tenant’s transactional data together can keep most work local to a bounded set of ranges. Reporting can use a separate analytical representation with an explicit freshness contract.

Now change the workload: every transaction transfers funds between randomly selected tenants worldwide. Tenant placement no longer makes the common transaction local. Distributed coordination becomes central. The same key layout that suited independent updates may now produce frequent cross-region transactions.

Neither workload is inherently more “distributed.” They impose different dependency graphs. Partitioning success depends on whether the partition boundary aligns with the application’s invariants and access patterns. A topology that scores well on total storage capacity can still perform poorly on the common transaction.

Read placement creates another choice. Historical or bounded-stale reads can serve near users under an appropriate contract. Strong current reads may need communication with authority or proof that a replica is sufficiently current. Making the interface explicit lets the application choose where freshness is essential and where it is unnecessary.

A mechanism comparison without a benchmark ranking

Table: Each row names documented mechanisms at its cited scope. It does not assert identical APIs, failure configurations, or performance.

The comparison asks who supplies query semantics, index maintenance, transaction recovery, and placement. It deliberately leaves latency to an equal-contract workload evaluation. Historical papers explain architectural ideas; current documentation and release behavior determine what a deployment actually offers.

A numerical path comparison

Assume a hypothetical operation with 5 ms local execution and a mandatory 70 ms remote quorum path. Ignore other costs for the moment. Its critical path is at least 75 ms if these are sequential. Making local execution instantaneous can save at most 5 ms, or about 5 / 75 = 6.67% of this simplified total.

Now suppose moving authority and a sufficient quorum nearer the client reduces the required path to 8 ms while preserving the chosen failure contract. The simplified total becomes 13 ms. This is a much larger improvement, but it may change which region can fail without interrupting writes. Geography is not a free configuration trick: placement determines both latency and survivability.

A second transaction contacts three participants in parallel with round-trip times 10, 25, and 70 ms. If it needs all responses, that stage takes at least about 70 ms rather than 105 ms. A subsequent 70 ms decision round can bring the communication component near 140 ms. Combining parallel and sequential stages correctly is essential.

These are hypothetical budget calculations, not measurements of the named systems. Their purpose is to show why a comparable benchmark must record transaction shape, placement, replication policy, read guarantees, and retry behavior.

Failure analysis through an uncertain outcome

A client submits transfer X. Participants prepare, the authoritative outcome becomes committed, and the client connection disappears before the response arrives. The client cannot infer abort from its timeout. Retrying X with a stable identity allows the application to recover the committed result.

If the coordinator instead fails before a durable decision, the protocol must resolve according to its recovery rules. A participant that has promised not to unilaterally change prepared data cannot simply abort because a timer expires unless the protocol permits that transition. This is the difference between an application deadline and an atomic-commit decision.

The storage layer also matters. A node can have a log record in memory without a durable copy, or a replicated durable record without having applied it to a query-serving structure. Recovery and serving protocols establish which state is authoritative. Observing one process’s local buffer does not establish the cluster-wide outcome.

A useful incident record therefore tracks operation ID, transaction ID where available, participant state, leadership epoch, durable decision, and client acknowledgment separately. Collapsing them into a single “success” bit makes diagnosis and safe retries harder.

The cost of a global namespace

A distributed key space makes data addressing convenient, but global services inside it can become bottlenecks. Metadata, timestamp allocation, configuration changes, transaction conflict processing, and routing caches each have capacity and failure behavior.

A design that separates roles can scale them independently within its architectural limits. It also requires observability across those roles. A storage node with spare bandwidth does not help if transaction coordination is saturated. Adding coordinators does not necessarily help if every transaction conflicts on one logical key.

A thought experiment makes this concrete. Suppose 90% of requests operate independently, but every transaction obtains a globally serialized audit sequence. If sequence allocation costs 50 microseconds of exclusive service per transaction, that mechanism alone has an idealized ceiling of 20,000 allocations/s. More storage partitions do not remove it. A partitioned audit identifier can improve capacity only if the application does not require one total order at allocation time.

This is a general architectural analysis, not a claim that any named product uses that exact mechanism. It illustrates how apparently minor application requirements can introduce global coordination into an otherwise partitionable workload.

Simulation and evidence

Distributed correctness is hard to establish from normal-operation tests because failures must occur at particular instruction and message boundaries. Deterministic simulation allows controlled schedules, network faults, and repeatable execution. FoundationDB’s paper describes simulation as a core development technique rather than a final production benchmark.

A comparative evaluation should combine mechanism understanding with workload evidence. Test hot-key transactions, independent tenant updates, multi-range operations, historical reads, failover, and recovery. Record which attempts are retried and whether acknowledgments meet the selected durability contract. A fast successful run without failures says little about ambiguous outcomes during leadership change.

Product choice should also include the work above the engine. A team needing SQL and mature query tooling may value a bundled interface. A team constructing a specialized service may value a transactional substrate. A team needing wide geographic transactions must budget communication regardless of interface preference.

An architectural decision record for the transfer service

A decision record can preserve the reasoning that a benchmark alone cannot. For the hypothetical transfer service, begin with four facts: account balances are exact, transfers may cross regional ownership, retries retain operation identity, and reporting can lag. Those facts immediately separate the transactional path from the reporting path.

The transactional path needs protected reads and writes for both balances and the operation record. It needs a durable outcome under the selected failure domains. Its latency target must include the longest mandatory geographic dependency. If that budget is unacceptable, the team must reconsider placement, reduce cross-region transactions through ownership rules, or change the business protocol. Swapping a local index cannot resolve a physically impossible latency requirement.

The reporting path can consume committed changes into a regional analytical copy. A dashboard can display the source position and lag so readers understand its freshness. That separation removes large scans from the operational transaction resource while introducing a new obligation: the change pipeline must handle duplication, retention, reprocessing, and schema changes. The architecture saves one kind of interference by accepting another maintained representation.

A suitable comparison then asks each candidate engine to implement the same contract. Where the engine supplies a declarative constraint, use it. Where a layer must implement the invariant, include that layer’s code and maintenance cost. Compare realistic retry behavior and failure recovery rather than excluding inconvenient components from one side.

Operational fit is also a mechanism question. Who can inspect a blocked transaction? How is a misrouted hot range discovered? Which configuration controls write survival across a region loss? How are old versions retained for reads and backups? These questions do not rank products universally; they identify whether the chosen team can safely operate the selected architecture.

Finally, record rejected assumptions. If the design assumes that every tenant is independent, say which future feature would invalidate that assumption. Cross-tenant transfers, global uniqueness, and globally ordered feeds can turn local operations into coordinated ones. Making those triggers explicit helps a future engineer recognize when an optimization’s original justification no longer holds.

The result is a durable explanation of why an engine and topology were selected. It connects business correctness to implementation mechanics, performance budgets, and operational ownership. That connection is the central skill developed throughout the course.

Synthesis

These systems show that distributed transaction design is a composition of ordering, replication, concurrency control, placement, and recovery. Their mechanisms cannot be reduced to “uses consensus” or “supports ACID.” Those labels leave the critical path and failure handling unspecified.

The strongest architectural question is concrete: for this transaction, under this placement and failure contract, which process knows what, which durable history proves it, and which messages must arrive before success? Answering that question provides a better basis for latency optimization than comparing isolated headline throughput figures.

Sources and further reading

• Spanner original paper, Google, 2012
• FoundationDB architecture paper, 2021
• FoundationDB 8.0 developer guide
• FoundationDB 8.0 architecture
• CockroachDB transaction layer
• CockroachDB replication layer

Course index · Lesson plan