RocksDB and Cassandra — extended reference

This reference preserves the earlier detailed chapter and its citations. Start with the guided explanation, then use these sections for additional mechanisms and qualifications. References to the original lesson sequence are historical.

Part 8: Architectural case studies and synthesis

Prerequisites: Lesson 43 and the preceding foundations.

Learning outcomes

• Separate the responsibilities of an embedded RocksDB engine and a Cassandra deployment.
• Explain LSM amplification, compaction, partition-key modeling, replication, and repair.
• Compare durability and convergence contracts before interpreting throughput claims.

A durable key-value library is not a cluster

An engineer selects an embedded storage library, adds an HTTP endpoint, and starts three copies for redundancy. Each copy now has durable local data. They still need a protocol to decide ownership, replicate accepted changes, route requests, handle conflicts, and recover after partitions. Local persistence solved one layer of the problem.

RocksDB and Cassandra make a useful comparison because they occupy different architectural scopes. RocksDB is an embedded storage-engine library. Cassandra is a distributed database with its own data model, placement, replication, and operational protocols. Similar storage ideas do not make their responsibilities interchangeable.

The storage-engine layer

RocksDB’s overview and compaction documentation describe an LSM-oriented local storage engine. It provides a reusable representation and access layer that a larger system can embed. Its behavior depends on configuration, APIs, and version; a hosting database still chooses higher-level transaction and cluster semantics.

The architectural consequence is composability. A product can reuse a mature local engine while specializing routing, SQL, replication, or application logic. It also inherits the need to budget compaction, cache, log, and device resources. A fast library call is only one stage in a remote client’s request.

A host must connect storage outcomes to its own acknowledgment protocol. If it replies before the required local persistence or replication boundary, the library cannot retroactively strengthen the promise. If it retries a logical increment after an uncertain result without idempotency, durable local storage does not prevent duplicate business effects.

The distributed-database layer

Cassandra’s architecture documentation describes partitioned distribution, replication, and a local storage layer. Its data-modeling guidance is query-driven, with tables designed around access patterns. The documented distributed responsibilities include convergence and repair beyond ordinary local file maintenance.

These facts motivate a different application interface from unrestricted relational joins. Keeping queries aligned with partition keys can make work predictable. Duplicating a fact across query-specific tables can improve selected reads while introducing maintenance and consistency responsibilities under the product’s supported operations.

The case analysis should distinguish ordinary writes, conditional transaction facilities, consistency settings, and versions. A blanket statement that the entire product is either weakly consistent or globally transactional would erase important operation-level differences.

Scenario A: a custom transactional service

Suppose a team needs a specialized service with a narrow key-value interface, strong ownership rules, and custom replication. Embedding a storage engine can reduce the amount of local indexing and compaction code the team must write. It does not remove the need for a transaction model if several keys participate in one invariant.

The team must define what the replicated log contains: logical commands, storage mutations, or another representation. It must define snapshot installation, metadata ownership, membership changes, and how storage errors affect client results. It also needs an operational path for backups and repairs.

This can be a rational architecture when the product’s constraints justify specialized engineering. It becomes risky when the team assumes a few network handlers and heartbeat checks are enough to create a correct distributed database. The earlier lessons describe the missing obligations in detail.

Scenario B: partitionable event and tenant data

Consider an illustrative service storing events by tenant and time. Most queries read a bounded range within one tenant. The service prioritizes predictable partition-local access and can express its data model around those queries.

A distributed partition-oriented database can align with this workload. The schema should bound partition growth and account for hot tenants and current-time concentration. A table keyed only by a global date could concentrate every active writer in one partition; a carefully chosen tenant or bucket dimension changes that distribution.

The architecture must still define freshness, conflict semantics, deletion retention, and repair. A query-specific representation that is convenient for reads should not be treated as a current global authority for an unrelated cross-partition invariant without a supported protocol.

LSM economics across replicas

The local storage work repeats across replicas. Suppose illustrative logical input is 10 MB/s, there are three full replicas, and total host-level write amplification is eight at each replica. Aggregate device writes are approximately 10 × 3 × 8 = 240 MB/s, under the stated equal-work assumptions.

This excludes network framing, commit metadata, repair, backups, and device-internal amplification. Compression and different replica workloads can change the result. The calculation’s purpose is to prevent a capacity plan from confusing one logical mutation stream with total cluster work.

Repair introduces another data path. A node that missed writes needs to compare and receive state. Compaction and repair can overlap, competing for the same device and network. A service provisioned only for healthy foreground ingestion can struggle precisely when redundancy is degraded and recovery is most urgent.

Deletions and resurrection

An LSM tombstone prevents old values in local runs from reappearing during reads. In a replicated database, another node can retain an older value. Removing the deletion marker before the distributed protocol considers it safe can permit that value to return during reconciliation.

The safe horizon therefore depends on more than local compaction. Repair schedules, node absence, and retention policies affect the decision. Cassandra’s repair documentation is an important operational reference for this class of responsibility. Exact automatic-repair capabilities vary by version and should be checked rather than inferred from older descriptions.

A local engine can correctly discard a version according to its local contract while a host violates a higher-level replica contract. Layer interfaces must communicate retention requirements. This is an example of why correctness does not compose automatically from individually correct components with mismatched assumptions.

Query shape and secondary representations

A relational query can combine facts through joins at read time. A query-driven partitioned schema may precompute or duplicate the combinations needed by known queries. This can trade read flexibility for predictable access and write-side maintenance.

A new product feature can reveal the tradeoff. A query by a previously unindexed attribute may require a new table, backfill, search system, or broad scan. The architecture should include the cost of changing access patterns, not only the initial query set.

Duplicated representations need authority rules. If an ownership fact appears in several tables, the product must state whether updates are atomic across them, eventually reconciled, or restricted to a supported boundary. An application cannot infer stronger semantics merely because all copies are stored in one named product.

Operational comparison

An embedded library is operated through its host process. The host chooses metrics, deployment, replica management, and access control. A distributed database exposes more of those responsibilities as product facilities, but the operator still configures placement, consistency, repair, and capacity.

Reusing an engine can reduce local implementation risk while increasing integration responsibility. Using a complete distributed service can reduce cluster-protocol implementation work while constraining data modeling and operational choices. The right boundary depends on what the team needs to own and can reliably maintain.

Trace a local write through its host

Consider a hypothetical service embedding RocksDB. A request arrives at the host, which authenticates it, resolves ownership, applies its logical transaction protocol, and invokes local storage operations. The local engine organizes mutable state, log records, sorted files, caches, and compaction according to its APIs and configuration. The host must decide which storage outcomes are sufficient before replying to the remote client. RocksDB overview

An atomic batch of local mutations is useful, but its meaning should be kept precise. Applying a set of writes atomically does not by itself protect a decision based on earlier reads against concurrent host activity. The host may need transaction support, conflict detection, locking, or a serial execution boundary. It must also connect an uncertain RPC result to a stable request identity. A library’s local atomicity cannot infer the application’s logical operation boundary.

remote request
    -> host identity, routing, admission
    -> host transaction / ownership protocol
    -> local storage batch and durability choice
    -> host replication / decision protocol, if required
    -> reply under the promised contract

The arrows identify responsibilities, not a universal ordering for every architecture. A replicated-state-machine host may establish durable command order before applying local storage. Another design may replicate storage effects under a different recovery protocol. Either way, the designer must explain how local state is rebuilt and which history authorizes it.

Suppose the host replicates a logical increment command. Replicas need equivalent command interpretation and stable retry semantics. If it replicates a final key/value change instead, it must ensure that the earlier computation producing that change was valid under concurrent activity. Copying effects does not retrospectively validate stale reads. This is the same projected-versus-committed state issue developed in Lesson 48.

WAL choice and replicated durability are distinct

A custom replicated system may be tempted to disable a local log because a higher-level replicated log already exists. That can be a coherent architecture only if the recovery proof covers every acknowledged effect, log-retention boundary, snapshot, and partial installation. It is not a generic tuning recommendation. The host must know whether the local engine’s durable state can be reconstructed after losing memory or suffering a partially written file set.

Conversely, enabling a local WAL does not make the service tolerate loss of that machine or disk. Local persistence and replica independence cover different failures. An acknowledgment claiming a replicated decision needs the required protocol evidence, not simply several reachable HTTP servers. Failure-domain placement also matters: three copies on one failing device or power domain are not equivalent to independent durable replicas.

One useful review asks what happens after each crash: host before local invocation, engine after append but before flush, quorum after decision but before local application, and client after response loss. The correct answer names recoverable evidence and the operation’s outcome. “The engine is durable” is too broad to resolve these distinct positions.

Model a Cassandra access pattern before writing tables

Consider event history with two queries: the latest 100 events for one tenant and one day’s events for a particular device. A partition-oriented schema should make those requested bounds explicit. A teaching sketch might use (tenant_id, day_bucket) as an ownership grouping and time plus event identity for order inside it. A second access path may require another representation keyed by device and day. This sketch illustrates query-driven modeling, not complete CQL syntax. Cassandra modeling

A day bucket bounds a time dimension only if the event rate within that day is itself manageable. If one tenant emits 50,000 events/s, it contributes 4.32 billion events in a day. With 200 payload bytes per event, that is 864 GB of logical payload before replicas, indexes, or storage overhead. A key that is well distributed across ordinary tenants can still create an enormous partition for one outlier.

Subdividing that tenant into 16 buckets reduces the idealized equal share to 3,125 events/s and 54 GB/day per bucket. It also changes reads: obtaining the tenant’s latest events can require consulting multiple buckets and merging their ordered candidates. Distribution trades a narrower write hotspot for extra routing and read coordination. The split must preserve stable event identities and a documented ordering rule.

Time buckets also create temporal skew. The newest interval receives most writes while historical intervals mostly serve reads. A capacity plan based on the total number of partitions can be misleading when only a small set is active. Retention and deletion work can later concentrate on old intervals. Partition design is therefore a time-varying workload decision, not a one-time hash exercise.

Follow a replicated write and an incomplete read

In a simplified fixed replica set A, B, and C, a coordinator sends a write to the replicas and waits for the acknowledgments required by the operation’s consistency setting. Acknowledgment at one replica has a different resilience and observation contract from acknowledgment at two. The implementation’s local durability settings remain part of what each acknowledgment means. Cassandra replication architecture

Suppose A and B record the new value while C is unreachable. A later read contacting only C can observe an older value. A quorum read over a suitable fixed configuration intersects the completed write set, but the full behavior still depends on version reconciliation, concurrent writes, unfinished operations, and the product’s protocol. Set arithmetic should not be presented as proof that every Cassandra operation is linearizable.

Conditional operations have a separate documented protocol and scope. They should be evaluated according to their own guarantee, rather than inferred from ordinary writes or consistency-setting names. Similarly, logged batches and a relational serializable transaction are not interchangeable concepts. An architecture review needs the exact operation class, not a blanket label attached to the whole product.

This matters for the ownership example. An event log that can reconcile duplicate observations may tolerate a different contract from a unique item transfer. Copying both into partitioned tables does not make their correctness requirements identical. If a feature needs a guarantee beyond the selected operation’s supported scope, the application needs a different design or engine for that invariant.

Repair and compaction solve different disagreements

Compaction reconciles local sorted representations and reclaims state when its retention rules permit. Replica repair compares distributed copies and brings them toward the required agreement. Running compaction successfully does not prove another node received the deletion that makes an old value obsolete. Running repair does not prove local files are optimally organized for future reads. Compaction, Cassandra repair

Take a deletion recorded on A and B while C is offline with an older value. If A and B discard all deletion evidence before the distributed protocol can reconcile C, C’s old value can be mistaken for surviving data during later reconciliation. The safe policy depends on repair completion, replica absence, tombstone grace and retention, and the actual operational procedures. The conceptual problem is distributed knowledge of deletion, not merely the age of one local file.

A repair interval expressed as a calendar setting is insufficient evidence by itself. Operators need to know whether repair actually completed across the relevant replicas and ranges, whether nodes remained absent beyond the supported horizon, and whether replacement procedures preserved the intended history. Version-specific automation can help but should be verified rather than assumed. Recovery correctness depends on completed work, not the existence of a schedule.

Calculate the maintenance recovery envelope

Suppose the earlier 240 MB/s aggregate write calculation runs on storage with 300 MB/s of effective bandwidth available to that write workload. In the simplified model, only 60 MB/s remains for extra write work before saturation. If recovery adds 120 MB/s of device demand, aggregate demand reaches 360 MB/s and the backlog grows at 60 MB/s. Ten minutes adds 36 GB of pending work under constant rates.

After the recovery surge ends, clearing 36 GB with 60 MB/s of spare capacity takes another ten minutes. That assumes foreground traffic remains unchanged and the background work can efficiently use all spare capacity. It omits additional reads, seeks, network interference, and nonlinear latency. The example illustrates a recovery envelope, not a measured property of either product.

A node replacement can combine network copy, local writes, compaction, and cache warming. Counting only the copy’s bytes may understate the work needed before the replacement is a useful replica. Reserving headroom during normal operation can therefore improve both failure recovery and foreground tail latency. A system at its healthy steady-state ceiling has no budget to restore redundancy without reducing some other work.

Local stalls can become distributed tail latency

An LSM system needs to slow or stop ingestion when foreground writes outrun flush and compaction capacity. Otherwise immutable tables, level overlap, and disk consumption can grow beyond safe bounds. A write stall can be a protective mechanism exposing insufficient maintenance capacity. Merely increasing a backlog threshold postpones the boundary while increasing retained work. RocksDB write stalls

In a distributed service, one replica’s stall may or may not delay a particular request, depending on the required acknowledgment set and failure state. If a fast quorum remains, the service may avoid that replica on the critical path for some operations. If another replica fails, the previously hidden laggard can become essential. Apparent resilience to one slow replica during healthy operation does not establish degraded performance.

The same analysis applies to read amplification. More overlapping runs can increase local read work; a distributed query can then combine that local delay with routing and replica response variability. Observability should connect request latency to local flush backlog, pending compaction, cache misses, and the specific replicas required for completion. An average cluster CPU chart can miss the decisive shard and device.

Match the adopted layer to the team’s responsibility

A query-specific second representation also needs a repair story of its own. Suppose the authoritative event was accepted, but a derived device lookup was not updated. Replica repair can make replicas agree on each stored partition while both representations still disagree about the business fact. Agreement among copies of the first table does not manufacture a missing row in the second table. The application must identify which representation is authoritative and whether a supported atomic operation, recoverable change stream, or reconciliation procedure maintains the relationship.

Stable event identity helps make that procedure repeatable. A backfill can derive the secondary row from the authoritative event, and a repeated delivery can replace or recognize the same logical result rather than append a second event. Retention must keep enough source history for rebuilding. If the source disappears before repair completes, the application may no longer possess evidence needed to distinguish missing history from intentional absence. This is an application-level consistency boundary above both local compaction and ordinary replica repair.

The table prevents a common procurement error: assuming that choosing a sophisticated component transfers responsibility for every surrounding protocol. It also prevents the reverse error of discarding a reusable engine because it does not claim to be a full distributed service. A component is valuable when its contract matches the layer the host actually needs.

RocksDB illustrates a reusable local representation boundary. Cassandra illustrates a broader partitioned service with a specific application and operational model. Their favorable workloads and limits differ because their responsibilities differ. The comparison is most useful when it makes those responsibilities visible before the team commits to an API, schema, or recovery procedure.

Architectural synthesis

Separate local representation from distributed authority. LSM storage explains the economics of writes, reads, space, and maintenance. Cluster protocols explain how several copies become one usable service under failures. Neither layer should be credited with guarantees supplied only by the other.

The next case study examines memory-oriented execution, where avoiding storage on the ordinary read path changes latency but leaves persistence, eviction, and failure semantics as explicit choices.

Sources and further reading

• RocksDB overview
• Cassandra 5.0 architecture
• Cassandra 5.0 repair

Course index · Lesson plan