Course contents
On this page
Local concurrency and execution — extended reference
Original detailed coverage and citations accompanying the focused local concurrency and execution lesson.
This reference preserves the earlier detailed chapter and its citations. Start with the guided explanation, then use these sections for additional mechanisms and qualifications. References to the original lesson sequence are historical.
Part 1: Foundations: understanding work, time, and resources
Prerequisites: Lesson 02 and the preceding foundations.
Learning outcomes
- Compare a thread-per-request server with an event loop under blocking and independent work.
- Identify synchronization and scheduling costs that limit parallel speedup.
- Explain why many waiting requests do not imply many requests executing simultaneously.
Many clients, few execution resources
A server accepts ten thousand connections. Most clients are idle, some await responses, and a few send bursts. A thread dump shows only dozens of workers. This is not a contradiction. A connection is a communication relationship; a thread is an execution resource; a transaction is a logical unit of database work. Treating them as interchangeable leads to misleading claims about concurrency and poor capacity choices.
The motivating failure is a service that adds one thread for every connection. It performs well at low load, then spends increasing time scheduling, waiting, and retaining per-thread memory. Another implementation uses one event loop and stays responsive until a callback performs a long synchronous computation. Both failures come from assigning work to execution resources without a complete model of blocking and ownership.
Concurrency means multiple operations are in progress over overlapping intervals. Parallelism means work executes at the same time on different execution resources. An asynchronous server can have high concurrency with little CPU parallelism. A parallel query can use many cores while serving only one client. Database design requires both terms to remain precise.
The lifecycle of a request
A request crosses several states. Bytes arrive, framing establishes a complete message, admission determines whether to accept work, scheduling assigns execution, the engine performs reads and changes, durability may be awaited, and a response is transmitted. Each state can have a separate queue and a separate owner.
socket bytes -> framed request -> admission -> runnable work
|
CPU execution or I/O wait
|
completion -> outputThis diagram distinguishes waiting from execution. A client awaiting a durable commit is in progress even if no application thread is actively computing on its behalf. An I/O completion can make work runnable without requiring a dedicated thread to have waited throughout the operation.
A complete implementation also handles cancellation and failure at each state. A disconnected client may leave queued work that should be removed, or an already committed transaction whose result must remain retrievable. Cancelling a waiting task does not imply that all of its external or database effects were reversed. The protocol must define which boundary has been crossed.
Processes and threads
A process provides an address space and an operating-system protection boundary. Threads within a process generally share its memory while maintaining their own execution state and stacks. Communication within a process can be a function call or shared-memory operation, but safety requires explicit ownership and synchronization. Communication between processes can use sockets, pipes, shared memory, or other mechanisms, each with its own boundary costs.
A process-per-connection architecture isolates some failures and state but uses operating-system resources per connection. A thread-per-request design is straightforward when blocking is expected and concurrency is bounded. A fixed worker pool limits active execution but needs an explicit policy for queued work. An event-driven architecture represents waiting operations as state and resumes them when progress becomes possible.
These are architectural patterns, not performance rankings. An event-driven server still performs CPU work, memory allocation, synchronization, and system calls. A thread-based server can use efficient asynchronous I/O beneath a higher-level interface. The relevant questions are how much work is runnable, how waiting is represented, and how ownership moves between stages.
Scheduling and context switching
The operating system schedules runnable threads onto available cores. A thread waiting for I/O or a lock may stop consuming a core while retaining memory and other resources. A context switch changes execution state and can disturb cache locality. The cost depends on hardware, address-space changes, working-set behavior, and scheduling conditions.
If a machine has eight useful CPU execution resources for the workload, making eighty CPU-bound workers runnable does not create eighty resources. It increases scheduling competition. More workers can improve utilization when operations frequently block, but excessive workers can increase queueing and memory pressure. The useful level depends on the mix of computing, I/O, and serialized critical sections.
A worker count should therefore follow measured service demands. If a request consumes 200 microseconds of CPU and spends 5 milliseconds awaiting storage, a synchronous worker can remain occupied much longer than its CPU requirement suggests. Asynchronous waiting may free execution capacity, while the request still occupies transaction metadata, buffers, and admission capacity. Removing a blocked thread does not make the in-flight operation free.
Event loops and asynchronous state machines
An event loop repeatedly processes ready events and advances associated work. A network read might deliver partial bytes, an I/O completion might release a buffer, or a timer might expire a deadline. The event loop should perform bounded work per event so unrelated ready operations can progress.
A simplified state machine makes this explicit:
on readable(connection):
append bounded bytes to input buffer
while a complete frame exists and work budget remains:
request = decode_one_frame()
if admission permits request:
enqueue(request)
else:
send_overload_result()
on completion(request):
enqueue response bytes
release request resourcesThe pseudocode omits transport error handling, authentication, parser bounds, and cancellation machinery; those remain required implementation responsibilities. Its purpose is to show bounded processing and the separation between receiving a request and executing it. Decoding an unlimited number of frames in one callback can let a busy connection monopolize the loop.
A long computation in the loop delays every other event assigned to it. Moving that computation to a worker pool can restore responsiveness, but the new queue needs capacity limits. Without them, the implementation has merely moved unbounded waiting from the event loop to another part of the process. Queue location changes observability and memory use, not the arrival rate.
Synchronization and critical sections
Shared mutable state needs a rule that prevents incompatible simultaneous access. A mutex gives one holder access to a protected region. A read/write lock distinguishes compatible readers from exclusive writers. Atomic operations can update certain values without a traditional mutex, but a multi-field invariant generally needs more than several individually atomic assignments.
A critical section is the work performed while holding exclusive protection. Its duration matters because contending operations cannot independently complete that same protected transition. If a lock guards a global counter for 10 microseconds, its ideal serialized capacity is 100,000 changes per second before other costs. If the holder performs a remote call and retains the lock for 10 milliseconds, the ideal ceiling falls to 100 changes per second.
This example does not establish the capacity of an entire engine. Other operations may proceed independently. It shows why protecting one shared fact for longer can dominate a contended workload despite ample unused CPU. It also shows why an application boundary can affect concurrency: a lock held across network exchanges is exposed to a different duration distribution from a local update.
Lock granularity trades metadata and coordination against independence. A global lock is simple but serializes otherwise independent work. Per-record locks create opportunities for overlap while requiring lock allocation, deadlock handling, and protection for relationships or ranges. The transaction-locking details appear in Lesson 20; here the important point is that finer granularity has real implementation costs.
Ownership can replace some synchronization
An alternative to sharing a mutable structure among workers is to assign one worker ownership and communicate through messages. A partition or actor processes its own messages in order. This can simplify reasoning and improve locality because the owned data tends to remain near the same execution resource.
The benefit depends on whether operations stay within ownership boundaries. An independent tenant partition is a strong candidate. A transfer touching two partitions requires a coordination protocol. A global counter remains global even if requests reach it through messages. Ownership moves the synchronization boundary into message ordering and cross-owner interaction.
Message passing also creates queues, copies or references, wakeups, and failure states. A queue that grows without bound can exhaust memory. A referenced buffer must remain valid until the recipient finishes. A worker crash can leave ownership ambiguous unless the larger system defines recovery. Actor terminology does not remove these responsibilities.
Little’s Law connects concurrency to latency
For a stable system over a suitable observation interval, Little’s Law states that average work in the system equals arrival or completion rate multiplied by average residence time. Using conventional symbols, L = lambda × W, where L is average in-flight work, lambda is rate, and W is time in the same boundary.
At an illustrative 20,000 completed operations per second and 5 milliseconds average residence time, the average concurrency is 20000 × 0.005 = 100 operations. If average residence time rises to 50 milliseconds at the same completion rate, average in-flight work rises to 1,000. The additional requests need memory and metadata even if they mostly wait.
The boundary matters. Counting only actively executing workers while measuring end-to-end request time violates the equation’s consistent-boundary requirement. An unstable growing queue also cannot be summarized as a steady operating point. Little’s Law is an accounting relation, not a guarantee that adding concurrency will increase throughput.
This relation explains a common operational surprise. A small latency increase can multiply retained in-flight state at a fixed request rate. Larger in-flight state can increase memory pressure, which increases latency further. Admission control breaks this feedback by limiting accepted work and returning explicit overload outcomes instead of allowing arbitrary accumulation.
Fairness and long operations
A scheduler determines not just total efficiency but who experiences delay. First-in, first-out scheduling is easy to describe, yet a single long operation can delay many short ones behind it. Priority scheduling can improve important traffic while starving lower-priority work. Per-tenant queues can limit interference but need a policy for borrowing idle capacity.
Preemption means interrupting work so another task can run. The operating system can preempt a thread, but it cannot safely interrupt an arbitrary database mutation at any instruction and expose half-updated state. An engine needs appropriate yield points, cancellation checks, transaction rollback, or resumable operators. Query and transaction semantics constrain scheduling choices.
Background work also needs fairness. Compaction, checkpoints, vacuuming, and replication catch-up preserve future performance and recoverability. Starving them can make foreground latency look good briefly while accumulating debt. Giving them unlimited priority can make the service unresponsive now. Production scheduling balances current requests with the work needed to keep future requests possible.
Case connection: fast command execution and slow neighbors
Redis’s latency documentation discusses the impact of slow commands and other sources of delay around its command-processing path. This provides a useful case connection for a general observation: when operations share a serialized execution stage, an expensive operation can delay otherwise cheap operations assigned to that stage. Exact threading and background-work behavior is version- and component-dependent, so “single threaded” should never be treated as a description of every task in the process.
The architectural inference is conditional. A serialized stage can be efficient for short work with strong locality and can avoid some shared-memory coordination. A workload with long computations, unbounded scans, or external waiting may need separate execution resources or a different division of responsibilities. The same engine can therefore perform very differently under two command mixes.
Follow an asynchronous read without assuming free execution
Suppose a connection receives a query whose index leaf is absent from memory. The event loop parses enough of the request to identify work, submits an I/O operation, records the continuation state, and returns to other ready work. When completion arrives, the engine schedules the continuation. The continuation validates the result, resumes traversal, and eventually queues response bytes. The pending query existed throughout, although no dedicated thread needed to block for its entire lifetime.
The continuation retains resources: request parameters, a deadline, references to transaction state, possibly a snapshot, and enough information to resume safely. If 100,000 pending requests retain 8 KiB each, the request state alone occupies about 781 MiB. This constructed calculation uses binary units: 100000 × 8192 / 1048576. It excludes connection buffers, allocator overhead, cached pages, and transaction metadata. Asynchrony changes how waiting is represented; it does not make waiting costless.
One implementation hazard is resuming after cancellation. A client may abandon the request while the underlying I/O remains in progress. The completion handler must not dereference freed request memory or publish a result into a reused connection slot. Reference ownership, generation identifiers, and explicit terminal states can prevent such errors. The important property is that each resource has a lifetime rule that holds across delayed callbacks.
A bounded scheduling state machine
An illustrative admission state machine makes ownership explicit:
on_receive(request):
validate_size_and_identity(request)
if queue_capacity_exhausted():
return explicit_overload
enqueue(request)
worker_step():
request = choose_ready_request()
if expired_before_execution(request):
finish_without_starting(request)
return
run_until_complete_or_safe_wait(request)
if waiting:
register_continuation_with_owned_state(request)
else:
release_execution_resources(request)
enqueue_bounded_response(request)This is architectural pseudocode, not a transaction engine. The expiration check is safe only before the operation starts. Once execution can commit, deadline expiration cannot be treated as proof that nothing happened. A real implementation defines cancellation points and preserves the outcome of operations whose result becomes uncertain. That distinction is central to retry handling in Lesson 5.
The pseudocode also separates execution resources from response resources. A slow client should not necessarily retain a scarce transaction worker after a result is ready. Detaching response delivery can improve isolation, but requires a bounded output queue and a defined policy for clients that never drain it. Otherwise memory becomes the new unbounded resource.
Admission itself can require fairness. A global bounded queue can be monopolized by one tenant. Per-tenant admission limits or fair queueing can protect other users, while a shared overflow budget can keep resources useful when some tenants are idle. These are policy choices with observable effects: rejected requests, latency distribution, and the degree to which tenants can borrow spare capacity.
The difference between a pool and a pipeline
A worker pool has several interchangeable execution resources. A pipeline divides work into stages such as parsing, planning, execution, logging, and delivery. A system can combine both: several parsing workers feed partition owners, and log completion resumes transactions. Improving one stage does not raise throughput beyond the slowest stage’s sustainable rate.
Assume a constructed pipeline accepts 20,000 parses per second, executes 12,000 transactions per second, and can durably complete 15,000 transaction records per second for the stated mix. Execution is the initial ceiling. Doubling parser capacity does little for steady throughput. Worse, allowing parsers to produce an unlimited queue can raise memory use and waiting. Feedback between stages is necessary so the front end stops admitting work faster than downstream capacity can handle.
Pipeline parallelism can overlap different requests without shortening the dependent path of one request. A transaction still traverses its necessary stages. Batching at one stage can increase aggregate efficiency while adding time before the batch fills. The effect depends on traffic intensity: under high load the batch may form immediately; under low load the configured timeout may dominate latency.
Batching can also change fairness. Processing an enormous batch from one connection can delay later work from others. A scheduler may cap per-turn work by rows, bytes, or elapsed time. The cap must respect safe interruption boundaries. A row limit does not guarantee a time limit if individual rows invoke expensive computation, so implementations sometimes combine budget dimensions.
Why more threads can reduce useful throughput
Each runnable worker competes for CPU time and can enlarge the live working set. If every worker touches a different set of pages, more concurrency can reduce cache locality. If they touch the same mutable structure, more workers can increase latch contention and cache-line transfers. If they issue storage requests faster than the device can finish them, they grow the device queue. These effects can appear well before the host reaches its advertised limits.
Suppose eight workers achieve 80,000 successful changes per second. Increasing to thirty-two workers creates more conflicts, and only 90,000 attempts per second succeed out of 180,000 attempts. The useful throughput rises modestly while failed work and tail latency rise sharply. Measuring attempts alone would exaggerate improvement. Measuring only CPU utilization would miss that much of the work no longer produces committed outcomes.
The appropriate concurrency limit is workload-dependent. Independent analytical tasks may benefit from more execution resources, while short contended transactions may favor fewer. A controller can adjust concurrency using observed completion time, queueing, and overload signals, but delayed feedback can oscillate. Fixed limits with well-understood margins are often easier to reason about than an adaptive policy whose behavior under failure is unknown.
Scenario: a transaction waits while retaining the wrong thing
A worker acquires an item lock and then waits for a remote pricing service. While it waits, its CPU is free but the item is unavailable to other buyers. Replacing a blocking call with an asynchronous call frees the thread; it does not free the logical lock. The item’s serialized occupancy remains the remote call’s duration. A thread-efficiency improvement therefore may leave the business bottleneck unchanged.
A better boundary might fetch a signed quote before entering the transaction, then validate the quote and allocate the item within a short local transaction. Whether this is correct depends on price-expiry and stock rules. Another possibility records a pending reservation, releases the transaction, and resolves it through a later transition. That creates expiration, cancellation, and crash-recovery responsibilities. Neither alternative is simply “make the function async.”
This example shows why scheduling and correctness have to be reasoned about together. Releasing a resource early is an optimization only when later work can safely proceed without assuming that resource remains exclusively owned. The engine must distinguish physical execution resources, transactional protection, snapshots, and external authority. A clean concurrency design names each one rather than treating all waiting as equivalent.
Visualize execution and waiting on different lanes
time ------------------------------------------------------------>
request A: CPU parse | wait for page ----------------| CPU finish
request B: CPU lookup | wait for log ----------------| done
event loop: A parse | B lookup | other ready work ... | A resume
shared item:
A owns transition ----- releases | B may own transition -----The lanes are schematic and their widths are not measured durations. The event loop can use CPU while A waits for its page. It cannot allow B to make an incompatible change to A’s protected item merely because A is not running. Execution scheduling and logical ownership are separate axes.
A thread-per-request design can express the same waiting pattern with blocked threads. An event-loop design retains continuations instead. If the workload is mostly independent network waiting, avoiding a large blocked-thread population can reduce scheduling and memory overhead. If the workload performs long CPU computation without yielding, an event loop can delay every continuation assigned to it. A separate bounded worker pool may be appropriate, while its input and completion queues still need limits.
The diagram also clarifies why adding connections does not add CPU capacity. Additional arrivals occupy more request lanes; available execution lanes remain constrained by cores, ownership, and scheduling. Concurrency is useful when it overlaps independent waiting with work, and harmful when it primarily lengthens queues behind the same protected transition.
Design conclusion
A concurrency design is a map of ownership, queues, and execution resources. Count connections, requests in progress, runnable workers, active transactions, and committed outcomes separately. Identify where operations wait, which resources they retain while waiting, and whether another independent operation can use the execution capacity.
The networking chapter follows the communication path itself. A server can schedule work efficiently while the network still imposes unavoidable propagation, loss-recovery, and connection-establishment costs. Understanding both paths is necessary before attributing latency to the database engine.