Pass rows from one operation to the next

A query asks for names of products costing less than 30 credits. We know what a scan and a filter do. How do we connect them so that the engine produces the answer without building a complete intermediate table after every step?

An execution plan connects operators. An executor runs those operators and manages their state. We will use a pull interface: an operator asks its input for the next row when it needs one.

Follow one request for output

Our rows arrive in this order: lamp at 35, tote at 20, notebook at 8. The filter must reject the lamp before it can supply the first result.

Output asks for one name
The projection asks the filter for one qualifying row.
Filter asks the scan
The scan supplies the lamp. 35 is not below 30, so the filter asks again.
Scan supplies the tote
20 passes. The filter returns its row; projection returns Canvas tote.
The caller asks again
The operators resume from their saved positions and eventually return Notebook.

Requests move toward the input. Qualifying data returns toward the output. Rejected rows do not become results.

Projection chooses output fields, such as name, from a row. Each operator retains enough state to resume: the scan’s current position, or a join’s lookup structure. A typical interface has open, next, and close operations.

End-of-input must be distinct from a row containing missing values and from an error. Otherwise a storage failure could look like a successful query that happened to return fewer rows.

Why batches are an alternative

A next call for every row adds control-flow overhead. A batch interface exchanges several rows or column values at a time. A filter can examine a compact group and record which positions passed. This can reduce repeated calls and improve memory access patterns.

Batching does not change the requested result. It changes how work is scheduled and represented. Large batches may retain more memory or delay returning the first result. The right size depends on the operators and workload; a diagram does not establish a speedup.

Some operations must collect data before producing output. A full sort generally cannot promise the smallest row until it has examined its input. Such a blocking operator breaks simple streaming through the whole plan.

Stop before the scan finishes

The caller cancels after receiving the tote. Is it enough to stop requesting names?

Account for retained resources

No. The engine must close or cancel the remaining operator state, releasing pinned pages, temporary files, and memory as required. A suspended scan or join may still own resources even though no more results are requested. Cleanup is part of the execution protocol.

We can now trace both data and control through a plan. Next we ask which changes to that plan preserve its answer.

CMU’s query-execution project supplies an implementation setting for operator executors and plan transformations.

Extended reference

The extended chapter preserves the original technical coverage, additional examples, and citations. The explanation above supplies the concepts needed for this page; the reference develops the specialist cases.

Find a section from the original chapter