Parallel Computing

Parallel Computing

MemCP evaluates independent functional work in parallel when the planner and runtime can prove that doing so is safe and useful. Storage operators process columns in batches and distribute sufficiently large shard work over a bounded worker set. Small batches remain local because scheduling may cost more than the work itself.

Functional code makes dependencies explicit: a function result follows from its inputs instead of hidden mutation of shared outer variables. That allows the runtime to run independent branches concurrently and combine their values later. Explicit shared state still exists through sessions, locks, caches, I/O, and storage writes, so “written in Scheme” does not by itself make an arbitrary callback parallel-safe.

MemCP's dialect was shaped around that property: imperative outer-scope mutation was omitted, set creates a scope-local binding, begin provides a local environment, and Scheme values/code can be serialized. Serialization is useful for cached/generated programs and is also a prerequisite for the planned remote scan model, but it does not by itself make multi-node execution available today.

Scan pipelines propagate request cancellation and early-stop conditions rather than always materializing every matching row. Nested scans and transaction-bound operations use bounded fanout to avoid goroutine explosion and lock cycles. Parallel reducers combine per-worker results using explicit neutral values and reduction functions.

Parallel primitives and scans

• parallel evaluates independent expressions and waits for their results.
• parallel_map and parallelN distribute sufficiently large collection work with bounded fanout.
• scan performs shard-local filter/map/reduce work and combines shard results in a second reduction phase.
• scan_order can parallelize eligible filtering/order preparation while preserving the serial order required by output and reduction.
• newsession, once, and mutex provide explicit coordination when state must be shared.

Choose an associative reducer and a correct neutral element. Floating-point addition, string concatenation, first/last selection, and side-effecting callbacks can produce order-sensitive results; the SQL planner retains serial boundaries where semantics require them.

Supported hot Scheme procedures may be compiled by the native x86-64 JIT. Compilation and guarded specialization are optional optimizations; unsupported procedures continue in the interpreter. None of these mechanisms promises linear scaling with core count: memory bandwidth, cardinality, compression, synchronization, query shape, and physical plan determine the result.

Publish repeatable measurements rather than a fixed speed-up. See Performance Measurement, Query Planner and Physical Lowering, and JIT Compilation.