Performance Measurement

Performance Measurement

Every published result must be reproducible and count successful work.

Record MemCP commit/build flags, comparison-system version/configuration, schema, data generator and row count, query text, ENGINE mode, hardware, filesystem or remote backend, memory limits, concurrency, warm/cold state, and raw samples.

Verify correctness before timing: HTTP status, error body, row count, values, and restart behavior. A fast 401 or other non-2xx response is not query throughput.

Current headline observations

MemCP has reached speedups of 10× and more over MariaDB/PostgreSQL in measured OLAP and search-oriented workflows. The relevant mechanisms include compact RecSet domains, compressed column scans, batch execution, late materialization, adaptive physical structures and shard parallelism.

Current isolated OLTP measurements show the opposite trade-off: those paths take about 1.3–2.0× as long on MemCP. In complete WordPress- and wiki-style page builds, this has made no significant difference to overall page-loading time in the measured application workflows. The application, the number and mix of queries, and the share of time spent outside SQL determine the end-to-end effect. By contrast, one filtered-list workflow over roughly one million documents took around 30 seconds on PostgreSQL and 1.6 seconds on MemCP for the same query. That makes search, filtering, grouping and analytical reads the clearest current migration targets while OLTP/JIT work continues.

These headline figures should be stated rather than hidden, but they do not replace the reproducibility record below. A benchmark publication must attach the exact competing versions, schema/query, dataset and distribution, hardware, durability settings, cache state, concurrency, result validation and raw samples. Do not combine the best result from one workload with the conditions of another.

Write-path measurements have also reached about 10× the throughput when changing the same table/workload from safe to logged. Report that result together with the durability difference: safe synchronizes WAL at commit, whereas logged leaves recent WAL bytes in volatile operating-system buffers. A comparison that changes ENGINE without naming the changed failure guarantee is incomplete.

Use calibrated SQL performance suites for end-to-end plans and Go benchmarks for isolated storage primitives. Run multiple samples, report distributions rather than the best run, and compare against current master under identical conditions.

make test
PERF_TEST=1 PERF_EXPLAIN=1 make test
go test ./storage/ -bench 'BenchmarkName' -run '^$' -count=5

For A/B changes, save raw output for both commits and use benchstat. Profiles should distinguish compile time, physical preparation, execution, allocation, lock waits, storage reload, and background maintenance. Attach EXPLAIN and EXPLAIN COMPILE for planner-sensitive results.

SQL performance regression framework

Suites below tests/performance/ are disabled during ordinary correctness runs unless PERF_TEST=1 is set. The runner stores machine-specific row counts and timings in .perf_baseline.json, performs warm-up work, repeats measured cases (five times by default), and reports the median. A result above the stored baseline plus tolerance fails the gate.

# Run with existing baselines
PERF_TEST=1 make test

# Recalibrate row counts/times for this machine
PERF_TEST=1 PERF_CALIBRATE=1 make test

# Freeze row counts while bisecting a regression
PERF_TEST=1 PERF_NORECALIBRATE=1 make test

# Include query-plan output
PERF_TEST=1 PERF_EXPLAIN=1 make test

Calibration aims for sufficiently long cases (currently roughly 10–20 seconds) and adjusts rows gradually. The runner also applies one protected pre-server wall-clock calibration for the execution architecture; a regression inside MemCP must not be allowed to enlarge its own time allowance.

A performance line shows the measured duration, threshold, calibrated rows, per-row time, heap and CPU utilization, for example:

Perf: COUNT (4.3s / 13s, 100,000,000 rows, 0.04µs/row, 25GB heap, 1522%/2400% CPU)

Read all fields together: lower per-row time is meaningless if the query failed, row count changed unexpectedly, the process swapped, or a different plan/durability mode was used.

Variable	Purpose
PERF_TEST=1	Enable performance suites.
PERF_CALIBRATE=1	Reset/update the local performance baseline.
PERF_NORECALIBRATE=1	Keep row counts fixed for commit-to-commit or bisect comparisons.
PERF_EXPLAIN=1	Print plans for performance cases.
PERF_REPEAT=N	Change the number of measured repetitions; default is five.

Do not commit a baseline produced while the machine was thermally throttled, heavily loaded, swapping, or using a different build configuration. Preserve the known-good baseline before a bisect.

For a regression bisect, calibrate on the known-good commit, keep a copy of its baseline, and freeze row counts while Git tests candidate commits:

git checkout GOOD_COMMIT
PERF_TEST=1 PERF_CALIBRATE=1 make test  # repeat until stable
cp .perf_baseline.json /tmp/memcp-perf-good.json

git bisect start HEAD GOOD_COMMIT
cp /tmp/memcp-perf-good.json .perf_baseline.json
git bisect run bash -c 'PERF_TEST=1 PERF_NORECALIBRATE=1 make test'

Storage microbenchmarks and profiles

For a storage encoding or hot loop, benchmark the smallest relevant package and compare several samples:

go test ./storage/ -bench 'BenchmarkEnumPerElem' -run '^$' -count=5 > /tmp/before.txt
# apply/build the candidate under the same conditions
go test ./storage/ -bench 'BenchmarkEnumPerElem' -run '^$' -count=5 > /tmp/after.txt
benchstat /tmp/before.txt /tmp/after.txt

Microbenchmarks establish mechanism, not end-to-end query benefit. Confirm the change with SQL, result validation, representative cardinalities, and warm/cold conditions. Use CPU, memory/allocation, mutex/block, and I/O profiles to explain a difference instead of attributing every gain to the edited function.

Coverage is not performance

Coverage builds are useful for finding untested paths but add instrumentation and must not supply release benchmark numbers. Build and collect coverage separately with MEMCP_COVERAGE=1 and MEMCP_COVERDIR when the goal is test quality.