MemCP - User contributions [en]

Benchmark MemCP vs. MariaDB on Wordpress

2026-09-09T21:54:02Z

Carli:

= Editing Benchmark MemCP vs. MariaDB on Wordpress =

'''MemCP makes repeated WordPress database work faster.''' On current master, MemCP-JIT completes this fixture's search plus total-count query '''36.0× faster''' than MariaDB, its comment count '''21.1× faster''', and its archive query '''6.2× faster'''. MemCP-JIT has lower measured warm latency across all seven query shapes in this run.

MemCP speaks the MySQL protocol and can also '''host PHP directly in the database process'''. WordPress is our example application: the hosting interface is general purpose and is also intended for MediaWiki and other PHP software. Try MemCP with your PHP hosting and measure the queries your application actually runs.

== MariaDB vs. MemCP vs. MemCP-JIT ==

Measured on master [https://github.com/launix-de/memcp/commit/43d0c81401616fafeeb456b22f4a47d6e25361d6 43d0c8140], including the SQL literal-parameterization improvement in PR #873. All three columns use the same seven WordPress query shapes and data snapshot. These are '''warm database round trips''', not HTTP page-load times. Lower is better.

'''Dataset size:''' 22,063 rows in wp_posts (including 2,000 generated articles and 20,000 revisions), 14,013 post-metadata rows, 6,001 comments and 503 options. Taxonomy data comprises 175 terms, 175 term-taxonomy entries and 7,701 object-to-taxonomy relationships. These are the measured snapshot's table counts, not the number of rows returned by each query.

Each value is the median of 11 batch means, each containing 100 complete query executions, after 30 warmups. For SQL_CALC_FOUND_ROWS statements, the measured operation also includes the subsequent SELECT FOUND_ROWS() on the same connection.

{| class="wikitable sortable"
! Query !! MariaDB (ms) !! MemCP (ms) !! MemCP-JIT (ms) !! MariaDB / JIT !! Result check
|-
| Option lookup: siteurl || 0.079 || 0.069 || '''0.056''' || 1.39× || Equal; 1 row
|-
| Elementor metadata join + LIMIT || 0.152 || 0.108 || '''0.079''' || 1.93× || Equal; 0 rows
|-
| Taxonomy DISTINCT/join, ordered by name || 0.224 || 0.245 || '''0.142''' || 1.58× || Equal
|-
| Comment count with join || 3.240 || 0.222 || '''0.153''' || 21.14× || Equal; 1 row
|-
| WordPress search + FOUND_ROWS || 30.405 || 1.278 || '''0.844''' || 36.01× || Equal; 10 rows
|-
| Yoast CAST metadata join + FOUND_ROWS || 0.197 || 0.248 || '''0.144''' || 1.37× || Equal; 0 rows
|-
| Monthly archive DISTINCT || 0.718 || 0.168 || '''0.116''' || 6.21× || Equal; 10 rows
|}

Timing values are from master 43d0c8140. Taxonomy result equality (all 15 rows, including their order) was verified separately with [https://github.com/launix-de/memcp/pull/871 PR #871].

The two metadata queries return zero rows on this snapshot. Their times describe proving an empty result, not retrieving a populated plugin workload. Tiny differences in the smallest queries are sensitive to scheduling on this shared development machine. We do not average these seven unrelated query shapes into a single WordPress speedup.

== Host PHP directly: remove a round trip for every small query ==

The optional FrankenPHP dependency supplies the PHP runtime; only MemCP's own bridge and PDO addon live in this repository. PHP requests use ZTS threads, with OPcache enabled. PHP folders are attached through the existing HTTP handler, alongside the SQL API and dashboard.

With a PHP-enabled build and an external ZTS PHP development installation:

<pre>
make php PHP_CONFIG=/path/to/zts/php-config
./memcp-php --no-repl -data /path/to/data --api-port=8098 --mysql-port=3307 --serve=/path/to/php-app lib/main.scm
</pre>

The application is mounted at /; /dashboard remains available. Scheme code can mount additional applications through servePHP. No WordPress source is bundled into MemCP.

The PDO addon supports authenticated memcp: connections and retains ordinary MySQL wire connections. A mysql: connection to localhost/127.0.0.1 and this MemCP server's own MySQL port can be routed internally; other destinations continue using their original driver. Both paths require database credentials. WordPress's use of mysqli is a separate connector question: PDO routing alone is not a claim that every WordPress installation or plugin uses PDO.

A separate microbenchmark measures SELECT 1 inside PHP, using an already established connection. Each round performs 100 warmups and 2,000 measured executions; seven rounds alternate the two transports. The timer includes query(), fetchColumn(), and closeCursor(), but excludes HTTP, connection establishment, and PHP startup. Pdo\Mysql::connect forces the real wire driver for the control, so automatic PDO routing cannot accidentally turn both measurements into RAM calls.

{| class="wikitable"
! PHP SELECT 1 transport !! Median of round medians
|-
| MySQL wire over local TCP || 37.19 µs
|-
| MemCP in-process PDO || '''11.57 µs'''
|}

This removes '''69% of the measured latency''', or 25.61 µs per tiny query. At 100 such queries per request, the arithmetic saving is approximately 2.56 ms; at 1,000 queries it is 25.61 ms. Those are extrapolations, not measured WordPress page-load improvements.

For a workload-specific budgeting example, halving 100–200 µs of overhead would save '''50–100 µs per call (50%)'''. Across 100 database calls, that would add up to 5–10 ms per request. Those figures illustrate how small savings accumulate; they were not measured here. The measured transport comparison above is the evidence for this setup; measure your own query mix before assigning a per-request saving.

== Reproducible setup ==

{| class="wikitable"
! Item !! Configuration
|-
| Measurement date || 9 September 2026 (Europe/Berlin)
|-
| CPU || AMD Ryzen 9 7900X3D, 12 physical cores / 24 logical CPUs
|-
| Memory / OS || Approximately 62 GiB RAM; Ubuntu 24.04.4 LTS, x86-64
|-
| MemCP || Master 43d0c81401616fafeeb456b22f4a47d6e25361d6; unchanged source for both SQL builds
|-
| SQL compiler/toolchain || Patched Go 1.27.0, revision 84fe25ee45468f5e3920c11c24e2bbb926fc3a72; GOEXPERIMENT empty vs. GOEXPERIMENT=jit
|-
| MariaDB || 10.11.14; query cache OFF; InnoDB buffer pool 128 MiB
|-
| PHP microbenchmark || Same MemCP master, built with Go 1.26.0; PHP 8.5.10 ZTS, Zend Engine 4.5.10, OPcache enabled; FrankenPHP v1.12.7; no Go JIT
|-
| SQL client || Python 3.12 / PyMySQL, established TCP connections to 127.0.0.1; complete result fetching; no connection setup timed
|-
| Database fixture || Read-only consistent snapshot of wordpress_bench_v2, copied to a separate MemCP database; MariaDB InnoDB vs. MemCP sloppy engine; rebuild before measurements
|-
| Measurement order || MariaDB, MemCP, MemCP-JIT; one client, no concurrent query load
|-
| Runtime settings || Query-plan cache, autoindexing and normal optimizer enabled; TracePrint and ScanDebugging disabled while timing
|}

This is a warm read comparison, not a matched-memory or durability benchmark. The host is shared with development services; cooperating builds/tests were paused for timing, but CPUs and memory were not reserved or pinned. Autoindexing must have time to activate: the warmup count deliberately exceeds its initial observation threshold. Compiled plans, indexes and maintained group/prejoin structures may all contribute to warm results. Their cold build cost and write-maintenance cost are outside the latency table.

Cold startup and initial automatic structure building are not represented by this table. Both MemCP modes reuse the same freshly imported, rebuilt directory sequentially; persisted structures and OS caches remain available. An earlier preparation pass populated some structures before the final measured run. This is deliberately a warm comparison, not a cold-start A/B test.

=== Fixture ===

The seven captured queries exercise option lookups, plugin metadata, taxonomy joins, comment counts, search and archives. The data is synthetic, with real WordPress query shapes. It is not a million-post or concurrent-production benchmark. The source WordPress installation identifies itself as 7.1; WordPress/PHP rendering does not execute in the SQL matrix, and installed plugin versions are not performance inputs once these exact queries are captured.

{| class="wikitable"
! Table !! Contents !! Rows
|-
| wp_posts || Posts, pages, revisions and other post types || 22,063
|-
| wp_postmeta || Post metadata || 14,013
|-
| wp_comments || Comments || 6,001
|-
| wp_options || Site options || 503
|-
| wp_terms || Terms (category/tag names) || 175
|-
| wp_term_taxonomy || Term assignments to taxonomy types || 175
|-
| wp_term_relationships || Object-to-taxonomy relationships || 7,701
|}

The corpus contains 2,000 varied generated articles across 16 topics and 20,000 revisions. Titles, excerpts and bodies vary; body lengths range from 396 to 6,996 bytes (median 1,763). The search query uses realistic, present in 80 generated articles; the full query also considers other post types. Low-cardinality fields such as status and categories naturally repeat. This version-2 corpus must not be mixed with older version-1 measurements.

== Try MemCP with your PHP hosting ==

'''MemCP is faster on the substantial repeated SQL work measured here. Try it with your own PHP hosting.''' Start with a copy of your site's data and replay its frequent queries: counts, searches, archives and plugin joins are useful starting points. Compare complete results and total counts, then measure full uncached HTTP requests and representative writes. You can retain the MySQL wire interface or explore direct PHP hosting to remove transport overhead from many small calls.

WordPress is the example, not the product boundary. The same PHP-folder handler and PDO bridge can host other PHP applications; application and plugin compatibility still needs to be checked for the software you deploy.

== Measurement artifacts and exact queries ==

Local artifact directory: <code>../wordpress-db-lab/measurements/wiki-master-43d0c8140/</code>. It contains results-master.json (raw batch means and complete query results), php-latency.json (all PHP samples), manifest.json, counts.json, corpus.json, text-parity.json, server logs, and a copied benchmark.py. The SQL runner can be repeated against the same fixture; its WP_BINARY and WP_WORKTREE overrides are for separately labelled candidate checks. Candidate measurements are not included in this master matrix.

Source queries: <code>../wordpress-db-lab/profiles/queries/</code>. They are reproduced below so that the benchmark shapes, including actual ORDER BY clauses, are unambiguous.

=== Option lookup: siteurl ===
<pre>
-- Copyright (C) 2026 MemCP WordPress lab contributors
-- SPDX-License-Identifier: AGPL-3.0-or-later
SELECT option_value FROM wp_options WHERE option_name = 'siteurl' LIMIT 1;
</pre>

=== Elementor metadata join + LIMIT ===
<pre>
SELECT wp_posts.ID FROM wp_posts INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id ) WHERE 1=1 AND ( ( wp_postmeta.meta_key = '_elementor_template_type' AND wp_postmeta.meta_value = 'landing-page' ) ) AND wp_posts.post_type = 'e-landing-page' AND ((wp_posts.post_status <> 'trash' AND wp_posts.post_status <> 'auto-draft')) GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC LIMIT 0, 1;
</pre>

=== Taxonomy DISTINCT/join, ordered by name ===
<pre>
SELECT DISTINCT t.term_id, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON t.term_id = tt.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('category', 'post_tag', 'post_format') AND tr.object_id IN (1, 21993, 21982, 21971, 21960) ORDER BY t.name ASC;
</pre>

=== Comment count with join ===
<pre>
SELECT COUNT(*) FROM wp_comments LEFT JOIN wp_posts AS wp_posts_to_exclude_reviews ON comment_post_ID = wp_posts_to_exclude_reviews.ID WHERE ( comment_approved = '1' ) AND comment_type NOT IN ('note') AND comment_type != 'order_note' AND comment_type != 'webhook_delivery' AND comment_type != 'action_log' AND wp_posts_to_exclude_reviews.post_type NOT IN ('product');
</pre>

=== WordPress search + FOUND_ROWS ===
<pre>
SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts WHERE 1=1 AND ( wp_posts.ID NOT IN ( SELECT object_id FROM wp_term_relationships WHERE term_taxonomy_id IN (166) ) ) AND (((wp_posts.post_title LIKE '%realistic%') OR (wp_posts.post_excerpt LIKE '%realistic%') OR (wp_posts.post_content LIKE '%realistic%'))) AND ((wp_posts.post_type = 'attachment' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'e-floating-buttons' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'page' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'post' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'product' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private'))) GROUP BY wp_posts.ID ORDER BY wp_posts.post_title LIKE '%realistic%' DESC, wp_posts.post_date DESC LIMIT 0, 10;
</pre>

=== Yoast CAST metadata join + FOUND_ROWS ===
<pre>
SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id ) WHERE 1=1 AND ( ( wp_postmeta.meta_key = '_yoast_wpseo_linkdex' AND CAST(wp_postmeta.meta_value AS SIGNED) BETWEEN '1' AND '40' ) ) AND wp_posts.post_type = 'post' AND ((wp_posts.post_status = 'publish')) GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC LIMIT 0, 10;
</pre>

=== Monthly archive DISTINCT ===
<pre>
SELECT DISTINCT YEAR( post_date ) AS year, MONTH( post_date ) AS month FROM wp_posts WHERE post_type = 'post' AND post_status != 'auto-draft' AND post_status != 'trash' ORDER BY post_date DESC;
</pre>

Benchmark MemCP vs. MariaDB on Wordpress

2026-09-09T21:44:50Z

Carli: /* Host PHP directly: remove a round trip for every small query */

'''MemCP makes repeated WordPress database work faster.''' On current master, MemCP-JIT completes this fixture's search plus total-count query '''36.0× faster''' than MariaDB, its comment count '''21.1× faster''', and its archive query '''6.2× faster'''. MemCP-JIT has lower measured warm latency across all seven query shapes in this run.

MemCP speaks the MySQL protocol and can also '''host PHP directly in the database process'''. WordPress is our example application: the hosting interface is general purpose and is also intended for MediaWiki and other PHP software. Try MemCP with your PHP hosting and measure the queries your application actually runs.

== MariaDB vs. MemCP vs. MemCP-JIT ==

Measured on master [https://github.com/launix-de/memcp/commit/43d0c81401616fafeeb456b22f4a47d6e25361d6 43d0c8140], including the SQL literal-parameterization improvement in PR #873. All three columns use the same seven WordPress query shapes and data snapshot. These are '''warm database round trips''', not HTTP page-load times. Lower is better.

'''Dataset size:''' 22,063 rows in wp_posts (including 2,000 generated articles and 20,000 revisions), 14,013 post-metadata rows, 6,001 comments and 503 options. Taxonomy data comprises 175 terms, 175 term-taxonomy entries and 7,701 object-to-taxonomy relationships. These are the measured snapshot's table counts, not the number of rows returned by each query.

Each value is the median of 11 batch means, each containing 100 complete query executions, after 30 warmups. For SQL_CALC_FOUND_ROWS statements, the measured operation also includes the subsequent SELECT FOUND_ROWS() on the same connection.

{| class="wikitable sortable"
! Query !! MariaDB (ms) !! MemCP (ms) !! MemCP-JIT (ms) !! MariaDB / JIT !! Result check
|-
| Option lookup: siteurl || 0.079 || 0.069 || '''0.056''' || 1.39× || Equal; 1 row
|-
| Elementor metadata join + LIMIT || 0.152 || 0.108 || '''0.079''' || 1.93× || Equal; 0 rows
|-
| Taxonomy DISTINCT/join, ordered by name || 0.224 || 0.245 || '''0.142''' || 1.58× || Equal
|-
| Comment count with join || 3.240 || 0.222 || '''0.153''' || 21.14× || Equal; 1 row
|-
| WordPress search + FOUND_ROWS || 30.405 || 1.278 || '''0.844''' || 36.01× || Equal; 10 rows
|-
| Yoast CAST metadata join + FOUND_ROWS || 0.197 || 0.248 || '''0.144''' || 1.37× || Equal; 0 rows
|-
| Monthly archive DISTINCT || 0.718 || 0.168 || '''0.116''' || 6.21× || Equal; 10 rows
|}

Timing values are from master 43d0c8140. Taxonomy result equality (all 15 rows, including their order) was verified separately with [https://github.com/launix-de/memcp/pull/871 PR #871].

The two metadata queries return zero rows on this snapshot. Their times describe proving an empty result, not retrieving a populated plugin workload. Tiny differences in the smallest queries are sensitive to scheduling on this shared development machine. We do not average these seven unrelated query shapes into a single WordPress speedup.

== Host PHP directly: remove a round trip for every small query ==

'''MemCP makes repeated WordPress database work faster.''' On current master, MemCP-JIT completes this fixture's search plus total-count query '''36.0× faster''' than MariaDB, its comment count '''21.1× faster''', and its archive query '''6.2× faster'''. MemCP-JIT has lower measured warm latency across all seven query shapes in this run.

MemCP speaks the MySQL protocol and can also '''host PHP directly in the database process'''. WordPress is our example application: the hosting interface is general purpose and is also intended for MediaWiki and other PHP software. Try MemCP with your PHP hosting and measure the queries your application actually runs.

== MariaDB vs. MemCP vs. MemCP-JIT ==

Measured on master [https://github.com/launix-de/memcp/commit/43d0c81401616fafeeb456b22f4a47d6e25361d6 43d0c8140], including the SQL literal-parameterization improvement in PR #873. All three columns use the same seven WordPress query shapes and data snapshot. These are '''warm database round trips''', not HTTP page-load times. Lower is better.

'''Dataset size:''' 22,063 rows in wp_posts (including 2,000 generated articles and 20,000 revisions), 14,013 post-metadata rows, 6,001 comments and 503 options. Taxonomy data comprises 175 terms, 175 term-taxonomy entries and 7,701 object-to-taxonomy relationships. These are the measured snapshot's table counts, not the number of rows returned by each query.

Each value is the median of 11 batch means, each containing 100 complete query executions, after 30 warmups. For SQL_CALC_FOUND_ROWS statements, the measured operation also includes the subsequent SELECT FOUND_ROWS() on the same connection.

{| class="wikitable sortable"
! Query !! MariaDB (ms) !! MemCP (ms) !! MemCP-JIT (ms) !! MariaDB / JIT !! Result check
|-
| Option lookup: siteurl || 0.079 || 0.069 || '''0.056''' || 1.39× || Equal; 1 row
|-
| Elementor metadata join + LIMIT || 0.152 || 0.108 || '''0.079''' || 1.93× || Equal; 0 rows
|-
| Taxonomy DISTINCT/join, ordered by name || 0.224 || 0.245 || '''0.142''' || 1.58× || Equal
|-
| Comment count with join || 3.240 || 0.222 || '''0.153''' || 21.14× || Equal; 1 row
|-
| WordPress search + FOUND_ROWS || 30.405 || 1.278 || '''0.844''' || 36.01× || Equal; 10 rows
|-
| Yoast CAST metadata join + FOUND_ROWS || 0.197 || 0.248 || '''0.144''' || 1.37× || Equal; 0 rows
|-
| Monthly archive DISTINCT || 0.718 || 0.168 || '''0.116''' || 6.21× || Equal; 10 rows
|}

Timing values are from master 43d0c8140. Taxonomy result equality (all 15 rows, including their order) was verified separately with [https://github.com/launix-de/memcp/pull/871 PR #871].

The two metadata queries return zero rows on this snapshot. Their times describe proving an empty result, not retrieving a populated plugin workload. Tiny differences in the smallest queries are sensitive to scheduling on this shared development machine. We do not average these seven unrelated query shapes into a single WordPress speedup.

== Host PHP directly: remove a round trip for every small query ==

The optional FrankenPHP dependency supplies the PHP runtime; only MemCP's own bridge and PDO addon live in this repository. PHP requests use ZTS threads, with OPcache enabled. PHP folders are attached through the existing HTTP handler, alongside the SQL API and dashboard.

With a PHP-enabled build and an external ZTS PHP development installation:

<pre>
make php PHP_CONFIG=/path/to/zts/php-config
./memcp-php --no-repl -data /path/to/data --api-port=8098 --mysql-port=3307 --serve=/path/to/php-app lib/main.scm
</pre>

The application is mounted at /; /dashboard remains available. Scheme code can mount additional applications through servePHP. No WordPress source is bundled into MemCP.

The PDO addon supports authenticated memcp: connections and retains ordinary MySQL wire connections. A mysql: connection to localhost/127.0.0.1 and this MemCP server's own MySQL port can be routed internally; other destinations continue using their original driver. Both paths require database credentials. WordPress's use of mysqli is a separate connector question: PDO routing alone is not a claim that every WordPress installation or plugin uses PDO.

A separate microbenchmark measures SELECT 1 inside PHP, using an already established connection. Each round performs 100 warmups and 2,000 measured executions; seven rounds alternate the two transports. The timer includes query(), fetchColumn(), and closeCursor(), but excludes HTTP, connection establishment, and PHP startup. Pdo\Mysql::connect forces the real wire driver for the control, so automatic PDO routing cannot accidentally turn both measurements into RAM calls.

{| class="wikitable"
! PHP SELECT 1 transport !! Median of round medians
|-
| MySQL wire over local TCP || 37.19 µs
|-
| MemCP in-process PDO || '''11.57 µs'''
|}

This removes '''69% of the measured latency''', or 25.61 µs per tiny query. At 100 such queries per request, the arithmetic saving is approximately 2.56 ms; at 1,000 queries it is 25.61 ms. Those are extrapolations, not measured WordPress page-load improvements.

For a workload-specific budgeting example, halving 100–200 µs of overhead would save '''50–100 µs per call (50%)'''. Across 100 database calls, that would add up to 5–10 ms per request. Those figures illustrate how small savings accumulate; they were not measured here. The measured transport comparison above is the evidence for this setup; measure your own query mix before assigning a per-request saving.

== Reproducible setup ==

{| class="wikitable"
! Item !! Configuration
|-
| Measurement date || 9 September 2026 (Europe/Berlin)
|-
| CPU || AMD Ryzen 9 7900X3D, 12 physical cores / 24 logical CPUs
|-
| Memory / OS || Approximately 62 GiB RAM; Ubuntu 24.04.4 LTS, x86-64
|-
| MemCP || Master 43d0c81401616fafeeb456b22f4a47d6e25361d6; unchanged source for both SQL builds
|-
| SQL compiler/toolchain || Patched Go 1.27.0, revision 84fe25ee45468f5e3920c11c24e2bbb926fc3a72; GOEXPERIMENT empty vs. GOEXPERIMENT=jit
|-
| MariaDB || 10.11.14; query cache OFF; InnoDB buffer pool 128 MiB
|-
| PHP microbenchmark || Same MemCP master, built with Go 1.26.0; PHP 8.5.10 ZTS, Zend Engine 4.5.10, OPcache enabled; FrankenPHP v1.12.7; no Go JIT
|-
| SQL client || Python 3.12 / PyMySQL, established TCP connections to 127.0.0.1; complete result fetching; no connection setup timed
|-
| Database fixture || Read-only consistent snapshot of wordpress_bench_v2, copied to a separate MemCP database; MariaDB InnoDB vs. MemCP sloppy engine; rebuild before measurements
|-
| Measurement order || MariaDB, MemCP, MemCP-JIT; one client, no concurrent query load
|-
| Runtime settings || Query-plan cache, autoindexing and normal optimizer enabled; TracePrint and ScanDebugging disabled while timing
|}

This is a warm read comparison, not a matched-memory or durability benchmark. The host is shared with development services; cooperating builds/tests were paused for timing, but CPUs and memory were not reserved or pinned. Autoindexing must have time to activate: the warmup count deliberately exceeds its initial observation threshold. Compiled plans, indexes and maintained group/prejoin structures may all contribute to warm results. Their cold build cost and write-maintenance cost are outside the latency table.

Cold startup and initial automatic structure building are not represented by this table. Both MemCP modes reuse the same freshly imported, rebuilt directory sequentially; persisted structures and OS caches remain available. An earlier preparation pass populated some structures before the final measured run. This is deliberately a warm comparison, not a cold-start A/B test.

=== Fixture ===

The seven captured queries exercise option lookups, plugin metadata, taxonomy joins, comment counts, search and archives. The data is synthetic, with real WordPress query shapes. It is not a million-post or concurrent-production benchmark. The source WordPress installation identifies itself as 7.1; WordPress/PHP rendering does not execute in the SQL matrix, and installed plugin versions are not performance inputs once these exact queries are captured.

{| class="wikitable"
! Table !! Contents !! Rows
|-
| wp_posts || Posts, pages, revisions and other post types || 22,063
|-
| wp_postmeta || Post metadata || 14,013
|-
| wp_comments || Comments || 6,001
|-
| wp_options || Site options || 503
|-
| wp_terms || Terms (category/tag names) || 175
|-
| wp_term_taxonomy || Term assignments to taxonomy types || 175
|-
| wp_term_relationships || Object-to-taxonomy relationships || 7,701
|}

The corpus contains 2,000 varied generated articles across 16 topics and 20,000 revisions. Titles, excerpts and bodies vary; body lengths range from 396 to 6,996 bytes (median 1,763). The search query uses realistic, present in 80 generated articles; the full query also considers other post types. Low-cardinality fields such as status and categories naturally repeat. This version-2 corpus must not be mixed with older version-1 measurements.

== Try MemCP with your PHP hosting ==

'''MemCP is faster on the substantial repeated SQL work measured here. Try it with your own PHP hosting.''' Start with a copy of your site's data and replay its frequent queries: counts, searches, archives and plugin joins are useful starting points. Compare complete results and total counts, then measure full uncached HTTP requests and representative writes. You can retain the MySQL wire interface or explore direct PHP hosting to remove transport overhead from many small calls.

WordPress is the example, not the product boundary. The same PHP-folder handler and PDO bridge can host other PHP applications; application and plugin compatibility still needs to be checked for the software you deploy.

== Measurement artifacts and exact queries ==

Local artifact directory: <code>../wordpress-db-lab/measurements/wiki-master-43d0c8140/</code>. It contains results-master.json (raw batch means and complete query results), php-latency.json (all PHP samples), manifest.json, counts.json, corpus.json, text-parity.json, server logs, and a copied benchmark.py. The SQL runner can be repeated against the same fixture; its WP_BINARY and WP_WORKTREE overrides are for separately labelled candidate checks. Candidate measurements are not included in this master matrix.

Source queries: <code>../wordpress-db-lab/profiles/queries/</code>. They are reproduced below so that the benchmark shapes, including actual ORDER BY clauses, are unambiguous.

=== Option lookup: siteurl ===
<pre>
-- Copyright (C) 2026 MemCP WordPress lab contributors
-- SPDX-License-Identifier: AGPL-3.0-or-later
SELECT option_value FROM wp_options WHERE option_name = 'siteurl' LIMIT 1;
</pre>

=== Elementor metadata join + LIMIT ===
<pre>
SELECT wp_posts.ID FROM wp_posts INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id ) WHERE 1=1 AND ( ( wp_postmeta.meta_key = '_elementor_template_type' AND wp_postmeta.meta_value = 'landing-page' ) ) AND wp_posts.post_type = 'e-landing-page' AND ((wp_posts.post_status <> 'trash' AND wp_posts.post_status <> 'auto-draft')) GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC LIMIT 0, 1;
</pre>

=== Taxonomy DISTINCT/join, ordered by name ===
<pre>
SELECT DISTINCT t.term_id, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON t.term_id = tt.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('category', 'post_tag', 'post_format') AND tr.object_id IN (1, 21993, 21982, 21971, 21960) ORDER BY t.name ASC;
</pre>

=== Comment count with join ===
<pre>
SELECT COUNT(*) FROM wp_comments LEFT JOIN wp_posts AS wp_posts_to_exclude_reviews ON comment_post_ID = wp_posts_to_exclude_reviews.ID WHERE ( comment_approved = '1' ) AND comment_type NOT IN ('note') AND comment_type != 'order_note' AND comment_type != 'webhook_delivery' AND comment_type != 'action_log' AND wp_posts_to_exclude_reviews.post_type NOT IN ('product');
</pre>

=== WordPress search + FOUND_ROWS ===
<pre>
SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts WHERE 1=1 AND ( wp_posts.ID NOT IN ( SELECT object_id FROM wp_term_relationships WHERE term_taxonomy_id IN (166) ) ) AND (((wp_posts.post_title LIKE '%realistic%') OR (wp_posts.post_excerpt LIKE '%realistic%') OR (wp_posts.post_content LIKE '%realistic%'))) AND ((wp_posts.post_type = 'attachment' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'e-floating-buttons' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'page' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'post' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'product' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private'))) GROUP BY wp_posts.ID ORDER BY wp_posts.post_title LIKE '%realistic%' DESC, wp_posts.post_date DESC LIMIT 0, 10;
</pre>

=== Yoast CAST metadata join + FOUND_ROWS ===
<pre>
SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id ) WHERE 1=1 AND ( ( wp_postmeta.meta_key = '_yoast_wpseo_linkdex' AND CAST(wp_postmeta.meta_value AS SIGNED) BETWEEN '1' AND '40' ) ) AND wp_posts.post_type = 'post' AND ((wp_posts.post_status = 'publish')) GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC LIMIT 0, 10;
</pre>

=== Monthly archive DISTINCT ===
<pre>
SELECT DISTINCT YEAR( post_date ) AS year, MONTH( post_date ) AS month FROM wp_posts WHERE post_type = 'post' AND post_status != 'auto-draft' AND post_status != 'trash' ORDER BY post_date DESC;
</pre>

== Reproducible setup ==

{| class="wikitable"
! Item !! Configuration
|-
| Measurement date || 9 September 2026 (Europe/Berlin)
|-
| CPU || AMD Ryzen 9 7900X3D, 12 physical cores / 24 logical CPUs
|-
| Memory / OS || Approximately 62 GiB RAM; Ubuntu 24.04.4 LTS, x86-64
|-
| MemCP || Master 43d0c81401616fafeeb456b22f4a47d6e25361d6; unchanged source for both SQL builds
|-
| SQL compiler/toolchain || Patched Go 1.27.0, revision 84fe25ee45468f5e3920c11c24e2bbb926fc3a72; GOEXPERIMENT empty vs. GOEXPERIMENT=jit
|-
| MariaDB || 10.11.14; query cache OFF; InnoDB buffer pool 128 MiB
|-
| PHP microbenchmark || Same MemCP master, built with Go 1.26.0; PHP 8.5.10 ZTS, Zend Engine 4.5.10, OPcache enabled; FrankenPHP v1.12.7; no Go JIT
|-
| SQL client || Python 3.12 / PyMySQL, established TCP connections to 127.0.0.1; complete result fetching; no connection setup timed
|-
| Database fixture || Read-only consistent snapshot of wordpress_bench_v2, copied to a separate MemCP database; MariaDB InnoDB vs. MemCP sloppy engine; rebuild before measurements
|-
| Measurement order || MariaDB, MemCP, MemCP-JIT; one client, no concurrent query load
|-
| Runtime settings || Query-plan cache, autoindexing and normal optimizer enabled; TracePrint and ScanDebugging disabled while timing
|}

This is a warm read comparison, not a matched-memory or durability benchmark. The host is shared with development services; cooperating builds/tests were paused for timing, but CPUs and memory were not reserved or pinned. Autoindexing must have time to activate: the warmup count deliberately exceeds its initial observation threshold. Compiled plans, indexes and maintained group/prejoin structures may all contribute to warm results. Their cold build cost and write-maintenance cost are outside the latency table.

Cold startup and initial automatic structure building are not represented by this table. Both MemCP modes reuse the same freshly imported, rebuilt directory sequentially; persisted structures and OS caches remain available. An earlier preparation pass populated some structures before the final measured run. This is deliberately a warm comparison, not a cold-start A/B test.

=== Fixture ===

The seven captured queries exercise option lookups, plugin metadata, taxonomy joins, comment counts, search and archives. The data is synthetic, with real WordPress query shapes. It is not a million-post or concurrent-production benchmark. The source WordPress installation identifies itself as 7.1; WordPress/PHP rendering does not execute in the SQL matrix, and installed plugin versions are not performance inputs once these exact queries are captured.

{| class="wikitable"
! Table !! Contents !! Rows
|-
| wp_posts || Posts, pages, revisions and other post types || 22,063
|-
| wp_postmeta || Post metadata || 14,013
|-
| wp_comments || Comments || 6,001
|-
| wp_options || Site options || 503
|-
| wp_terms || Terms (category/tag names) || 175
|-
| wp_term_taxonomy || Term assignments to taxonomy types || 175
|-
| wp_term_relationships || Object-to-taxonomy relationships || 7,701
|}

The corpus contains 2,000 varied generated articles across 16 topics and 20,000 revisions. Titles, excerpts and bodies vary; body lengths range from 396 to 6,996 bytes (median 1,763). The search query uses realistic, present in 80 generated articles; the full query also considers other post types. Low-cardinality fields such as status and categories naturally repeat. This version-2 corpus must not be mixed with older version-1 measurements.

== Try MemCP with your PHP hosting ==

'''MemCP is faster on the substantial repeated SQL work measured here. Try it with your own PHP hosting.''' Start with a copy of your site's data and replay its frequent queries: counts, searches, archives and plugin joins are useful starting points. Compare complete results and total counts, then measure full uncached HTTP requests and representative writes. You can retain the MySQL wire interface or explore direct PHP hosting to remove transport overhead from many small calls.

WordPress is the example, not the product boundary. The same PHP-folder handler and PDO bridge can host other PHP applications; application and plugin compatibility still needs to be checked for the software you deploy.

== Measurement artifacts and exact queries ==

Local artifact directory: <code>../wordpress-db-lab/measurements/wiki-master-43d0c8140/</code>. It contains results-master.json (raw batch means and complete query results), php-latency.json (all PHP samples), manifest.json, counts.json, corpus.json, text-parity.json, server logs, and a copied benchmark.py. The SQL runner can be repeated against the same fixture; its WP_BINARY and WP_WORKTREE overrides are for separately labelled candidate checks. Candidate measurements are not included in this master matrix.

Source queries: <code>../wordpress-db-lab/profiles/queries/</code>. They are reproduced below so that the benchmark shapes, including actual ORDER BY clauses, are unambiguous.

=== Option lookup: siteurl ===
<syntaxhighlight lang="sql">
-- Copyright (C) 2026 MemCP WordPress lab contributors
-- SPDX-License-Identifier: AGPL-3.0-or-later
SELECT option_value FROM wp_options WHERE option_name = 'siteurl' LIMIT 1;
</syntaxhighlight>

=== Elementor metadata join + LIMIT ===
<syntaxhighlight lang="sql">
SELECT wp_posts.ID FROM wp_posts INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id ) WHERE 1=1 AND ( ( wp_postmeta.meta_key = '_elementor_template_type' AND wp_postmeta.meta_value = 'landing-page' ) ) AND wp_posts.post_type = 'e-landing-page' AND ((wp_posts.post_status <> 'trash' AND wp_posts.post_status <> 'auto-draft')) GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC LIMIT 0, 1;
</syntaxhighlight>

=== Taxonomy DISTINCT/join, ordered by name ===
<syntaxhighlight lang="sql">
SELECT DISTINCT t.term_id, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON t.term_id = tt.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('category', 'post_tag', 'post_format') AND tr.object_id IN (1, 21993, 21982, 21971, 21960) ORDER BY t.name ASC;
</syntaxhighlight>

=== Comment count with join ===
<syntaxhighlight lang="sql">
SELECT COUNT(*) FROM wp_comments LEFT JOIN wp_posts AS wp_posts_to_exclude_reviews ON comment_post_ID = wp_posts_to_exclude_reviews.ID WHERE ( comment_approved = '1' ) AND comment_type NOT IN ('note') AND comment_type != 'order_note' AND comment_type != 'webhook_delivery' AND comment_type != 'action_log' AND wp_posts_to_exclude_reviews.post_type NOT IN ('product');
</syntaxhighlight>

=== WordPress search + FOUND_ROWS ===
<syntaxhighlight lang="sql">
SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts WHERE 1=1 AND ( wp_posts.ID NOT IN ( SELECT object_id FROM wp_term_relationships WHERE term_taxonomy_id IN (166) ) ) AND (((wp_posts.post_title LIKE '%realistic%') OR (wp_posts.post_excerpt LIKE '%realistic%') OR (wp_posts.post_content LIKE '%realistic%'))) AND ((wp_posts.post_type = 'attachment' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'e-floating-buttons' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'page' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'post' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'product' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private'))) GROUP BY wp_posts.ID ORDER BY wp_posts.post_title LIKE '%realistic%' DESC, wp_posts.post_date DESC LIMIT 0, 10;
</syntaxhighlight>

=== Yoast CAST metadata join + FOUND_ROWS ===
<syntaxhighlight lang="sql">
SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id ) WHERE 1=1 AND ( ( wp_postmeta.meta_key = '_yoast_wpseo_linkdex' AND CAST(wp_postmeta.meta_value AS SIGNED) BETWEEN '1' AND '40' ) ) AND wp_posts.post_type = 'post' AND ((wp_posts.post_status = 'publish')) GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC LIMIT 0, 10;
</syntaxhighlight>

=== Monthly archive DISTINCT ===
<syntaxhighlight lang="sql">
SELECT DISTINCT YEAR( post_date ) AS year, MONTH( post_date ) AS month FROM wp_posts WHERE post_type = 'post' AND post_status != 'auto-draft' AND post_status != 'trash' ORDER BY post_date DESC;
</syntaxhighlight>

Benchmark MemCP vs. MariaDB on Wordpress

2026-09-09T21:43:07Z

Carli:

Benchmark MemCP vs. MariaDB on Wordpress

2026-09-09T21:39:03Z

Carli:

Benchmark MemCP vs. MariaDB on Wordpress

2026-09-09T21:36:55Z

Carli:

Benchmark MemCP vs. MariaDB on Wordpress

2026-09-09T21:36:38Z

Carli:

= Benchmark MemCP vs. MariaDB on Wordpress =

'''MemCP makes repeated WordPress database work faster.''' On current master, MemCP-JIT completes this fixture's search plus total-count query '''36.0× faster''' than MariaDB, its comment count '''21.1× faster''', and its archive query '''6.2× faster'''. All six query shapes with matching results have lower warm latency with JIT in this run. The seventh, a taxonomy query, is retained in the matrix with a clearly marked collation mismatch.

MemCP speaks the MySQL protocol and can also '''host PHP directly in the database process'''. WordPress is our example application: the hosting interface is general purpose and is also intended for MediaWiki and other PHP software. Try MemCP with your PHP hosting and measure the queries your application actually runs.

== MariaDB vs. MemCP vs. MemCP-JIT ==

Measured on master [https://github.com/launix-de/memcp/commit/43d0c81401616fafeeb456b22f4a47d6e25361d6 43d0c8140], including the SQL literal-parameterization improvement in PR #873. All three columns use the same seven WordPress query shapes and data snapshot. These are '''warm database round trips''', not HTTP page-load times. Lower is better.

Each value is the median of 11 batch means, each containing 100 complete query executions, after 30 warmups. For SQL_CALC_FOUND_ROWS statements, the measured operation also includes the subsequent SELECT FOUND_ROWS() on the same connection.

{| class="wikitable sortable"
! Query !! MariaDB (ms) !! MemCP (ms) !! MemCP-JIT (ms) !! MariaDB / JIT !! Result check
|-
| Option lookup: siteurl || 0.079 || 0.069 || '''0.056''' || 1.39× || Equal; 1 row
|-
| Elementor metadata join + LIMIT || 0.152 || 0.108 || '''0.079''' || 1.93× || Equal; 0 rows
|-
| Taxonomy DISTINCT/join, ordered by name || 0.224 || 0.245 || '''0.142''' || Diagnostic only || See correctness notes below
|-
| Comment count with join || 3.240 || 0.222 || '''0.153''' || 21.14× || Equal; 1 row
|-
| WordPress search + FOUND_ROWS || 30.405 || 1.278 || '''0.844''' || 36.01× || Equal; 10 rows
|-
| Yoast CAST metadata join + FOUND_ROWS || 0.197 || 0.248 || '''0.144''' || 1.37× || Equal; 0 rows
|-
| Monthly archive DISTINCT || 0.718 || 0.168 || '''0.116''' || 6.21× || Equal; 10 rows
|}

The two metadata queries return zero rows on this snapshot. Their times describe proving an empty result, not retrieving a populated plugin workload. Tiny differences in the smallest queries are sensitive to scheduling on this shared development machine. We do not average these seven unrelated query shapes into a single WordPress speedup.

== Host PHP directly: remove a round trip for every small query ==

The optional FrankenPHP dependency supplies the PHP runtime; only MemCP's own bridge and PDO addon live in this repository. PHP requests use ZTS threads, with OPcache enabled. PHP folders are attached through the existing HTTP handler, alongside the SQL API and dashboard.

With a PHP-enabled build and an external ZTS PHP development installation:

<syntaxhighlight lang="bash">
make php PHP_CONFIG=/path/to/zts/php-config
./memcp-php --no-repl -data /path/to/data --api-port=8098 --mysql-port=3307 --serve=/path/to/php-app lib/main.scm
</syntaxhighlight>

The application is mounted at /; /dashboard remains available. Scheme code can mount additional applications through servePHP. No WordPress source is bundled into MemCP.

The PDO addon supports authenticated memcp: connections and retains ordinary MySQL wire connections. A mysql: connection to localhost/127.0.0.1 and this MemCP server's own MySQL port can be routed internally; other destinations continue using their original driver. Both paths require database credentials. WordPress's use of mysqli is a separate connector question: PDO routing alone is not a claim that every WordPress installation or plugin uses PDO.

A separate microbenchmark measures SELECT 1 inside PHP, using an already established connection. Each round performs 100 warmups and 2,000 measured executions; seven rounds alternate the two transports. The timer includes query(), fetchColumn(), and closeCursor(), but excludes HTTP, connection establishment, and PHP startup. Pdo\Mysql::connect forces the real wire driver for the control, so automatic PDO routing cannot accidentally turn both measurements into RAM calls.

{| class="wikitable"
! PHP SELECT 1 transport !! Median of round medians
|-
| MySQL wire over local TCP || 37.19 µs
|-
| MemCP in-process PDO || '''11.57 µs'''
|}

This removes '''69% of the measured latency''', or 25.61 µs per tiny query. At 100 such queries per request, the arithmetic saving is approximately 2.56 ms; at 1,000 queries it is 25.61 ms. Those are extrapolations, not measured WordPress page-load improvements.

For a workload-specific budgeting example, halving 100–200 µs of overhead would save '''50–100 µs per call (50%)'''. Across 100 database calls, that would add up to 5–10 ms per request. Those figures illustrate how small savings accumulate; they were not measured here. The measured transport comparison above is the evidence for this setup; measure your own query mix before assigning a per-request saving.

== Reproducible setup ==

{| class="wikitable"
! Item !! Configuration
|-
| Measurement date || 9 September 2026 (Europe/Berlin)
|-
| CPU || AMD Ryzen 9 7900X3D, 12 physical cores / 24 logical CPUs
|-
| Memory / OS || Approximately 62 GiB RAM; Ubuntu 24.04.4 LTS, x86-64
|-
| MemCP || Master 43d0c81401616fafeeb456b22f4a47d6e25361d6; unchanged source for both SQL builds
|-
| SQL compiler/toolchain || Patched Go 1.27.0, revision 84fe25ee45468f5e3920c11c24e2bbb926fc3a72; GOEXPERIMENT empty vs. GOEXPERIMENT=jit
|-
| MariaDB || 10.11.14; query cache OFF; InnoDB buffer pool 128 MiB
|-
| PHP microbenchmark || Same MemCP master, built with Go 1.26.0; PHP 8.5.10 ZTS, Zend Engine 4.5.10, OPcache enabled; FrankenPHP v1.12.7; no Go JIT
|-
| SQL client || Python 3.12 / PyMySQL, established TCP connections to 127.0.0.1; complete result fetching; no connection setup timed
|-
| Database fixture || Read-only consistent snapshot of wordpress_bench_v2, copied to a separate MemCP database; MariaDB InnoDB vs. MemCP sloppy engine; rebuild before measurements
|-
| Measurement order || MariaDB, MemCP, MemCP-JIT; one client, no concurrent query load
|-
| Runtime settings || Query-plan cache, autoindexing and normal optimizer enabled; TracePrint and ScanDebugging disabled while timing
|}

This is a warm read comparison, not a matched-memory or durability benchmark. The host is shared with development services; cooperating builds/tests were paused for timing, but CPUs and memory were not reserved or pinned. Autoindexing must have time to activate: the warmup count deliberately exceeds its initial observation threshold. Compiled plans, indexes and maintained group/prejoin structures may all contribute to warm results. Their cold build cost and write-maintenance cost are outside the latency table.

Cold startup and initial automatic structure building are not represented by this table. Both MemCP modes reuse the same freshly imported, rebuilt directory sequentially; persisted structures and OS caches remain available. An earlier preparation pass populated some structures before the final measured run. This is deliberately a warm comparison, not a cold-start A/B test.

=== Fixture ===

The seven captured queries exercise option lookups, plugin metadata, taxonomy joins, comment counts, search and archives. The data is synthetic, with real WordPress query shapes. It is not a million-post or concurrent-production benchmark. The source WordPress installation identifies itself as 7.1; WordPress/PHP rendering does not execute in the SQL matrix, and installed plugin versions are not performance inputs once these exact queries are captured.

{| class="wikitable"
! Table !! Rows
|-
| wp_posts || 22,063
|-
| wp_postmeta || 14,013
|-
| wp_comments || 6,001
|-
| wp_options || 503
|-
| wp_terms || 175
|-
| wp_term_taxonomy || 175
|-
| wp_term_relationships || 7,701
|}

The corpus contains 2,000 varied generated articles across 16 topics and 20,000 revisions. Titles, excerpts and bodies vary; body lengths range from 396 to 6,996 bytes (median 1,763). The search query uses realistic, present in 80 generated articles; the full query also considers other post types. Low-cardinality fields such as status and categories naturally repeat. This version-2 corpus must not be mixed with older version-1 measurements.

== Correctness and scope ==

Six of the seven exact queries return matching results in all three modes, including the total counts: '''83''' for the search and '''0''' for the Yoast query. After import, rebuild and restart, all 22,063 post IDs, titles, excerpts and body texts also match a canonical SHA-256 comparison. The previously reported blob-content mismatch and search total of 82 were '''not reproduced in this fresh master run'''; they are not carried forward as current failures.

The remaining difference is the taxonomy query. Its exact captured SQL '''does contain ORDER BY t.name ASC'''. Both engines return the same 15 projected rows, but in a different name order. MariaDB exposes wp_terms.name as utf8mb4_unicode_520_ci; this fresh MemCP import exposes utf8mb4. Therefore its timing is diagnostic and excluded from accepted speedup claims. This statement applies to this saved query, not to every WordPress taxonomy query. Without an ORDER BY, a different row order alone would not be a correctness failure.

This benchmark does not include the unmerged type/collation fix in PR #871. It reports current master as measured. The taxonomy collation discrepancy remains a migration issue for this schema and should be resolved before relying on equivalent ordering.

These are query-level correctness checks and a post-text integrity check, not an exhaustive application compatibility test. INSERT/UPDATE workloads, parallel PHP requests, maintenance costs, cold starts and HTTP page-load latency are outside this matrix.

== Try MemCP with your PHP hosting ==

'''MemCP is faster on the substantial repeated SQL work measured here. Try it with your own PHP hosting.''' Start with a copy of your site's data and replay its frequent queries: counts, searches, archives and plugin joins are useful starting points. Compare complete results and total counts, then measure full uncached HTTP requests and representative writes. You can retain the MySQL wire interface or explore direct PHP hosting to remove transport overhead from many small calls.

WordPress is the example, not the product boundary. The same PHP-folder handler and PDO bridge can host other PHP applications; application and plugin compatibility still needs to be checked for the software you deploy.

== Measurement artifacts and exact queries ==

Local artifact directory: <code>../wordpress-db-lab/measurements/wiki-master-43d0c8140/</code>. It contains results-master.json (raw batch means and complete query results), php-latency.json (all PHP samples), manifest.json, counts.json, corpus.json, text-parity.json, server logs, and a copied benchmark.py. The SQL runner can be repeated against the same fixture; its WP_BINARY and WP_WORKTREE overrides are for separately labelled candidate checks. Candidate measurements are not included in this master matrix.

Source queries: <code>../wordpress-db-lab/profiles/queries/</code>. They are reproduced below so that the benchmark shapes, including actual ORDER BY clauses, are unambiguous.

=== Option lookup: siteurl ===
<syntaxhighlight lang="sql">
-- Copyright (C) 2026 MemCP WordPress lab contributors
-- SPDX-License-Identifier: AGPL-3.0-or-later
SELECT option_value FROM wp_options WHERE option_name = 'siteurl' LIMIT 1;
</syntaxhighlight>

=== Elementor metadata join + LIMIT ===
<syntaxhighlight lang="sql">
SELECT wp_posts.ID FROM wp_posts INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id ) WHERE 1=1 AND ( ( wp_postmeta.meta_key = '_elementor_template_type' AND wp_postmeta.meta_value = 'landing-page' ) ) AND wp_posts.post_type = 'e-landing-page' AND ((wp_posts.post_status <> 'trash' AND wp_posts.post_status <> 'auto-draft')) GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC LIMIT 0, 1;
</syntaxhighlight>

=== Taxonomy DISTINCT/join, ordered by name ===
<syntaxhighlight lang="sql">
SELECT DISTINCT t.term_id, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON t.term_id = tt.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('category', 'post_tag', 'post_format') AND tr.object_id IN (1, 21993, 21982, 21971, 21960) ORDER BY t.name ASC;
</syntaxhighlight>

=== Comment count with join ===
<syntaxhighlight lang="sql">
SELECT COUNT(*) FROM wp_comments LEFT JOIN wp_posts AS wp_posts_to_exclude_reviews ON comment_post_ID = wp_posts_to_exclude_reviews.ID WHERE ( comment_approved = '1' ) AND comment_type NOT IN ('note') AND comment_type != 'order_note' AND comment_type != 'webhook_delivery' AND comment_type != 'action_log' AND wp_posts_to_exclude_reviews.post_type NOT IN ('product');
</syntaxhighlight>

=== WordPress search + FOUND_ROWS ===
<syntaxhighlight lang="sql">
SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts WHERE 1=1 AND ( wp_posts.ID NOT IN ( SELECT object_id FROM wp_term_relationships WHERE term_taxonomy_id IN (166) ) ) AND (((wp_posts.post_title LIKE '%realistic%') OR (wp_posts.post_excerpt LIKE '%realistic%') OR (wp_posts.post_content LIKE '%realistic%'))) AND ((wp_posts.post_type = 'attachment' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'e-floating-buttons' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'page' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'post' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'product' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private'))) GROUP BY wp_posts.ID ORDER BY wp_posts.post_title LIKE '%realistic%' DESC, wp_posts.post_date DESC LIMIT 0, 10;
</syntaxhighlight>

=== Yoast CAST metadata join + FOUND_ROWS ===
<syntaxhighlight lang="sql">
SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id ) WHERE 1=1 AND ( ( wp_postmeta.meta_key = '_yoast_wpseo_linkdex' AND CAST(wp_postmeta.meta_value AS SIGNED) BETWEEN '1' AND '40' ) ) AND wp_posts.post_type = 'post' AND ((wp_posts.post_status = 'publish')) GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC LIMIT 0, 10;
</syntaxhighlight>

=== Monthly archive DISTINCT ===
<syntaxhighlight lang="sql">
SELECT DISTINCT YEAR( post_date ) AS year, MONTH( post_date ) AS month FROM wp_posts WHERE post_type = 'post' AND post_status != 'auto-draft' AND post_status != 'trash' ORDER BY post_date DESC;
</syntaxhighlight>

Benchmark MemCP vs. MariaDB on Wordpress

2026-09-09T21:34:24Z

Carli:

= Benchmark MemCP vs. MariaDB on Wordpress =

'''MemCP makes repeated WordPress database work faster.''' On current master, MemCP-JIT completes this fixture's search plus total-count query '''36.0× faster''' than MariaDB, its comment count '''21.1× faster''', and its archive query '''6.2× faster'''. All six query shapes with matching results have lower warm latency with JIT in this run. The seventh, a taxonomy query, is retained in the matrix with a clearly marked collation mismatch.

MemCP speaks the MySQL protocol and can also '''host PHP directly in the database process'''. WordPress is our example application: the hosting interface is general purpose and is also intended for MediaWiki and other PHP software. Try MemCP with your PHP hosting and measure the queries your application actually runs.

== MariaDB vs. MemCP vs. MemCP-JIT ==

Measured on master [https://github.com/launix-de/memcp/commit/43d0c81401616fafeeb456b22f4a47d6e25361d6 43d0c8140], including the SQL literal-parameterization improvement in PR #873. All three columns use the same seven WordPress query shapes and data snapshot. These are '''warm database round trips''', not HTTP page-load times. Lower is better.

Each value is the median of 11 batch means, each containing 100 complete query executions, after 30 warmups. For SQL_CALC_FOUND_ROWS statements, the measured operation also includes the subsequent SELECT FOUND_ROWS() on the same connection.

{| class="wikitable sortable"
! Query !! MariaDB (ms) !! MemCP (ms) !! MemCP-JIT (ms) !! MariaDB / JIT !! Result check
|-
| Option lookup: siteurl || 0.079 || 0.069 || '''0.056''' || 1.39× || Equal; 1 row
|-
| Elementor metadata join + LIMIT || 0.152 || 0.108 || '''0.079''' || 1.93× || Equal; 0 rows
|-
| Taxonomy DISTINCT/join, ordered by name || 0.224 || 0.245 || '''0.142''' || Diagnostic only || See correctness notes below
|-
| Comment count with join || 3.240 || 0.222 || '''0.153''' || 21.14× || Equal; 1 row
|-
| WordPress search + FOUND_ROWS || 30.405 || 1.278 || '''0.844''' || 36.01× || Equal; 10 rows
|-
| Yoast CAST metadata join + FOUND_ROWS || 0.197 || 0.248 || '''0.144''' || 1.37× || Equal; 0 rows
|-
| Monthly archive DISTINCT || 0.718 || 0.168 || '''0.116''' || 6.21× || Equal; 10 rows
|}

The two metadata queries return zero rows on this snapshot. Their times describe proving an empty result, not retrieving a populated plugin workload. Tiny differences in the smallest queries are sensitive to scheduling on this shared development machine. We do not average these seven unrelated query shapes into a single WordPress speedup.

== Host PHP directly: remove a round trip for every small query ==

The optional FrankenPHP dependency supplies the PHP runtime; only MemCP's own bridge and PDO addon live in this repository. PHP requests use ZTS threads, with OPcache enabled. PHP folders are attached through the existing HTTP handler, alongside the SQL API and dashboard.

With a PHP-enabled build and an external ZTS PHP development installation:

<syntaxhighlight lang="bash">
make php PHP_CONFIG=/path/to/zts/php-config
./memcp-php --no-repl -data /path/to/data --api-port=8098 --mysql-port=3307 --serve=/path/to/php-app lib/main.scm
</syntaxhighlight>

The application is mounted at /; /dashboard remains available. Scheme code can mount additional applications through servePHP. No WordPress source is bundled into MemCP.

The PDO addon supports authenticated memcp: connections and retains ordinary MySQL wire connections. A mysql: connection to localhost/127.0.0.1 and this MemCP server's own MySQL port can be routed internally; other destinations continue using their original driver. Both paths require database credentials. WordPress's use of mysqli is a separate connector question: PDO routing alone is not a claim that every WordPress installation or plugin uses PDO.

A separate microbenchmark measures SELECT 1 inside PHP, using an already established connection. Each round performs 100 warmups and 2,000 measured executions; seven rounds alternate the two transports. The timer includes query(), fetchColumn(), and closeCursor(), but excludes HTTP, connection establishment, and PHP startup. Pdo\Mysql::connect forces the real wire driver for the control, so automatic PDO routing cannot accidentally turn both measurements into RAM calls.

{| class="wikitable"
! PHP SELECT 1 transport !! Median of round medians
|-
| MySQL wire over local TCP || 37.19 µs
|-
| MemCP in-process PDO || '''11.57 µs'''
|}

This removes '''69% of the measured latency''', or 25.61 µs per tiny query. At 100 such queries per request, the arithmetic saving is approximately 2.56 ms; at 1,000 queries it is 25.61 ms. Those are extrapolations, not measured WordPress page-load improvements.

For a workload-specific budgeting example, halving 100–200 µs of overhead would save '''50–100 µs per call (50%)'''. Across 100 database calls, that would add up to 5–10 ms per request. Those figures illustrate how small savings accumulate; they were not measured here. The measured transport comparison above is the evidence for this setup; measure your own query mix before assigning a per-request saving.

== Reproducible setup ==

{| class="wikitable"
! Item !! Configuration
|-
| Measurement date || 9 September 2026 (Europe/Berlin)
|-
| CPU || AMD Ryzen 9 7900X3D, 12 physical cores / 24 logical CPUs
|-
| Memory / OS || Approximately 62 GiB RAM; Ubuntu 24.04.4 LTS, x86-64
|-
| MemCP || Master 43d0c81401616fafeeb456b22f4a47d6e25361d6; unchanged source for both SQL builds
|-
| SQL compiler/toolchain || Patched Go 1.27.0, revision 84fe25ee45468f5e3920c11c24e2bbb926fc3a72; GOEXPERIMENT empty vs. GOEXPERIMENT=jit
|-
| MariaDB || 10.11.14; query cache OFF; InnoDB buffer pool 128 MiB
|-
| PHP microbenchmark || Same MemCP master, built with Go 1.26.0; PHP 8.5.10 ZTS, Zend Engine 4.5.10, OPcache enabled; FrankenPHP v1.12.7; no Go JIT
|-
| SQL client || Python 3.12 / PyMySQL, established TCP connections to 127.0.0.1; complete result fetching; no connection setup timed
|-
| Database fixture || Read-only consistent snapshot of wordpress_bench_v2, copied to a separate MemCP database; MariaDB InnoDB vs. MemCP sloppy engine; rebuild before measurements
|-
| Measurement order || MariaDB, MemCP, MemCP-JIT; one client, no concurrent query load
|-
| Runtime settings || Query-plan cache, autoindexing and normal optimizer enabled; TracePrint and ScanDebugging disabled while timing
|}

This is a warm read comparison, not a matched-memory or durability benchmark. The host is shared with development services; cooperating builds/tests were paused for timing, but CPUs and memory were not reserved or pinned. Autoindexing must have time to activate: the warmup count deliberately exceeds its initial observation threshold. Compiled plans, indexes and maintained group/prejoin structures may all contribute to warm results. Their cold build cost and write-maintenance cost are outside the latency table.

Cold startup and initial automatic structure building are not represented by this table. Both MemCP modes reuse the same freshly imported, rebuilt directory sequentially; persisted structures and OS caches remain available. An earlier preparation pass populated some structures before the final measured run. This is deliberately a warm comparison, not a cold-start A/B test.

=== Fixture ===

The seven captured queries exercise option lookups, plugin metadata, taxonomy joins, comment counts, search and archives. The data is synthetic, with real WordPress query shapes. It is not a million-post or concurrent-production benchmark. The source WordPress installation identifies itself as 7.1; WordPress/PHP rendering does not execute in the SQL matrix, and installed plugin versions are not performance inputs once these exact queries are captured.

{| class="wikitable"
! Table !! Rows
|-
| wp_posts || 22,063
|-
| wp_postmeta || 14,013
|-
| wp_comments || 6,001
|-
| wp_options || 503
|-
| wp_terms || 175
|-
| wp_term_taxonomy || 175
|-
| wp_term_relationships || 7,701
|}

The corpus contains 2,000 varied generated articles across 16 topics and 20,000 revisions. Titles, excerpts and bodies vary; body lengths range from 396 to 6,996 bytes (median 1,764). The search query uses realistic, present in 80 generated articles; the full query also considers other post types. Low-cardinality fields such as status and categories naturally repeat. This version-2 corpus must not be mixed with older version-1 measurements.

== Correctness and scope ==

Six of the seven exact queries return matching results in all three modes, including the total counts: '''83''' for the search and '''0''' for the Yoast query. After import, rebuild and restart, all 22,063 post IDs, titles, excerpts and body texts also match a canonical SHA-256 comparison. The previously reported blob-content mismatch and search total of 82 were '''not reproduced in this fresh master run'''; they are not carried forward as current failures.

The remaining difference is the taxonomy query. Its exact captured SQL '''does contain ORDER BY t.name ASC'''. Both engines return the same 15 projected rows, but in a different name order. MariaDB exposes wp_terms.name as utf8mb4_unicode_520_ci; this fresh MemCP import exposes utf8mb4. Therefore its timing is diagnostic and excluded from accepted speedup claims. This statement applies to this saved query, not to every WordPress taxonomy query. Without an ORDER BY, a different row order alone would not be a correctness failure.

This benchmark does not include the unmerged type/collation fix in PR #871. It reports current master as measured. The taxonomy collation discrepancy remains a migration issue for this schema and should be resolved before relying on equivalent ordering.

These are query-level correctness checks and a post-text integrity check, not an exhaustive application compatibility test. INSERT/UPDATE workloads, parallel PHP requests, maintenance costs, cold starts and HTTP page-load latency are outside this matrix.

== Try MemCP with your PHP hosting ==

'''MemCP is faster on the substantial repeated SQL work measured here. Try it with your own PHP hosting.''' Start with a copy of your site's data and replay its frequent queries: counts, searches, archives and plugin joins are useful starting points. Compare complete results and total counts, then measure full uncached HTTP requests and representative writes. You can retain the MySQL wire interface or explore direct PHP hosting to remove transport overhead from many small calls.

WordPress is the example, not the product boundary. The same PHP-folder handler and PDO bridge can host other PHP applications; application and plugin compatibility still needs to be checked for the software you deploy.

== Measurement artifacts and exact queries ==

Local artifact directory: <code>../wordpress-db-lab/measurements/wiki-master-43d0c8140/</code>. It contains results-master.json (raw batch means and complete query results), php-latency.json (all PHP samples), manifest.json, counts.json, server logs, and a copied benchmark.py. The SQL runner can be repeated against the same fixture; its WP_BINARY and WP_WORKTREE overrides are for separately labelled candidate checks. Candidate measurements are not included in this master matrix.

Source queries: <code>../wordpress-db-lab/profiles/queries/</code>. They are reproduced below so that the benchmark shapes, including actual ORDER BY clauses, are unambiguous.

=== Option lookup: siteurl ===
<syntaxhighlight lang="sql">
-- Copyright (C) 2026 MemCP WordPress lab contributors
-- SPDX-License-Identifier: AGPL-3.0-or-later
SELECT option_value FROM wp_options WHERE option_name = 'siteurl' LIMIT 1;
</syntaxhighlight>

=== Elementor metadata join + LIMIT ===
<syntaxhighlight lang="sql">
SELECT wp_posts.ID FROM wp_posts INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id ) WHERE 1=1 AND ( ( wp_postmeta.meta_key = '_elementor_template_type' AND wp_postmeta.meta_value = 'landing-page' ) ) AND wp_posts.post_type = 'e-landing-page' AND ((wp_posts.post_status <> 'trash' AND wp_posts.post_status <> 'auto-draft')) GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC LIMIT 0, 1;
</syntaxhighlight>

=== Taxonomy DISTINCT/join, ordered by name ===
<syntaxhighlight lang="sql">
SELECT DISTINCT t.term_id, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON t.term_id = tt.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('category', 'post_tag', 'post_format') AND tr.object_id IN (1, 21993, 21982, 21971, 21960) ORDER BY t.name ASC;
</syntaxhighlight>

=== Comment count with join ===
<syntaxhighlight lang="sql">
SELECT COUNT(*) FROM wp_comments LEFT JOIN wp_posts AS wp_posts_to_exclude_reviews ON comment_post_ID = wp_posts_to_exclude_reviews.ID WHERE ( comment_approved = '1' ) AND comment_type NOT IN ('note') AND comment_type != 'order_note' AND comment_type != 'webhook_delivery' AND comment_type != 'action_log' AND wp_posts_to_exclude_reviews.post_type NOT IN ('product');
</syntaxhighlight>

=== WordPress search + FOUND_ROWS ===
<syntaxhighlight lang="sql">
SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts WHERE 1=1 AND ( wp_posts.ID NOT IN ( SELECT object_id FROM wp_term_relationships WHERE term_taxonomy_id IN (166) ) ) AND (((wp_posts.post_title LIKE '%realistic%') OR (wp_posts.post_excerpt LIKE '%realistic%') OR (wp_posts.post_content LIKE '%realistic%'))) AND ((wp_posts.post_type = 'attachment' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'e-floating-buttons' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'page' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'post' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private')) OR (wp_posts.post_type = 'product' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled' OR wp_posts.post_status = 'private'))) GROUP BY wp_posts.ID ORDER BY wp_posts.post_title LIKE '%realistic%' DESC, wp_posts.post_date DESC LIMIT 0, 10;
</syntaxhighlight>

=== Yoast CAST metadata join + FOUND_ROWS ===
<syntaxhighlight lang="sql">
SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id ) WHERE 1=1 AND ( ( wp_postmeta.meta_key = '_yoast_wpseo_linkdex' AND CAST(wp_postmeta.meta_value AS SIGNED) BETWEEN '1' AND '40' ) ) AND wp_posts.post_type = 'post' AND ((wp_posts.post_status = 'publish')) GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC LIMIT 0, 10;
</syntaxhighlight>

=== Monthly archive DISTINCT ===
<syntaxhighlight lang="sql">
SELECT DISTINCT YEAR( post_date ) AS year, MONTH( post_date ) AS month FROM wp_posts WHERE post_type = 'post' AND post_status != 'auto-draft' AND post_status != 'trash' ORDER BY post_date DESC;
</syntaxhighlight>

Benchmark MemCP vs. MariaDB on Wordpress

2026-09-08T18:50:50Z

Carli: Created page with "  = Benchmark MemCP vs. MariaDB on Wordpress = WordPress does more than fetch individual posts. Category pages, comment counters, search, archive lists, and plugin metadata queries repeatedly join and aggregate data. These are the places where a different database engine can make a noticeable difference. We compared '''MariaDB, MemCP, and MemCP with JIT''' using seven SQL..."

= Benchmark MemCP vs. MariaDB on Wordpress =

WordPress does more than fetch individual posts. Category pages, comment counters, search, archive lists, and plugin metadata queries repeatedly join and aggregate data. These are the places where a different database engine can make a noticeable difference.

We compared '''MariaDB, MemCP, and MemCP with JIT''' using seven SQL queries captured from a WordPress installation. On the tested master revision, JIT-enabled MemCP completed the comment-count query '''25.5 times faster''' and the archive query '''4.8 times faster''' than MariaDB. A simple option lookup was effectively tied. Two small metadata queries still favored MariaDB.

For administrators, the takeaway is specific: '''MemCP offers substantial potential for recurring aggregates, but the right migration decision depends on your site's query mix, correctness, and cache behavior—not one headline multiplier.'''

{| class="wikitable"
| '''Status of this measurement — 8 September 2026'''
The tested master revision has two known correctness issues exposed by this fixture: a blob-reference decoding problem and lost collation in the grouped taxonomy plan. Search also returns an incorrect total count. The affected timings below are diagnostic only, not accepted speedup claims. This is a development benchmark supporting a staging evaluation, '''not a recommendation to replace a production WordPress database with this revision'''.
|}

== The results at a glance ==

The following values are '''warm end-to-end database latency''': sending SQL over an established TCP/MySQL connection and receiving the complete result. For the search and SEO queries, the time also includes a subsequent <code>SELECT FOUND_ROWS()</code> on the same connection.

'''Lower is better.''' The comparison column is MariaDB versus MemCP JIT—not JIT versus vanilla.

{| class="wikitable sortable"
! WordPress workload
! MariaDB
! MemCP
! MemCP + JIT
! MemCP JIT compared with MariaDB
! Result check
|-
| Option lookup (<code>siteurl</code>)
| 0.082 ms
| 0.116 ms
| 0.081 ms
| Effectively tied; approximately 1% lower latency
| Same single row
|-
| Plugin metadata join with LIMIT
| 0.141 ms
| 0.211 ms
| 0.200 ms
| 41.9% higher latency
| Same empty result
|-
| Taxonomy join and DISTINCT
| 0.222 ms
| 0.222 ms
| 0.179 ms
| Not scored: ordering differs
| Same 15 rows, different order
|-
| Comment COUNT with exclusion join
| 4.053 ms
| 0.173 ms
| '''0.159 ms'''
| '''25.5× faster; 96.1% lower latency'''
| Same aggregate result
|-
| WordPress search and total count
| 31.457 ms
| 2.192 ms
| 1.766 ms
| Not scored: total count differs
| Same first 10 rows; total 82 instead of 83
|-
| SEO metadata CAST/range join
| 0.177 ms
| 0.225 ms
| 0.198 ms
| 11.8% higher latency
| Same empty result and total of zero
|-
| DISTINCT year/month archive
| 0.666 ms
| 0.158 ms
| '''0.138 ms'''
| '''4.8× faster; 79.2% lower latency'''
| Same 10 ordered rows
|}

The empty metadata results are intentional disclosures, not successful-hit performance tests. They measure how quickly each engine establishes that there are no matching records in this snapshot. They do not establish performance for populated plugin result sets.

There is no combined “WordPress speedup” score. The frequency of these queries differs between sites, and a correctness failure must not be averaged into a favorable result.

== What this means for a WordPress administrator ==

=== Look for expensive repeated database work ===

A site dominated by indexed point lookups has less to gain from the results here: the option lookup took about 0.08 ms in both MariaDB and JIT-enabled MemCP. A site spending database time on counts, joins, and archive aggregates has a more promising starting point.

For example, the measured comment-count query saves approximately '''3.89 ms per execution'''. That can matter if the query is frequent or occurs repeatedly while rendering an uncached request. It does not imply that the entire PHP request becomes 25 times faster.

Use your own slow-query trace or query-monitoring output to identify the statements that dominate database time. Include frontend pages, administration screens, searches, and scheduled plugin work.

=== Understand what is warm ===

MemCP can reuse compiled query plans and incrementally maintained relational structures such as group caches and prejoins. These structures help avoid repeating a join or aggregate from scratch.

A warm read benchmark benefits from that reuse. It does '''not''' measure the complete cost of maintaining those structures under writes. New comments, imports, metadata updates, and bulk edits therefore belong in a separate staging test.

JIT is another part of execution: eligible code is compiled to native instructions. It is not permission to change SQL results, and it does not make every query faster. The vanilla column shows that much of the aggregate advantage is already present without JIT.

=== Budget for the first request ===

Fast warm responses do not imply that the first request is equally fast. In this run, selected first-query and warm values were:

{| class="wikitable"
! Query
! MemCP first query
! MemCP + JIT first query
! MemCP + JIT warm median
|-
| Plugin metadata join
| 42.3 ms
| 33.2 ms
| 0.200 ms
|-
| Comment COUNT
| 52.1 ms
| 66.2 ms
| 0.159 ms
|-
| WordPress search — correctness issue
| 1,957.7 ms
| 2,553.0 ms
| 1.766 ms
|-
| Archive
| 13,652.3 ms
| 14,560.0 ms
| 0.138 ms
|}

Preparation can also occur after the first execution. During the 20 warmup requests, the plugin metadata query had an approximately 650 ms outlier in each MemCP mode; the search had approximately 580 ms outliers.

These are '''observed first-query latencies after restarting each MemCP process''', not a controlled empty-cache comparison: persisted structures and operating-system caches were retained, and vanilla ran before JIT against the same data directory. Server startup itself is not included in these values.

For a migration rehearsal, measure first requests after restart, cache eviction, and representative writes separately. Do not use the warm table to set cold-start service-level expectations.

== A more useful WordPress fixture ==

The queries come from a WordPress installation using common plugins, including WooCommerce, Elementor, Yoast SEO, Advanced Custom Fields, and Contact Form 7. No plugin receives a special place in the scoring; the seven statements cover distinct kinds of database work.

The measured '''version-2 queryset fixture''' contains these seven tables—not the complete live WordPress installation:

{| class="wikitable"
! Table
! Rows
|-
| <code>wp_posts</code>
| 22,063, including 20,000 revisions
|-
| <code>wp_postmeta</code>
| 14,013
|-
| <code>wp_comments</code>
| 6,001
|-
| <code>wp_options</code>
| 503
|-
| <code>wp_terms</code>
| 175
|-
| <code>wp_term_taxonomy</code>
| 175
|-
| <code>wp_term_relationships</code>
| 7,701
|}

Version 1 repeated essentially the same search-bearing paragraph in every generated article. Version 2 replaces that with 2,000 different titles, excerpts, and article bodies across 16 topics. Article bodies range from 396 to 6,996 bytes, with a median of 1,764 bytes. Revisions retain their parent document's content with an editorial change; low-cardinality fields such as status and categories still repeat naturally.

The generator deliberately provides several search frequencies over the 2,000 generated articles:

* <code>nightjar</code>: 10 matches, or 0.5%;
* <code>realistic</code>: 80 matches, or 4%;
* <code>performance</code>: 240 matches, or 12%;
* <code>quasarless</code>: no matches.

These are corpus checks, '''not four separately timed search workloads'''. The captured search query uses <code>realistic</code>. Its SQL also considers pages and other post types and applies additional predicates, so its complete MariaDB total is 83 rather than 80.

This is a controlled synthetic dataset with real captured query shapes. It is not evidence of performance at millions of posts or under concurrent production traffic. The gains on counts and archives justify testing larger datasets; they do not replace that test.

== Correctness before speed ==

Results were compared across engines, including <code>FOUND_ROWS()</code> where applicable. Five query shapes agreed; two did not.

=== Taxonomy ordering ===

Both databases expose <code>wp_terms.name</code> with <code>utf8mb4_unicode_520_ci</code>. A direct ordered scan agrees, but the full DISTINCT/join plan loses that collation at its grouped sort. For example, MemCP places <code>Uncategorized</code> before lower-case tag names.

This is '''not an unspecified tie order'''. It is a correctness issue, and the taxonomy timing is excluded from speedup claims.

=== Blob content and search totals ===

Full table contents matched checksums immediately after import. After MemCP compaction, 33 long post-content values were observed as 32-byte blob hashes instead of their text. A hash beginning with <code>!</code> collides with the legacy escaped-string encoding.

For one affected post, MariaDB reports a text length of 5,327 bytes and MemCP reports 32. The search's first ten rows agree, but MemCP reports 82 total matches versus MariaDB's 83. Consequently the search timing is also excluded from accepted speedup claims.

These findings must be resolved and the affected workloads revalidated before using this revision for migration. No query was removed from the table to conceal them. All 21 measurements completed without a process crash.

== How the comparison was run ==

{| class="wikitable"
! Item
! Configuration
|-
| MemCP source
| [https://github.com/launix-de/memcp/commit/ce26bfb3deb883efc09272965cc1a3b9eb177e24 Master ce26bfb3d], including the merged Q1/Q3 planner improvements
|-
| Date
| 8 September 2026
|-
| MariaDB
| 10.11.14; query cache disabled; InnoDB buffer pool 128 MiB
|-
| MemCP builds
| Identical source, both built with the patched Go 1.27.0 toolchain; <code>GOEXPERIMENT</code> empty versus <code>GOEXPERIMENT=jit</code>
|-
| Host
| AMD Ryzen 9 7900X3D, 12 cores / 24 logical CPUs; approximately 62 GiB RAM
|-
| Client
| Python/PyMySQL over local TCP; established connections, autocommit
|-
| Sampling
| 20 warmup executions, then 11 batches of 100 executions per query and engine
|-
| Reported statistic
| Median of the 11 batch means, in milliseconds per query or query/FOUND_ROWS pair
|-
| Execution order
| MariaDB, MemCP vanilla, MemCP JIT
|-
| Run duration
| Approximately 93 seconds, excluding builds, fixture import, and investigation
|}

The host was shared with development work and was not isolated for benchmarking. Small percentage differences should not be overinterpreted. The results are not a memory-budget-matched or concurrency-throughput comparison.

The query-plan cache and normal execution behavior remained enabled. We did not substitute hand-written plans or force a special prejoin before timing. The setup contains no application-level page-cache measurement: this is database latency, not HTTP page-render time.

The measurements are specific to this master snapshot and fixture. Earlier version-1 numbers should not be mixed with this table or used to calculate an improvement attributable solely to a planner patch.

== A practical staging checklist ==

Before deciding whether MemCP will speed up your WordPress installation:

# '''Capture your workload.''' Include expensive frontend queries, admin pages, search, and the plugins your site actually uses.
# '''Use one consistent data snapshot.''' Preserve IDs, relationships, text, and collations. Compare contents after import and after compression/restart—not just table row counts.
# '''Check complete results.''' Validate ordering, pagination, NULL behavior, aggregates, and total counts, including <code>FOUND_ROWS()</code>.
# '''Separate first use from steady state.''' Record startup/cache-build spikes, warm medians, and tail latency.
# '''Exercise writes and concurrency.''' Test comments, edits, imports, plugin jobs, cache maintenance, and recovery. This read-only query benchmark does not establish their cost or compatibility.
# '''Measure whole requests too.''' PHP, plugin execution, network requests, and page caching can dominate total page time even when SQL becomes faster.
# '''Keep production unchanged until validation passes.''' Retain your MariaDB deployment and a tested backup/rollback path during evaluation. The known correctness issues above must be closed first.

MemCP's strongest result here is the reduction of recurring aggregate work from milliseconds to a fraction of a millisecond. For WordPress administrators, that is a concrete reason to evaluate it—but equivalent results and predictable operations remain the entry requirements for any speed claim.

Main Page

2026-09-07T20:48:47Z

Carli: /* Optimizations */

= MemCP Database – Fast, Compressed SQL for OLTP and OLAP =

<div style="padding:2.5rem 2rem; margin:0 0 1.5rem; border-radius:14px; background:linear-gradient(135deg,#3d6208 0%,#64910c 52%,#76b512 100%); color:#fff; text-align:center;">
<div style="font-size:2.6rem; line-height:1.1; font-weight:700; margin-bottom:.7rem;">MemCP</div>
<div style="font-size:1.45rem; line-height:1.35; font-weight:600; margin:0 auto .9rem; max-width:56rem;">A fast, compressed, MySQL-protocol-compatible columnar database for modern OLTP and OLAP workloads</div>
<div style="font-size:1.05rem; line-height:1.6; margin:0 auto 1.4rem; max-width:54rem;">MemCP is an open-source SQL database written in Go. It combines persistent in-memory operation, adaptive column compression, parallel query execution and direct application APIs so operational data can be queried and analyzed without maintaining a separate analytical copy.</div>
<div style="margin-bottom:1.5rem;">Beta Open Source MySQL Protocol + HTTP APIs</div>
<div class="plainlinks">[[Install MemCP with Docker|Get started with Docker]] [[Supported SQL|Explore SQL support]] [https://github.com/launix-de/memcp View on GitHub]</div>
</div>

[[File:Memcp-Load.png|center|frameless|1000px|alt=MemCP database dashboard and workload overview]]

<div style="padding:.85rem 1rem; margin:1rem 0 2rem; border-left:5px solid #d99b00; background:#fff7d6; color:#332600;">
'''Development status: Beta.''' MemCP is under active development. Check [[Supported SQL]], [[Current Status and Open Issues]] and the durability requirements of your workload before migrating production data.
</div>

<div style="padding:1rem 1.15rem; margin:1rem 0 2rem; border-left:5px solid #76b512; background:#f5f8f0; color:#17202a;">
'''Your MySQL queries are too slow?''' Run the same representative workload on MemCP instead of assuming another index or a larger MySQL server is the only answer. Compare results, durability and authenticated end-to-end latency; if the migration gates pass, move the performance-critical workload to MemCP. [[MySQL is too slow|Start the MemCP performance evaluation →]]
</div>

<div style="padding:1rem 1.15rem; margin:1rem 0 2rem; border:1px solid #b7c99a; border-radius:9px; background:#f5f8f0; color:#17202a;">
'''Observed performance profile:''' MemCP has achieved '''speedups of 10× and more over MariaDB/PostgreSQL''' in measured OLAP and search-oriented workflows, where RecSets and compressed column scans avoid wide row materialization. In one filtered-list workflow over roughly one million documents, the same query took around '''30 seconds on PostgreSQL''' and '''1.6 seconds on MemCP'''. Isolated OLTP paths currently take about '''1.3–2.0× as long''', but in complete WordPress- and wiki-style page builds this has made '''no significant difference to overall page-loading time''' in the measured application workflows. These are workload observations, not universal guarantees; [[Performance Measurement|reproduce them on your data]].
</div>

== What is MemCP? ==

'''MemCP Database''' is a persistent, compressed, column-oriented SQL database for mixed OLTP and OLAP workloads. It speaks the MySQL client protocol, also exposes SQL over HTTP, and executes supported queries through a functional compiler and a parallel column-storage engine.

The name on this site refers to the database project. MemCP Database is '''not''' a Model Context Protocol (MCP) memory server and is unrelated to the C/C++ <code>memcpy()</code> memory-copy function. When writing about or linking to the project, the unambiguous name is '''MemCP Database'''.

== Problems MemCP solves ==

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(240px,1fr)); gap:1rem; margin:1rem 0 2rem;">
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.1rem; font-weight:700; margin-bottom:.4rem;">Slow filtered lists on millions of rows</div>
Complex <code>WHERE</code>, joins, membership tests, <code>ORDER BY</code> and <code>LIMIT</code> can make a row store inspect or materialize far more data than the page returns. MemCP uses RecSets, late materialization, adaptive indexes and ordered braking to keep the working domain compact. [[MySQL is too slow|Evaluate the query]] · [[RecSets|How RecSets work]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.1rem; font-weight:700; margin-bottom:.4rem;">Slow GROUP BY, COUNT and dashboards</div>
Column scans read only referenced compressed values. Parallel shard-local aggregation, group caches and computed structures target repeated analytical queries over fresh operational data. [[Columnar Storage]] · [[Temporary Computed Columns]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.1rem; font-weight:700; margin-bottom:.4rem;">Slow database writes caused by fsync</div>
Durability is selectable per table. <code>safe</code> protects commits through power loss; <code>logged</code> has measured about 10× its write throughput when process-crash recovery is sufficient. [[Persistency and Performance Guarantees|Choose a write strategy]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.1rem; font-weight:700; margin-bottom:.4rem;">Database writes wear out an SD card</div>
For reconstructible edge data, <code>sloppy</code> avoids a continuous WAL and normally publishes one compressed generation every 15 minutes, reducing constant flash writes while making the loss window explicit. [[Hardware Requirements]] · [[Persistency and Performance Guarantees]]
</div>
</div>

== Choose your path ==

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(220px,1fr)); gap:1rem; margin:1rem 0 2rem;">
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Evaluate MemCP</div>
Understand the workload model, current status, hardware needs and differences from MySQL.

[[What is OLTP and OLAP|OLTP and OLAP]] · [[Comparison: MemCP vs. MySQL|Compare with MySQL]] · [[Hardware Requirements|Hardware requirements]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Build an application</div>
Use the MySQL protocol, SQL over HTTP, RDF or application-specific endpoints inside MemCP.

[[Advanced SQL Tutorial|SQL tutorial]] · [[SQL over REST|SQL over HTTP]] · [[Database Tools compatibility with MemCP|Client tooling]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Operate MemCP</div>
Plan deployment, migration, persistence, storage backends, settings and performance measurement.

[[Deployment]] · [[Persistency and Performance Guarantees|Durability]] · [[Migration from MySQL and PostgreSQL|Migration]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Understand and contribute</div>
Explore the storage engine, optimizer, embedded Scheme runtime and project internals.

[[Query Planner and Physical Lowering|Query planner]] · [[RecSets]] · [[Columnar Storage]] · [[Contributing]]
</div>
</div>

== Quickstart ==

=== Run with Docker ===

<pre>
docker run --name memcp \
-e ROOT_PASSWORD='choose-a-password' \
-p 4321:4321 -p 3307:3307 \
-v memcp-data:/data \
carli2/memcp:latest
</pre>

Then open <code>http://localhost:4321</code> and connect a MySQL application to <code>127.0.0.1</code> on port '''3307'''. The volume mounted at <code>/data</code> retains persistent table data across container replacement. See [[Install MemCP with Docker|the complete Docker guide]] before deploying it as a service.

=== Build from source ===

<pre>
git clone https://github.com/launix-de/memcp
cd memcp
go mod download
make
./memcp --api-port=4321 --mysql-port=3307 lib/main.scm
</pre>

Connect with MySQL tooling:

<pre>
mysql -h 127.0.0.1 -u root -p -P 3307
# Enter the password selected for this data directory.
</pre>

The development default for a fresh data directory is <code>admin</code>. Change it before exposing MemCP to another machine. MemCP can also be supervised with PM2:

<pre>
pm2 start ./memcp --name memcp -- \
--no-repl -data ./data --api-port=4321 --mysql-port=3307 lib/main.scm
</pre>

Background deployments must use <code>--no-repl</code> so that closing standard input does not terminate the interactive console and stop the server. See [[Compile MemCP from Source|Build from Source]], [[Deployment]] and [[MemCP Console]].

== Key features ==

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(260px,1fr)); gap:.8rem; margin:1rem 0;">
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''High performance''' Parallel, batch-oriented query execution is designed for multicore CPUs, compact working sets and fast persistent storage, serving both OLTP and OLAP workloads.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Columnar storage''' Data is stored by column for improved compression, a smaller memory footprint and fast analytical access.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Persistent in-memory operation''' MemCP is designed to keep active data in memory while offering configurable per-table durability and persistence backends.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Fast configurable writes''' Choose durability per table: <code>safe</code> protects committed data through power loss, while measured <code>logged</code> paths have reached about 10× its write throughput; <code>sloppy</code> batches reconstructible data into the normal 15-minute rebuild cycle.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Built-in APIs''' SQL over HTTP and in-database services can remove an extra middleware hop for suitable applications.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Adaptive compression''' Bit-packing, dictionary encoding and sequence compression reduce the bytes read for suitable data. Historical imports have reached reductions around 80% versus their MySQL/MariaDB representation; measure your own schema.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Simple deployment''' Start with Docker, PM2 or the native binary. The compact application has historically had an installation footprint of approximately 10 MB.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Extensible frontends''' The Go storage engine and embedded Scheme environment support SQL, RDF, REST and custom application interfaces.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #9ad32d; background:#f5f8f0; color:#17202a;">'''Cost-based query execution''' Logical decorrelation and join optimization are separated from cost-based physical selection of scans, indexes, RecSets, reusable caches and execution pipelines. [[Query Planner and Physical Lowering|How MemCP plans and lowers queries →]] · [[RecSets|How compact record sets process large domains →]]</div>
</div>

== Why MemCP? ==

Traditional relational databases were designed around spinning disks and comparatively small numbers of CPU cores. MemCP rethinks storage and query execution for modern multicore systems, large caches, fast storage and mixed workloads.

Typical use cases include:

* real-time dashboards and analytics;
* data-heavy SaaS platforms;
* embedded systems with limited resources;
* high-throughput OLTP/OLAP hybrids.

Whether MemCP is a good fit depends on the required SQL compatibility, durability, query mix, data size and operational environment. Review [[Supported SQL]], [[Persistency and Performance Guarantees]] and [[Performance Measurement]] rather than treating benchmark figures as universal guarantees.

== MemCP vs. MySQL ==

{| class="wikitable" style="width:100%;"
! Feature
! MySQL
! MemCP
|-
| Storage model
| Primarily row-based
| Column-based and compressed
|-
| Performance focus
| General-purpose relational workloads
| Parallel, in-memory-oriented execution for mixed operational and analytical workloads
|-
| In-memory capability
| Available through selected engines and caching
| Central design goal and default operating model
|-
| REST API integration
| Normally external
| Built in
|-
| Installation footprint
| Common server installations are approximately 150 MB or larger
| The native application has historically been approximately 10 MB
|-
| Open source
| ✅
| ✅
|}

MemCP provides MySQL protocol compatibility but does not claim to implement every MySQL feature. See [[Comparison: MemCP vs. MySQL]] for a detailed comparison and [[Database Tools compatibility with MemCP]] for client compatibility.

== Architecture overview ==

* '''Tables, schemas and columns''': Familiar SQL structures use a compressed columnar physical layout. See [[Databases, Tables and Columns]] and [[Columnar Storage]].
* '''Transaction model''': Delta and main storage support mixed OLTP and OLAP semantics. See [[Transactions and Isolation]] and [[Shards, RecordIDs, Main Storage, Delta Storage]].
* '''Query planning''': Logical optimization is separated from physical execution decisions. See [[Query Planner and Physical Lowering]].
* '''Persistence''': Per-table durability can use filesystem, S3 or Ceph/RADOS storage. See [[Persistency and Performance Guarantees]] and [[Storage Backends]].
* '''Operations''': The dashboard, metrics, process inspection and memory controls support day-to-day operation. See [[Dashboard and Operations]] and [[Memory Management and Eviction]].
* '''Scale-out roadmap''': MemCP is single-node today. A leaderless CRUSH/RADOS and MOESI-inspired cluster design is planned, but must not be treated as an available production feature. See [[Cluster Monitor]].
* '''Frontends''': MemCP provides multiple query and application interfaces:
** SQL through the MySQL wire protocol and SQL over HTTP;
** an RDF/graph query frontend;
** custom APIs through in-database web applications.

== Documentation ==

The following navigation remains on the start page so users and search engines can reach every major documentation area directly.

=== Introduction and evaluation ===

* [[What is OLTP and OLAP]]
* [[History of the MemCP project]]
* [[Hardware Requirements]]
* [[Persistency and Performance Guarantees]]
* [[Current Status and Open Issues]]
* [[MySQL is too slow|SQL performance problems MemCP solves]]
* [[Comparison: MemCP vs. MySQL]]

=== Getting started ===

* [[Install MemCP with Docker|Install with Docker]]
* [[With Singularity|Install with Singularity/Apptainer]]
* [[Compile MemCP from Source|Build from Source]]
* [[Contributing]]
* [[Introduction to Scheme]]
* [[Full SCM API documentation]]

=== Administration ===

* [[Deployment]]
* [[Migration from MySQL and PostgreSQL]]
* [[Settings]]
* [[Security and Authentication]]
* [[Dashboard and Operations]]
* [[Memory Management and Eviction]]
* [[Process Hibernation]]
* [[Performance Measurement]]
* [[MemCP Console]]

=== Frontends ===

==== SQL frontend ====

* [[Supported SQL]]
* [[Advanced SQL Tutorial]]
* [[JSON|JSON and SQL/JSON]]
* [[Triggers]]
* [[SQL over REST]]
* [[Database Tools compatibility with MemCP|Supported Tooling]]
* [[Query Planner and Physical Lowering|How SQL operators are implemented]]
* [[Add custom SQL operators to MemCP]]

==== RDF frontend ====

* [[Introduction to RDF]]
* [[Advanced Graph Querying]]
* [https://github.com/launix-de/rdfop RDF browser and templating example]

==== Custom frontends ====

* [[In-Database WebApps and REST Services]]
* [[MemCP for Microservices]]
* [[Websockets in MemCP]]

=== Persistence backends and storage ===

* [[File System]]
* [[Storage Backends|S3-compatible and Ceph/RADOS storage]]
* [[Cluster Monitor]]

=== Internals ===

==== How MemCP works ====

* [[Databases, Tables and Columns]]
* [[Shards, RecordIDs, Main Storage, Delta Storage]]
* [[Columnar Storage]]
* [[Transactions and Isolation]]
* [[Query Planner and Physical Lowering]]
* [[Full SCM API documentation]]

==== Scheme documentation ====

* [[SCM Builtins]]
* [[Arithmetic / Logic]]
* [[Strings]]
* [[Streams]]
* [[Lists]]
* [[Associative Lists / Dictionaries]]
* [[Date]]
* [[Vectors]]
* [[Parsers]]
* [[Sync]]
* [[IO]]
* [[Storage]]

==== Optimizations ====

* [[Benchmark MemCP vs. MariaDB on Wordpress|Benchmarks]]
* [[Query Planner and Physical Lowering]]
* [[In-Memory Compression, Columnar Compression Techniques]]
* [[Temporary Computed Columns]]
* [[Data Auto Sharding and Auto Indexing]]
* [[Parallel Computing]]

== Further reading ==

* [https://github.com/launix-de/memcp MemCP on GitHub]
* [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]
* [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]
* [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]

Additional blog posts on design decisions, compression techniques and performance optimization are available on the [https://launix.de/launix/ Launix blog].

== Community ==

MemCP is an open-source project maintained by developers for developers. Contributions are welcome in the form of bug reports, feature requests and pull requests.

See [[Contributing]] and the [https://github.com/launix-de/memcp GitHub repository].

Main Page

2026-09-07T20:48:15Z

Carli: /* Optimizations */

= MemCP Database – Fast, Compressed SQL for OLTP and OLAP =

<div style="padding:2.5rem 2rem; margin:0 0 1.5rem; border-radius:14px; background:linear-gradient(135deg,#3d6208 0%,#64910c 52%,#76b512 100%); color:#fff; text-align:center;">
<div style="font-size:2.6rem; line-height:1.1; font-weight:700; margin-bottom:.7rem;">MemCP</div>
<div style="font-size:1.45rem; line-height:1.35; font-weight:600; margin:0 auto .9rem; max-width:56rem;">A fast, compressed, MySQL-protocol-compatible columnar database for modern OLTP and OLAP workloads</div>
<div style="font-size:1.05rem; line-height:1.6; margin:0 auto 1.4rem; max-width:54rem;">MemCP is an open-source SQL database written in Go. It combines persistent in-memory operation, adaptive column compression, parallel query execution and direct application APIs so operational data can be queried and analyzed without maintaining a separate analytical copy.</div>
<div style="margin-bottom:1.5rem;">Beta Open Source MySQL Protocol + HTTP APIs</div>
<div class="plainlinks">[[Install MemCP with Docker|Get started with Docker]] [[Supported SQL|Explore SQL support]] [https://github.com/launix-de/memcp View on GitHub]</div>
</div>

[[File:Memcp-Load.png|center|frameless|1000px|alt=MemCP database dashboard and workload overview]]

<div style="padding:.85rem 1rem; margin:1rem 0 2rem; border-left:5px solid #d99b00; background:#fff7d6; color:#332600;">
'''Development status: Beta.''' MemCP is under active development. Check [[Supported SQL]], [[Current Status and Open Issues]] and the durability requirements of your workload before migrating production data.
</div>

<div style="padding:1rem 1.15rem; margin:1rem 0 2rem; border-left:5px solid #76b512; background:#f5f8f0; color:#17202a;">
'''Your MySQL queries are too slow?''' Run the same representative workload on MemCP instead of assuming another index or a larger MySQL server is the only answer. Compare results, durability and authenticated end-to-end latency; if the migration gates pass, move the performance-critical workload to MemCP. [[MySQL is too slow|Start the MemCP performance evaluation →]]
</div>

<div style="padding:1rem 1.15rem; margin:1rem 0 2rem; border:1px solid #b7c99a; border-radius:9px; background:#f5f8f0; color:#17202a;">
'''Observed performance profile:''' MemCP has achieved '''speedups of 10× and more over MariaDB/PostgreSQL''' in measured OLAP and search-oriented workflows, where RecSets and compressed column scans avoid wide row materialization. In one filtered-list workflow over roughly one million documents, the same query took around '''30 seconds on PostgreSQL''' and '''1.6 seconds on MemCP'''. Isolated OLTP paths currently take about '''1.3–2.0× as long''', but in complete WordPress- and wiki-style page builds this has made '''no significant difference to overall page-loading time''' in the measured application workflows. These are workload observations, not universal guarantees; [[Performance Measurement|reproduce them on your data]].
</div>

== What is MemCP? ==

'''MemCP Database''' is a persistent, compressed, column-oriented SQL database for mixed OLTP and OLAP workloads. It speaks the MySQL client protocol, also exposes SQL over HTTP, and executes supported queries through a functional compiler and a parallel column-storage engine.

The name on this site refers to the database project. MemCP Database is '''not''' a Model Context Protocol (MCP) memory server and is unrelated to the C/C++ <code>memcpy()</code> memory-copy function. When writing about or linking to the project, the unambiguous name is '''MemCP Database'''.

== Problems MemCP solves ==

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(240px,1fr)); gap:1rem; margin:1rem 0 2rem;">
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.1rem; font-weight:700; margin-bottom:.4rem;">Slow filtered lists on millions of rows</div>
Complex <code>WHERE</code>, joins, membership tests, <code>ORDER BY</code> and <code>LIMIT</code> can make a row store inspect or materialize far more data than the page returns. MemCP uses RecSets, late materialization, adaptive indexes and ordered braking to keep the working domain compact. [[MySQL is too slow|Evaluate the query]] · [[RecSets|How RecSets work]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.1rem; font-weight:700; margin-bottom:.4rem;">Slow GROUP BY, COUNT and dashboards</div>
Column scans read only referenced compressed values. Parallel shard-local aggregation, group caches and computed structures target repeated analytical queries over fresh operational data. [[Columnar Storage]] · [[Temporary Computed Columns]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.1rem; font-weight:700; margin-bottom:.4rem;">Slow database writes caused by fsync</div>
Durability is selectable per table. <code>safe</code> protects commits through power loss; <code>logged</code> has measured about 10× its write throughput when process-crash recovery is sufficient. [[Persistency and Performance Guarantees|Choose a write strategy]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.1rem; font-weight:700; margin-bottom:.4rem;">Database writes wear out an SD card</div>
For reconstructible edge data, <code>sloppy</code> avoids a continuous WAL and normally publishes one compressed generation every 15 minutes, reducing constant flash writes while making the loss window explicit. [[Hardware Requirements]] · [[Persistency and Performance Guarantees]]
</div>
</div>

== Choose your path ==

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(220px,1fr)); gap:1rem; margin:1rem 0 2rem;">
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Evaluate MemCP</div>
Understand the workload model, current status, hardware needs and differences from MySQL.

[[What is OLTP and OLAP|OLTP and OLAP]] · [[Comparison: MemCP vs. MySQL|Compare with MySQL]] · [[Hardware Requirements|Hardware requirements]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Build an application</div>
Use the MySQL protocol, SQL over HTTP, RDF or application-specific endpoints inside MemCP.

[[Advanced SQL Tutorial|SQL tutorial]] · [[SQL over REST|SQL over HTTP]] · [[Database Tools compatibility with MemCP|Client tooling]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Operate MemCP</div>
Plan deployment, migration, persistence, storage backends, settings and performance measurement.

[[Deployment]] · [[Persistency and Performance Guarantees|Durability]] · [[Migration from MySQL and PostgreSQL|Migration]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Understand and contribute</div>
Explore the storage engine, optimizer, embedded Scheme runtime and project internals.

[[Query Planner and Physical Lowering|Query planner]] · [[RecSets]] · [[Columnar Storage]] · [[Contributing]]
</div>
</div>

== Quickstart ==

=== Run with Docker ===

<pre>
docker run --name memcp \
-e ROOT_PASSWORD='choose-a-password' \
-p 4321:4321 -p 3307:3307 \
-v memcp-data:/data \
carli2/memcp:latest
</pre>

Then open <code>http://localhost:4321</code> and connect a MySQL application to <code>127.0.0.1</code> on port '''3307'''. The volume mounted at <code>/data</code> retains persistent table data across container replacement. See [[Install MemCP with Docker|the complete Docker guide]] before deploying it as a service.

=== Build from source ===

<pre>
git clone https://github.com/launix-de/memcp
cd memcp
go mod download
make
./memcp --api-port=4321 --mysql-port=3307 lib/main.scm
</pre>

Connect with MySQL tooling:

<pre>
mysql -h 127.0.0.1 -u root -p -P 3307
# Enter the password selected for this data directory.
</pre>

The development default for a fresh data directory is <code>admin</code>. Change it before exposing MemCP to another machine. MemCP can also be supervised with PM2:

<pre>
pm2 start ./memcp --name memcp -- \
--no-repl -data ./data --api-port=4321 --mysql-port=3307 lib/main.scm
</pre>

Background deployments must use <code>--no-repl</code> so that closing standard input does not terminate the interactive console and stop the server. See [[Compile MemCP from Source|Build from Source]], [[Deployment]] and [[MemCP Console]].

== Key features ==

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(260px,1fr)); gap:.8rem; margin:1rem 0;">
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''High performance''' Parallel, batch-oriented query execution is designed for multicore CPUs, compact working sets and fast persistent storage, serving both OLTP and OLAP workloads.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Columnar storage''' Data is stored by column for improved compression, a smaller memory footprint and fast analytical access.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Persistent in-memory operation''' MemCP is designed to keep active data in memory while offering configurable per-table durability and persistence backends.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Fast configurable writes''' Choose durability per table: <code>safe</code> protects committed data through power loss, while measured <code>logged</code> paths have reached about 10× its write throughput; <code>sloppy</code> batches reconstructible data into the normal 15-minute rebuild cycle.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Built-in APIs''' SQL over HTTP and in-database services can remove an extra middleware hop for suitable applications.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Adaptive compression''' Bit-packing, dictionary encoding and sequence compression reduce the bytes read for suitable data. Historical imports have reached reductions around 80% versus their MySQL/MariaDB representation; measure your own schema.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Simple deployment''' Start with Docker, PM2 or the native binary. The compact application has historically had an installation footprint of approximately 10 MB.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Extensible frontends''' The Go storage engine and embedded Scheme environment support SQL, RDF, REST and custom application interfaces.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #9ad32d; background:#f5f8f0; color:#17202a;">'''Cost-based query execution''' Logical decorrelation and join optimization are separated from cost-based physical selection of scans, indexes, RecSets, reusable caches and execution pipelines. [[Query Planner and Physical Lowering|How MemCP plans and lowers queries →]] · [[RecSets|How compact record sets process large domains →]]</div>
</div>

== Why MemCP? ==

Traditional relational databases were designed around spinning disks and comparatively small numbers of CPU cores. MemCP rethinks storage and query execution for modern multicore systems, large caches, fast storage and mixed workloads.

Typical use cases include:

* real-time dashboards and analytics;
* data-heavy SaaS platforms;
* embedded systems with limited resources;
* high-throughput OLTP/OLAP hybrids.

Whether MemCP is a good fit depends on the required SQL compatibility, durability, query mix, data size and operational environment. Review [[Supported SQL]], [[Persistency and Performance Guarantees]] and [[Performance Measurement]] rather than treating benchmark figures as universal guarantees.

== MemCP vs. MySQL ==

{| class="wikitable" style="width:100%;"
! Feature
! MySQL
! MemCP
|-
| Storage model
| Primarily row-based
| Column-based and compressed
|-
| Performance focus
| General-purpose relational workloads
| Parallel, in-memory-oriented execution for mixed operational and analytical workloads
|-
| In-memory capability
| Available through selected engines and caching
| Central design goal and default operating model
|-
| REST API integration
| Normally external
| Built in
|-
| Installation footprint
| Common server installations are approximately 150 MB or larger
| The native application has historically been approximately 10 MB
|-
| Open source
| ✅
| ✅
|}

MemCP provides MySQL protocol compatibility but does not claim to implement every MySQL feature. See [[Comparison: MemCP vs. MySQL]] for a detailed comparison and [[Database Tools compatibility with MemCP]] for client compatibility.

== Architecture overview ==

* '''Tables, schemas and columns''': Familiar SQL structures use a compressed columnar physical layout. See [[Databases, Tables and Columns]] and [[Columnar Storage]].
* '''Transaction model''': Delta and main storage support mixed OLTP and OLAP semantics. See [[Transactions and Isolation]] and [[Shards, RecordIDs, Main Storage, Delta Storage]].
* '''Query planning''': Logical optimization is separated from physical execution decisions. See [[Query Planner and Physical Lowering]].
* '''Persistence''': Per-table durability can use filesystem, S3 or Ceph/RADOS storage. See [[Persistency and Performance Guarantees]] and [[Storage Backends]].
* '''Operations''': The dashboard, metrics, process inspection and memory controls support day-to-day operation. See [[Dashboard and Operations]] and [[Memory Management and Eviction]].
* '''Scale-out roadmap''': MemCP is single-node today. A leaderless CRUSH/RADOS and MOESI-inspired cluster design is planned, but must not be treated as an available production feature. See [[Cluster Monitor]].
* '''Frontends''': MemCP provides multiple query and application interfaces:
** SQL through the MySQL wire protocol and SQL over HTTP;
** an RDF/graph query frontend;
** custom APIs through in-database web applications.

== Documentation ==

The following navigation remains on the start page so users and search engines can reach every major documentation area directly.

=== Introduction and evaluation ===

* [[What is OLTP and OLAP]]
* [[History of the MemCP project]]
* [[Hardware Requirements]]
* [[Persistency and Performance Guarantees]]
* [[Current Status and Open Issues]]
* [[MySQL is too slow|SQL performance problems MemCP solves]]
* [[Comparison: MemCP vs. MySQL]]

=== Getting started ===

* [[Install MemCP with Docker|Install with Docker]]
* [[With Singularity|Install with Singularity/Apptainer]]
* [[Compile MemCP from Source|Build from Source]]
* [[Contributing]]
* [[Introduction to Scheme]]
* [[Full SCM API documentation]]

=== Administration ===

* [[Deployment]]
* [[Migration from MySQL and PostgreSQL]]
* [[Settings]]
* [[Security and Authentication]]
* [[Dashboard and Operations]]
* [[Memory Management and Eviction]]
* [[Process Hibernation]]
* [[Performance Measurement]]
* [[MemCP Console]]

=== Frontends ===

==== SQL frontend ====

* [[Supported SQL]]
* [[Advanced SQL Tutorial]]
* [[JSON|JSON and SQL/JSON]]
* [[Triggers]]
* [[SQL over REST]]
* [[Database Tools compatibility with MemCP|Supported Tooling]]
* [[Query Planner and Physical Lowering|How SQL operators are implemented]]
* [[Add custom SQL operators to MemCP]]

==== RDF frontend ====

* [[Introduction to RDF]]
* [[Advanced Graph Querying]]
* [https://github.com/launix-de/rdfop RDF browser and templating example]

==== Custom frontends ====

* [[In-Database WebApps and REST Services]]
* [[MemCP for Microservices]]
* [[Websockets in MemCP]]

=== Persistence backends and storage ===

* [[File System]]
* [[Storage Backends|S3-compatible and Ceph/RADOS storage]]
* [[Cluster Monitor]]

=== Internals ===

==== How MemCP works ====

* [[Databases, Tables and Columns]]
* [[Shards, RecordIDs, Main Storage, Delta Storage]]
* [[Columnar Storage]]
* [[Transactions and Isolation]]
* [[Query Planner and Physical Lowering]]
* [[Full SCM API documentation]]

==== Scheme documentation ====

* [[SCM Builtins]]
* [[Arithmetic / Logic]]
* [[Strings]]
* [[Streams]]
* [[Lists]]
* [[Associative Lists / Dictionaries]]
* [[Date]]
* [[Vectors]]
* [[Parsers]]
* [[Sync]]
* [[IO]]
* [[Storage]]

==== Optimizations ====

* [[Benchmarks]]
* [[Query Planner and Physical Lowering]]
* [[In-Memory Compression, Columnar Compression Techniques]]
* [[Temporary Computed Columns]]
* [[Data Auto Sharding and Auto Indexing]]
* [[Parallel Computing]]

== Further reading ==

* [https://github.com/launix-de/memcp MemCP on GitHub]
* [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]
* [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]
* [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]

Additional blog posts on design decisions, compression techniques and performance optimization are available on the [https://launix.de/launix/ Launix blog].

== Community ==

MemCP is an open-source project maintained by developers for developers. Contributions are welcome in the form of bug reports, feature requests and pull requests.

See [[Contributing]] and the [https://github.com/launix-de/memcp GitHub repository].

RecSets

2026-09-01T23:53:05Z

Carli:

= RecSets: compact, parallel record-set processing =

<blockquote>'''Verified against commit <code>c42e19eba</code> on 27 August 2026.''' RecSets are an internal physical execution mechanism. Applications normally use SQL and let the optimizer choose them.</blockquote>

A '''RecSet''' is a query-local subset of one MemCP table. Instead of materializing complete rows, it records the physical record IDs that currently qualify. The result remains table-shaped: <code>scan</code>, <code>scan_batch</code>, <code>scan_order</code>, <code>scan_order_multi</code>, and <code>scan_exists</code> can consume it as a source while retaining columnar reads, indexes, transaction visibility, and parallel shard execution.

This separation is valuable on large relations. A cheap predicate can identify a narrow working set once; expensive residual predicates, correlated access checks, joins, projections, ordering, and aggregation then operate only on those candidates. Millions of base rows need not become millions of Scheme lists or wide temporary rows.

== What a RecSet represents ==

A RecSet contains membership, not result rows:

* it belongs to exactly one base table and one transaction/visibility context;
* it has set semantics: no duplicates and no join multiplicity;
* it carries no result order;
* it is query-local and is neither persisted nor a stable public record-ID format;
* it can be an exact set of TRUE rows or a safe candidate superset followed by a residual predicate.

The essential implementation is deliberately small:

<pre>
type recSet struct {
table *table
shards []recSetShard
count int64
}

type recSetShard struct {
shard *storageShard
kind recSetRepresentation
universe uint32
data []uint32
used uint32
count int64
}
</pre>

Each shard owns its own record-ID universe. This mirrors MemCP's storage layout and makes construction and set algebra naturally parallel without sharing a global bitmap or exposing record IDs across shards.

== Adaptive representation ==

One fixed representation would waste memory for either sparse or dense sets. Every RecSet shard therefore chooses among three forms:

{| class="wikitable"
! Representation !! Layout !! Best fit !! Membership and combination
|-
| Ranges || Sorted <code>(base, count)</code> pairs || Long runs, full/empty shards, or a nearly full set with a few holes || Range lookup; linear range sweeps
|-
| Positive IDs || Sorted, deduplicated <code>uint32</code> record IDs || A small, scattered result || Sorted membership lookup; linear list merge/intersection
|-
| Bitmap || One bit per record-ID position || Dense or highly fragmented membership || Direct bit test and word-wise Boolean operations
|}

There is intentionally no separate “full”, “empty”, or negative representation. An empty shard is zero ranges; a full shard is one range; “everything except a few holes” is a small number of ranges. A fragmented set eventually becomes a bitmap.

For orientation, an uncompressed bitmap for a one-million-position shard needs approximately 125 kB. A positive-ID result uses four bytes per hit before slice overhead, and a range uses eight bytes per run. The builder chooses according to the observed shape rather than a SQL selectivity guess. Immutable search-index RecSets have an additional compressed form: their positive IDs and ranges can themselves use MemCP's packed integer storage, and their small interface can intersect a mutable query-local RecSet in place.

=== One-pass adaptive construction ===

The builder starts with ranges. More than three isolated singleton runs indicate sparse, non-clustered data and trigger conversion to a positive list. If the compact representation exceeds the bitmap budget, it escalates to a bitmap.

The filter predicate is evaluated '''exactly once per candidate record ID'''. A representation change copies already collected IDs or ranges; it never repeats the scan or replays Scheme predicates. This matters because a predicate may include decoded column values, JIT/interpreter work, search-index checks, or an expensive correlated condition. After construction, range and positive results are copied into right-sized slices so a small result does not keep the worst-case bitmap allocation alive.

Ranges are also split at the immutable-main/delta boundary. Consumers can then bulk-read a main-storage run without accidentally spanning into transaction deltas.

== Building a RecSet in the scan/filter path ==

<code>scan_recset</code> uses the normal MemCP scan machinery rather than walking every row blindly:

# extract equality, range, LIKE/search, computed, and RecSet boundaries from the filter;
# order useful boundaries and derive index limits;
# schedule eligible shards in parallel;
# prepare only the columns named by the filter and optimize its Scheme procedure;
# read index candidates in batches of up to 1,024 record IDs;
# apply transaction/deletion visibility and the residual predicate once;
# feed qualifying IDs to the adaptive shard builder.

This keeps late materialization intact. Creating <code>WHERE tenant_id = 7 AND state = 'open'</code> as a RecSet needs the filter columns, but it does not have to decode payload, JSON, or text columns that are only required by a later projection.

<pre>
(begin
(define tx ((context "session") "__memcp_tx"))
(define orders (table "shop" "orders"))
(define open_orders
(scan_recset tx orders
'("tenant_id" "state")
(lambda (tenant state)
(and (equal? tenant 7) (equal? state "open")))))
(recset_count open_orders))
</pre>

The low-level example illustrates the operator contract. SQL users should normally write the predicate in SQL and inspect the optimizer's decision with <code>EXPLAIN PHYSICAL</code>.

=== Narrowing an existing RecSet ===

Passing a RecSet as the input of <code>scan_recset</code> adds another condition without revisiting rows outside the existing membership:

<pre>
(define recent_open
(scan_recset tx open_orders
'("created_at")
(lambda (created_at) (>= created_at cutoff))))
</pre>

For a very sparse input, MemCP walks its IDs directly. For a broader input, it combines the exact RecSet boundary with other index/search boundaries before reading residual columns. This is the intended pattern for “cheap selective predicate first, expensive predicate second”, including ACL or search conditions with their own subscans.

When scanning a RecSet, consecutive visible main-storage IDs are coalesced into runs. Each required column is fetched for the whole run with <code>GetValueRange</code>; visibility remains row-specific, while column decoding becomes batched and cache-friendly. Delta rows use their transaction-aware delta path.

== Membership inside another scan ==

The pseudo-column <code>$recset_contains</code> supplies a closure bound to the current row. It tests one or more same-table RecSets without exposing physical record IDs:

<pre>
(scan tx orders
'("$recset_contains" "total")
(lambda (contains total)
(and (contains open_orders) (> total 1000)))
'("id")
(lambda (id) id)
(lambda (sum id) (+ sum id))
0)
</pre>

The closure caches the current RecSet and its shard entry, avoiding a repeated shard search for successive membership calls. If an exact RecSet boundary already proves the membership condition, the scan can provide an always-true closure instead of probing it again. Multiple <code>$recset_contains</code> parameters are supported when a filter needs independent sets.

== Set algebra, in parallel ==

<pre>
(define visible (recset_intersect (list tenant_rows active_rows)))
(define either (recset_union (list starred_rows recent_rows)))
(define allowed (recset_difference (list visible blocked_rows muted_rows)))
(define other_visible_rows (recset_not visible))
</pre>

Union, intersection, and difference require operands from the same table. Sorted range/list invariants allow linear two-pointer or N-way sweeps; bitmap pairs use word operations. Mixed representation pairs have specialized implementations instead of expanding everything into one common row list.

Each shard combination is independent and runs through MemCP's bounded fan-out scheduler. Results are sorted back into deterministic shard order after parallel completion. This lets Boolean work over millions of record positions use available cores while keeping synchronization out of the inner membership loop.

<code>recset_not</code> deserves special attention: it constructs the currently visible table universe and subtracts the input. It does not merely invert raw bits, because that could resurrect deleted rows or rows invisible to the transaction. At SQL level, a RecSet complement is also not automatically equivalent to <code>NOT predicate</code> when NULL/UNKNOWN is possible.

== Projecting membership through a join ==

<code>recset_project_join</code> reads distinct source keys from one RecSet and creates a RecSet over matching rows of another table. This is useful for EXISTS/IN carriers, ACL links, tags, and search-result relations:

<pre>
(define permitted_links
(scan_recset tx access_links
'("user_id")
(lambda (user_id) (equal? user_id current_user))))

(define permitted_documents
(recset_project_join tx permitted_links
'("document_id") documents '("id")))

(scan_order tx permitted_documents
'() (lambda () true)
'("created_at" "id") '(< <)
0 0 50
'("id" "title")
(lambda (id title) (list id title))
(lambda (rows row) (append rows (list row)))
'() false)
</pre>

Composite source/target keys are supported. At execution time the operator knows the actual number of distinct source keys and target rows per shard. It chooses between indexed point probes and a dense target scan. A single integral key can use an O(1) hash membership test during the dense scan; generic composite keys use sorted tuples and binary lookup.

The current cost constants were calibrated with production-shaped A/B fixtures. One recorded case projected about 80,000 integral keys through 1.84 million target rows in roughly 8 ms on the calibration machine, while another case with 105 keys and 22 indexed target shards favored point probes. These are decision-model observations, '''not portable latency guarantees'''; hardware, encodings, shard layout, visibility state, and surrounding query work matter.

For repeated scalar membership checks, <code>recset_key_index</code> builds an immutable lookup closure over one or more columns:

<pre>
(define contains_document
(recset_key_index tx permitted_documents '("id")))

(contains_document 12345)
</pre>

== Ordering, LIMIT, and adaptive batches ==

A RecSet itself is unordered. <code>scan_order</code> applies an index/order from the base table and intersects it with RecSet membership. Depending on cardinality, index span, and LIMIT, the execution kernel can:

* drive the base ordered index and test RecSet membership;
* translate sparse RecSet IDs into ordered index positions and sort those positions;
* iterate the RecSet directly when no useful ordering is required.

The crossover is calibrated by a benchmark with 800,000 rows, RecSet cardinalities from 16 to 720,000, bounded and unbounded scans, and both distributed and adversarial late hits. <code>EXPLAIN PHYSICAL</code> can report the adaptive RecSet boundary and runtime alternatives such as <code>ordered_inverse_recset</code> and <code>ordered_base_membership</code>.

For an ORDER BY/LIMIT query whose expensive acceptance condition lives on another relation, <code>scan_order_batch_accept</code> preserves order without evaluating the whole candidate domain. It draws an initial ordered RecSet of <code>OFFSET + LIMIT</code> candidates, asks a batch function to return an exact accepted subset, and doubles subsequent disjoint batches until enough rows are accepted or the source is exhausted. The accepted RecSet is used as a mask against the already ordered vector, so it is not rescanned in physical-ID order.

This is especially useful for ordered document/search/ACL queries: project a candidate batch to the access relation, filter it in parallel, project accepted membership back, and stop once the requested page is complete.

== How the optimizer uses RecSets ==

RecSets appear only during physical lowering. The logical planner first establishes joins, domains, SQL truth semantics, grouping, ordering, and multiplicity. The physical cost model can then compare:

{| class="wikitable"
! Choice !! Often attractive when
|-
| Direct indexed probe || The driving side is small and each lookup is selective.
|-
| RecSet carrier || A filtered domain is reused, projected, combined, or much narrower than the base relation.
|-
| Prepared key/group table || Keys or aggregate results need relational reuse and more than membership.
|-
| Dense fused scan || Most rows are visited once and building a reusable carrier would add work.
|-
| Ordered RecSet iterator || An existing membership domain must be consumed in base-table order or with a small LIMIT.
|}

Candidate RecSets do not permit the optimizer to drop correctness checks. Approximate LIKE/full-text boundaries and other safe supersets retain their residual SQL predicate. Nullable <code>IN</code>/<code>NOT IN</code> and general three-valued logic are handled before truth-only Boolean set identities are applied.

Use these views when investigating a plan:

<pre>
EXPLAIN IR SELECT ...;
EXPLAIN REORDER SELECT ...;
EXPLAIN PHYSICAL SELECT ...;
</pre>

<code>EXPLAIN PHYSICAL</code> exposes membership-carrier choices and runtime access alternatives for projected and ordered RecSets. A remaining roadmap item is richer tracing for why the same RecSet domain was built or reused and for the exact ordered-iterator mode chosen at runtime.

== Operational and correctness limits ==

* Do not retain a RecSet as application state or persist its physical IDs.
* Algebra operands must refer to the same table; projected joins deliberately return a new target-table RecSet.
* A RecSet is tied to its transaction/visibility context. Batch-accept callbacks must return a same-table, same-transaction subset.
* Scanning a RecSet for reads is supported; mutation callbacks such as <code>$update</code> are rejected on a RecSet source.
* Cancellation is checked before shard work is scheduled. Once entered, one shard's RecSet build runs atomically.
* Set cardinality is not SQL row cardinality after a one-to-many join, and membership does not retain duplicate keys.
* Ordered output needs <code>scan_order</code>; include a unique tie-breaker when stable pagination matters.

== Low-level operator map ==

{| class="wikitable"
! Operator !! Purpose
|-
| <code>scan_recset</code> || Build from a table or narrow an existing RecSet.
|-
| <code>recset_count</code> || Return stored membership cardinality.
|-
| <code>recset_union</code> / <code>recset_intersect</code> || Combine same-table memberships.
|-
| <code>recset_difference</code> / <code>recset_not</code> || Subtract sets or complement against visible rows.
|-
| <code>recset_project_join</code> || Project distinct keys into a target-table RecSet.
|-
| <code>recset_key_index</code> || Build an immutable scalar/composite-key membership closure.
|-
| <code>$recset_contains</code> || Test current-row membership inside a scan predicate.
|-
| <code>scan_order_batch_accept</code> || Filter progressively larger ordered candidate RecSets before LIMIT.
|}

The generated [[Storage]] reference is authoritative for exact signatures. For the surrounding execution architecture see [[Scan]], [[Query Planner and Physical Lowering]], [[Columnar Storage]], [[Data Auto Sharding and Auto Indexing]], and [[Parallel Computing]].

Supported SQL

2026-08-28T10:03:52Z

Carli: /* Planner inspection */

= Supported SQL =

<blockquote>Verified against development commit <code>c42e19eba</code> on 27 August 2026. MemCP is Beta and does not claim complete MySQL or PostgreSQL compatibility. Unsupported shapes should fail explicitly; test a required workload before migration.
</blockquote>
This page is a capability map for application evaluation. MemCP accepts a broad MySQL-oriented dialect through its MySQL protocol and HTTP endpoint, plus PostgreSQL-oriented syntax through a separate HTTP endpoint. Both compile into the same logical planner and storage runtime, but accepted surface syntax and returned metadata can differ by frontend.

Support is defined by tested semantics, not merely by whether the parser accepts a statement. Compatibility-only clauses may intentionally be no-ops, and uncommon combinations can remain unsupported even when their individual operators are available. Use this overview to select candidates, then verify every required query, constraint, transaction, prepared statement, and client behavior against the deployed release.

== Query language ==

* <code>SELECT</code>, aliases, <code>WHERE</code>, <code>DISTINCT</code>, <code>GROUP BY</code>, <code>HAVING</code>, <code>ORDER BY</code>, <code>LIMIT</code>, and <code>OFFSET</code>;
* <code>INNER</code>, <code>LEFT</code>, <code>RIGHT</code>, and <code>CROSS JOIN</code>;
* <code>UNION</code> and <code>UNION ALL</code>, including ordered/limited set results;
* derived tables and logical SQL views;
* scalar, <code>EXISTS</code>, <code>NOT EXISTS</code>, <code>IN</code>, and <code>NOT IN</code> subqueries, including many correlated and grouped forms;
* aggregates <code>COUNT</code>, <code>SUM</code>, <code>AVG</code>, <code>MIN</code>, <code>MAX</code>, and <code>GROUP_CONCAT</code>;
* window functions including <code>ROW_NUMBER</code>, <code>RANK</code>, <code>DENSE_RANK</code>, <code>LEAD</code>, <code>LAG</code>, <code>FIRST_VALUE</code>, and <code>LAST_VALUE</code> with partition/order clauses.


== Data modification and schema ==

* single- and multi-row <code>INSERT</code>, <code>INSERT IGNORE</code>, <code>INSERT … SELECT</code>, and MySQL or PostgreSQL upsert syntax;
* <code>UPDATE</code> and <code>DELETE</code>, including qualified and supported multi-table forms;
* <code>CREATE</code>, <code>ALTER</code>, <code>RENAME</code>, <code>TRUNCATE</code>, and <code>DROP</code> for databases and tables;
* <code>CREATE VIEW</code>, indexes, primary/unique keys, foreign keys, defaults, <code>AUTO_INCREMENT</code>, computed expressions, types, collations, and per-table ENGINE;
* persistent BEFORE/AFTER triggers for INSERT, UPDATE, and DELETE.

Foreign keys are enforced for the tested RESTRICT, CASCADE, and SET NULL paths. Some compatibility DDL is intentionally accepted as a no-op; for example, a client accepting <code>ADD FULLTEXT</code> does not imply a MySQL-compatible full-text index implementation.


== Transactions, security, and metadata ==

* <code>BEGIN</code>, <code>COMMIT</code>, and <code>ROLLBACK</code>;
* users, passwords, grants, revokes, and database access policies;
* <code>SHOW DATABASES</code>, tables, columns, indexes, create-table metadata, variables, engines, processlist, and selected <code>INFORMATION_SCHEMA</code> relations;
* prepared statements over the MySQL protocol;
* <code>KILL QUERY</code> and <code>KILL CONNECTION</code>.

See [[Transactions_and_Isolation|Transactions and Isolation]] and [[Security_and_Authentication|Security and Authentication]] for the operational contract.


== Expressions ==

MemCP implements SQL NULL/three-valued logic, CASE, COALESCE/NULLIF/IFNULL, arithmetic and bitwise operators, comparisons, BETWEEN, IN lists, LIKE/ILIKE, REGEXP/RLIKE, MATCH…AGAINST compatibility, casts, string functions, mathematical functions, hashes, date/time functions, intervals, and timezone conversion.

The exact parser and function catalog changes more often than this overview. Use the generated API reference and the SQL test taxonomy for detailed coverage.


== JSON ==

MemCP stores declared <code>JSON</code>/<code>JSONB</code> values in a native tagged BSON representation. The MySQL dialect implements JSON validation, type inspection, path extraction, constructors, aggregates, mutation functions, containment/search operations, <code>JSON_TABLE</code>, and the <code>-></code>/<code>->></code> operators. The PostgreSQL dialect provides its corresponding <code>json</code>/<code>jsonb</code> operators, constructors, aggregates, path functions, and set-returning expansion forms. Deterministic path expressions can be materialized and indexed through the ordinary computed-column/index machinery. See [[JSON]] for the detailed compatibility contract and deviations.


== Planner inspection ==

<code>EXPLAIN SELECT ...;
EXPLAIN IR SELECT ...;
EXPLAIN REORDER SELECT ...;
EXPLAIN PHYSICAL SELECT ...;
EXPLAIN COMPILE SELECT ...;</code>

See [[Query Planner and Physical Lowering]].


== Known limitations ==

MemCP is not a byte-for-byte MySQL server implementation. Metadata details, administrative commands, uncommon syntax, advanced locking modes, XA, stored procedures, and untested SQL-standard edge cases may be absent. Consult the [https://github.com/launix-de/memcp/issues current issue tracker] and add a must-pass and must-fail test for every application-critical construct.

Main Page

2026-08-27T06:47:12Z

Carli:

= MemCP – A Modern In-Memory Columnar Database =

<div style="padding:2.5rem 2rem; margin:0 0 1.5rem; border-radius:14px; background:linear-gradient(135deg,#3d6208 0%,#64910c 52%,#76b512 100%); color:#fff; text-align:center;">
<div style="font-size:2.6rem; line-height:1.1; font-weight:700; margin-bottom:.7rem;">MemCP</div>
<div style="font-size:1.45rem; line-height:1.35; font-weight:600; margin:0 auto .9rem; max-width:56rem;">A fast, compressed, MySQL-compatible columnar database for modern OLTP and OLAP workloads</div>
<div style="font-size:1.05rem; line-height:1.6; margin:0 auto 1.4rem; max-width:54rem;">MemCP combines persistent in-memory operation, adaptive column compression, parallel query execution and direct application APIs in a lightweight database written in Go.</div>
<div style="margin-bottom:1.5rem;">Beta Open Source MySQL Protocol + HTTP APIs</div>
<div class="plainlinks">[[Install MemCP with Docker|Get started with Docker]] [[Supported SQL|Explore SQL support]] [https://github.com/launix-de/memcp View on GitHub]</div>
</div>

[[File:Memcp-Load.png|center|frameless|1000px|alt=MemCP database dashboard and workload overview]]

<div style="padding:.85rem 1rem; margin:1rem 0 2rem; border-left:5px solid #d99b00; background:#fff7d6; color:#332600;">
'''Development status: Beta.''' MemCP is under active development. Check [[Supported SQL]], [[Current Status and Open Issues]] and the durability requirements of your workload before migrating production data.
</div>

== Choose your path ==

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(220px,1fr)); gap:1rem; margin:1rem 0 2rem;">
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Evaluate MemCP</div>
Understand the workload model, current status, hardware needs and differences from MySQL.

[[What is OLTP and OLAP|OLTP and OLAP]] · [[Comparison: MemCP vs. MySQL|Compare with MySQL]] · [[Hardware Requirements|Hardware requirements]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Build an application</div>
Use the MySQL protocol, SQL over HTTP, RDF or application-specific endpoints inside MemCP.

[[Advanced SQL Tutorial|SQL tutorial]] · [[SQL over REST|SQL over HTTP]] · [[Database Tools compatibility with MemCP|Client tooling]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Operate MemCP</div>
Plan deployment, migration, persistence, storage backends, settings and performance measurement.

[[Deployment]] · [[Persistency and Performance Guarantees|Durability]] · [[Migration from MySQL and PostgreSQL|Migration]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Understand and contribute</div>
Explore the storage engine, optimizer, embedded Scheme runtime and project internals.

[[Query Planner and Physical Lowering|Query planner]] · [[Columnar Storage]] · [[Contributing]]
</div>
</div>

== Quickstart ==

=== Run with Docker ===

<syntaxhighlight lang="bash">
docker run --name memcp \
-e ROOT_PASSWORD='choose-a-password' \
-p 4321:4321 -p 3307:3307 \
carli2/memcp:latest
</syntaxhighlight>

Then open <code>http://localhost:4321</code> and connect a MySQL application to <code>127.0.0.1</code> on port '''3307'''. Add a volume mounted at <code>/data</code> when the container must retain persistent data. See [[Install MemCP with Docker|the complete Docker guide]] before deploying it as a service.

=== Build from source ===

<syntaxhighlight lang="bash">
git clone https://github.com/launix-de/memcp
cd memcp
go mod download
make
./memcp --api-port=4321 --mysql-port=3307 lib/main.scm
</syntaxhighlight>

Connect with MySQL tooling:

<syntaxhighlight lang="bash">
mysql -h 127.0.0.1 -u root -p -P 3307
# Enter the password selected for this data directory.
</syntaxhighlight>

The development default for a fresh data directory is <code>admin</code>. Change it before exposing MemCP to another machine. MemCP can also be supervised with PM2:

<syntaxhighlight lang="bash">
pm2 start ./memcp --name memcp -- \
--no-repl -data ./data --api-port=4321 --mysql-port=3307 lib/main.scm
</syntaxhighlight>

Background deployments must use <code>--no-repl</code> so that closing standard input does not terminate the interactive console and stop the server. See [[Compile MemCP from Source|Build from Source]], [[Deployment]] and [[MemCP Console]].

== Key features ==

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(260px,1fr)); gap:.8rem; margin:1rem 0;">
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''High performance''' NUMA-aware, parallelized query execution is optimized for multicore CPUs, large caches and NVMe SSDs, serving both OLTP and OLAP workloads.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Columnar storage''' Data is stored by column for improved compression, a smaller memory footprint and fast analytical access.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Persistent in-memory operation''' MemCP is designed to keep active data in memory while offering configurable per-table durability and persistence backends.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Built-in APIs''' SQL over HTTP and in-database services can remove an extra middleware hop for suitable applications.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Adaptive compression''' Bit-packing, dictionary encoding and sequence compression can reduce suitable datasets by up to 80% compared with their MySQL/MariaDB representation.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Simple deployment''' Start with Docker, PM2 or the native binary. The compact application has historically had an installation footprint of approximately 10 MB.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Extensible frontends''' The Go storage engine and embedded Scheme environment support SQL, RDF, REST and custom application interfaces.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #9ad32d; background:#f5f8f0; color:#17202a;">'''Cost-based query execution''' Logical decorrelation and join optimization are separated from cost-based physical selection of scans, indexes, RecSets, reusable caches and execution pipelines. [[Query Planner and Physical Lowering|How MemCP plans and lowers queries →]]</div>
</div>

== Why MemCP? ==

Traditional relational databases were designed around spinning disks and comparatively small numbers of CPU cores. MemCP rethinks storage and query execution for modern multicore systems, large caches, fast storage and mixed workloads.

Typical use cases include:

* real-time dashboards and analytics;
* data-heavy SaaS platforms;
* embedded systems with limited resources;
* high-throughput OLTP/OLAP hybrids.

Whether MemCP is a good fit depends on the required SQL compatibility, durability, query mix, data size and operational environment. Review [[Supported SQL]], [[Persistency and Performance Guarantees]] and [[Performance Measurement]] rather than treating benchmark figures as universal guarantees.

== MemCP vs. MySQL ==

{| class="wikitable" style="width:100%;"
! Feature
! MySQL
! MemCP
|-
| Storage model
| Primarily row-based
| Column-based and compressed
|-
| Performance focus
| General-purpose relational workloads
| NUMA-aware, in-memory execution for mixed operational and analytical workloads
|-
| In-memory capability
| Available through selected engines and caching
| Central design goal and default operating model
|-
| REST API integration
| Normally external
| Built in
|-
| Installation footprint
| Common server installations are approximately 150 MB or larger
| The native application has historically been approximately 10 MB
|-
| Open source
| ✅
| ✅
|}

MemCP provides MySQL protocol compatibility but does not claim to implement every MySQL feature. See [[Comparison: MemCP vs. MySQL]] for a detailed comparison and [[Database Tools compatibility with MemCP]] for client compatibility.

== Architecture overview ==

* '''Tables, schemas and columns''': Familiar SQL structures use a compressed columnar physical layout. See [[Databases, Tables and Columns]] and [[Columnar Storage]].
* '''Transaction model''': Delta and main storage support mixed OLTP and OLAP semantics. See [[Transactions and Isolation]] and [[Shards, RecordIDs, Main Storage, Delta Storage]].
* '''Query planning''': Logical optimization is separated from physical execution decisions. See [[Query Planner and Physical Lowering]].
* '''Persistence''': Per-table durability can use filesystem, S3 or Ceph/RADOS storage. See [[Persistency and Performance Guarantees]].
* '''Frontends''': MemCP provides multiple query and application interfaces:
** SQL through the MySQL wire protocol and SQL over HTTP;
** an RDF/graph query frontend;
** custom APIs through in-database web applications.

== Documentation ==

The following navigation remains on the start page so users and search engines can reach every major documentation area directly.

=== Introduction and evaluation ===

* [[What is OLTP and OLAP]]
* [[History of the MemCP project]]
* [[Hardware Requirements]]
* [[Persistency and Performance Guarantees]]
* [[Current Status and Open Issues]]
* [[Comparison: MemCP vs. MySQL]]

=== Getting started ===

* [[Install MemCP with Docker|Install with Docker]]
* [[With Singularity|Install with Singularity/Apptainer]]
* [[Compile MemCP from Source|Build from Source]]
* [[Contributing]]
* [[Introduction to Scheme]]
* [[Full SCM API documentation]]

=== Administration ===

* [[Deployment]]
* [[Migration from MySQL and PostgreSQL]]
* [[Settings]]
* [[Process Hibernation]]
* [[Performance Measurement]]
* [[MemCP Console]]

=== Frontends ===

==== SQL frontend ====

* [[Supported SQL]]
* [[Advanced SQL Tutorial]]
* [[SQL over REST]]
* [[Database Tools compatibility with MemCP|Supported Tooling]]
* [[How SQL Operators are implemented on MemCP]]
* [[Add custom SQL operators to MemCP]]

==== RDF frontend ====

* [[Introduction to RDF]]
* [[Advanced Graph Querying]]
* [[RDF templating and model driven development]]

==== Custom frontends ====

* [[In-Database WebApps|In-Database WebApps and REST Services]]
* [[MemCP for Microservices]]
* [[Websockets in MemCP]]

=== Persistence backends and storage ===

* [[File System]]
* [[S3 Buckets]]
* [[Ceph/Rados]]
* [[Cluster Monitor]]

=== Internals ===

==== How MemCP works ====

* [[Databases, Tables and Columns]]
* [[Shards, RecordIDs, Main Storage, Delta Storage]]
* [[Columnar Storage]]
* [[Transactions and Isolation]]
* [[Query Planner and Physical Lowering]]
* [[Full SCM API documentation]]

==== Scheme documentation ====

* [[SCM Builtins]]
* [[Arithmetic / Logic]]
* [[Strings]]
* [[Streams]]
* [[Lists]]
* [[Associative Lists / Dictionaries]]
* [[Window Functions]]
* [[Date]]
* [[Timezone]]
* [[Vectors]]
* [[Parsers]]
* [[Sync]]
* [[IO]]
* [[Storage]]
* [[Transactions]]
* [[JIT in Scheme|JIT]]

==== Optimizations ====

* [[Query Planner and Physical Lowering]]
* [[In-Memory Compression, Columnar Compression Techniques]]
* [[Temporary Columns]]
* [[Data Auto Sharding and Auto Indexing]]
* [[Parallel Computing]]

== Further reading ==

* [https://github.com/launix-de/memcp MemCP on GitHub]
* [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]
* [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]
* [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]

Additional blog posts on design decisions, compression techniques and performance optimization are available on the [https://launix.de/launix/ Launix blog].

== Community ==

MemCP is an open-source project maintained by developers for developers. Contributions are welcome in the form of bug reports, feature requests and pull requests.

See [[Contributing]] and the [https://github.com/launix-de/memcp GitHub repository].

Window Functions

2026-08-27T06:38:43Z

Carli: Created page with "← Back to Full SCM API documentation == stream_emit == invokes a streaming callback immediately; marks ordering-sensitive emission as an observable effect '''Allowed number of parameters:''' 2–2 === Parameters === * '''emit''' (<code>func(value:any) -> any</code>): * '''value''' (<code>any</code>): === Returns === <code>any</code> == stream_window_reduce == applies OFFSET/LIMIT and a serial redu..."

← Back to [[Full SCM API documentation]]

== stream_emit ==

invokes a streaming callback immediately; marks ordering-sensitive emission as an observable effect

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''emit''' (<code>func(value:any) -> any</code>):
* '''value''' (<code>any</code>):


=== Returns ===

<code>any</code>

== stream_window_reduce ==

applies OFFSET/LIMIT and a serial reducer to complete values emitted by a nested streaming producer without collecting an intermediate relation

'''Allowed number of parameters:''' 5–5


=== Parameters ===

* '''offset''' (<code>number</code>): number of complete producer values to skip
* '''limit''' (<code>number</code>): maximum values to reduce, or -1 for no limit
* '''reduce''' (<code>func(acc:any, value:any) -> any</code>): serial accumulator over complete values
* '''neutral''' (<code>any</code>): initial accumulator
* '''producer''' (<code>func(emit:func) -> any</code>): nested streaming plan called with a one-value emit callback


=== Returns ===

<code>any</code>

== window_mut ==

Ring buffer shift-insert for window functions. (window_mut window emit_fn vals) writes vals (a list of stride values) into the current slot, increments counter. If skip>0, decrements skip. Otherwise calls (emit_fn oldest_v0 oldest_v1 ... newest_v0 newest_v1) with all slot values ordered oldest-to-newest. Returns updated window.

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''window''' (<code>list</code>): ring buffer accumulator
* '''emit_fn''' (<code>func(values:any...) -> any</code>): callback receiving all window values oldest-to-newest
* '''vals''' (<code>list</code>): list of stride values to insert


=== Returns ===

<code>list</code>

== window_flush ==

Flush remaining window buffer by shifting in nils. (window_flush window emit_fn count) shifts in count positions of nils, calling emit_fn for each displaced position. Returns nil.

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''window''' (<code>list</code>): ring buffer accumulator
* '''emit_fn''' (<code>func(values:any...) -> any</code>): callback receiving all window values oldest-to-newest
* '''count''' (<code>number</code>): number of nil positions to shift in


=== Returns ===

<code>nil</code>

== show_processlist ==

returns a list of active sessions for SHOW [FULL] PROCESSLIST; pass true for full info

'''Allowed number of parameters:''' 0–1


=== Parameters ===

* '''full''' (<code>bool</code>): if true, include full Info text ''(optional)''


=== Returns ===

<code>list</code>

== connection_id ==

returns the process-list ID of the current session (MySQL CONNECTION_ID() equivalent)

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>int</code>

== kill_query ==

cancel the running query in session id; returns true if a query was killed

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''id''' (<code>int</code>): session ID from SHOW PROCESSLIST


=== Returns ===

<code>bool</code>

Full SCM API documentation

2026-08-27T06:38:23Z

Carli:

= Documentation =

* [[SCM Builtins]]
* [[Arithmetic / Logic]]
* [[Strings]]
* [[Streams]]
* [[Lists]]
* [[Associative Lists / Dictionaries]]
* [[Window Functions]]
* [[Date]]
* [[Timezone]]
* [[Vectors]]
* [[Parsers]]
* [[Sync]]
* [[IO]]
* [[Storage]]
* [[Transactions]]
* [[JIT in Scheme|JIT]]

Vectors

2026-08-27T06:37:29Z

Carli:

= Vectors =

The '''Vectors''' module provides vector mathematics functionality for the SCM programming language. This module includes:

* '''Vector operations''': Functions to perform mathematical operations on vectors (dot)
* '''Distance calculations''': Support for different distance metrics including DOT, COSINE, and EUCLIDEAN

These functions provide essential tools for working with vectors in mathematical computations, enabling vector analysis and similarity calculations in SCM programs.

← Back to [[Full SCM API documentation]]

== dot ==

produced the dot product

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''v1''' (<code>list</code>): vector1
* '''v2''' (<code>list</code>): vector2
* '''mode''' (<code>string</code>): DOT, COSINE, EUCLIDEAN, default is DOT ''(optional)''


=== Returns ===

<code>number</code>

Transactions

2026-08-27T06:36:53Z

Carli: Created page with " ← Back to Full SCM API documentation == tx_commit == Commits the current transaction. '''Allowed number of parameters:''' 1–1 === Parameters === * '''session''' (<code>func(key:string?, value:any?) -> any</code>): the session function that holds tx state === Returns === <code>bool</code> == tx_rollback == Rolls back the current transaction. '''Allowed number of parameters:''' 1–1 <span..."

← Back to [[Full SCM API documentation]]

== tx_commit ==

Commits the current transaction.

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''session''' (<code>func(key:string?, value:any?) -> any</code>): the session function that holds tx state


=== Returns ===

<code>bool</code>

== tx_rollback ==

Rolls back the current transaction.

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''session''' (<code>func(key:string?, value:any?) -> any</code>): the session function that holds tx state


=== Returns ===

<code>bool</code>

== with_autocommit ==

Executes fn inside an implicit TxCursorStability transaction if no explicit transaction is active in session. Commits on success, rolls back on error, and re-raises any panic so the caller's error handler still fires. If an explicit transaction is active (session["transaction"] != nil), fn is executed without any wrapping.

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''session''' (<code>func(key:string?, value:any?) -> any</code>): the session function holding tx state
* '''fn''' (<code>func</code>): zero-argument function to execute


=== Returns ===

<code>any</code>

Full SCM API documentation

2026-08-27T06:36:39Z

Carli:

= Documentation =

* [[SCM Builtins]]
* [[Arithmetic / Logic]]
* [[Strings]]
* [[Streams]]
* [[Lists]]
* [[Associative Lists / Dictionaries]]
* [[Date]]
* [[Timezone]]
* [[Vectors]]
* [[Parsers]]
* [[Sync]]
* [[IO]]
* [[Storage]]
* [[Transactions]]
* [[JIT in Scheme|JIT]]

Timezone

2026-08-27T06:36:18Z

Carli:

← Back to [[Full SCM API documentation]]

== unix_timestamp ==

returns a unix timestamp (integer seconds since epoch)

'''Allowed number of parameters:''' 0–1


=== Parameters ===

* '''dt''' (<code>any</code>): optional datetime value to convert ''(optional)''


=== Returns ===

<code>int</code>

== system_time_zone ==

returns the operating system's local timezone name

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>string</code>

== convert_tz ==

converts a datetime from one timezone to another

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''dt''' (<code>any</code>): datetime value
* '''from_tz''' (<code>string</code>): source timezone
* '''to_tz''' (<code>string</code>): target timezone


=== Returns ===

<code>date</code>

== from_unixtime ==

converts a unix timestamp to a datetime in the session timezone

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''unix_ts''' (<code>number</code>): unix timestamp (seconds since epoch)
* '''format''' (<code>string</code>): optional MySQL format string ''(optional)''


=== Returns ===

<code>date</code>

== utc_timestamp ==

returns the current UTC datetime

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>date</code>

== utc_date ==

returns the current UTC date (midnight)

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>date</code>

== utc_time ==

returns the current UTC time (as a datetime at epoch date)

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>date</code>

== sysdate ==

returns the current datetime (re-evaluated per call, unlike now())

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>date</code>

== at_time_zone ==

PostgreSQL AT TIME ZONE operator: converts between timezones

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''dt''' (<code>any</code>): datetime value
* '''zone''' (<code>string</code>): target timezone


=== Returns ===

<code>date</code>

== timestampdiff ==

returns the difference between two datetimes in the given unit

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''unit''' (<code>string</code>): SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, YEAR
* '''dt1''' (<code>any</code>): first datetime
* '''dt2''' (<code>any</code>): second datetime


=== Returns ===

<code>int</code>

Timezone

2026-08-27T06:35:23Z

Carli: Created page with " == unix_timestamp == returns a unix timestamp (integer seconds since epoch) '''Allowed number of parameters:''' 0–1 === Parameters === * '''dt''' (<code>any</code>): optional datetime value to convert ''(optional)'' === Returns === <code>int</code> == system_time_zone == returns the operating system's local timezone name '''Allowed number of parameters:''' 0–0 === Para..."

== unix_timestamp ==

returns a unix timestamp (integer seconds since epoch)

'''Allowed number of parameters:''' 0–1


=== Parameters ===

* '''dt''' (<code>any</code>): optional datetime value to convert ''(optional)''


=== Returns ===

<code>int</code>

== system_time_zone ==

returns the operating system's local timezone name

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>string</code>

== convert_tz ==

converts a datetime from one timezone to another

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''dt''' (<code>any</code>): datetime value
* '''from_tz''' (<code>string</code>): source timezone
* '''to_tz''' (<code>string</code>): target timezone


=== Returns ===

<code>date</code>

== from_unixtime ==

converts a unix timestamp to a datetime in the session timezone

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''unix_ts''' (<code>number</code>): unix timestamp (seconds since epoch)
* '''format''' (<code>string</code>): optional MySQL format string ''(optional)''


=== Returns ===

<code>date</code>

== utc_timestamp ==

returns the current UTC datetime

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>date</code>

== utc_date ==

returns the current UTC date (midnight)

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>date</code>

== utc_time ==

returns the current UTC time (as a datetime at epoch date)

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>date</code>

== sysdate ==

returns the current datetime (re-evaluated per call, unlike now())

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>date</code>

== at_time_zone ==

PostgreSQL AT TIME ZONE operator: converts between timezones

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''dt''' (<code>any</code>): datetime value
* '''zone''' (<code>string</code>): target timezone


=== Returns ===

<code>date</code>

== timestampdiff ==

returns the difference between two datetimes in the given unit

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''unit''' (<code>string</code>): SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, YEAR
* '''dt1''' (<code>any</code>): first datetime
* '''dt2''' (<code>any</code>): second datetime


=== Returns ===

<code>int</code>

Full SCM API documentation

2026-08-27T06:35:15Z

Carli:

Sync

2026-08-27T06:29:25Z

Carli:

= Sync =

The '''Sync''' module provides synchronization and concurrency functionality for the SCM programming language. This module includes:

* '''Session management''': Functions to create threadsafe key-value stores (newsession)
* '''Context handling''': Context management with session support (context)
* '''Timing control''': Functions to pause execution (sleep)
* '''Function caching''': One-time execution wrappers with result caching (once)
* '''Thread synchronization''': Mutex creation for serialized access (mutex)

These functions provide essential tools for managing concurrent operations, shared state, and synchronization in multi-threaded SCM programs.

← Back to [[Full SCM API documentation]]

== newpromise ==

Creates a single-value promise cell (thread-safe via CAS spin-lock). Returns a tagPromise Scmer. (newpromise) allocates a [2]Scmer backing; (newpromise list) reuses an existing ≥2-element slice as backing with zero extra allocation. API: (p "value") reads current value (nil if pending), (p "value" v) resolves, (p "once" v) resolves once (panics if already fulfilled/failed), (p "once" v msg) resolves once with custom panic message, (p "state") returns state (nil/true/false), (p "fail") sets failed and clears the stored value, (p "fail" err) sets failed and stores err as payload.

'''Allowed number of parameters:''' 0–1


=== Parameters ===

* '''list''' (<code>any</code>): optional: ≥2-element slice to use as backing ''(optional)''


=== Returns ===

<code>func(operation:string, value:any?, msg:string?) -> any</code>

== newsession ==

Creates a new session which is a threadsafe key-value store. Besides get/set/list, get_or_compute_scoped shares concurrent computation by a query-local handle.

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>func(key_or_operation:any?, value_scope_or_key:any?, key_or_producer:any?, scoped_producer:func?) -> any</code>

== with_session ==

Executes a function with the given session installed in the execution context, so storage operations can access the session's transaction state.

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''session''' (<code>func(key:any?, value:any?) -> any</code>): the session to install
* '''fn''' (<code>func</code>): the function to execute


=== Returns ===

<code>any</code>

== context ==

Context helper function. Each context also contains a session. (context func args) creates a new context and runs func in that context, (context "session") reads the session variable, (context "check") will check the liveliness of the context and otherwise throw an error

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''args...''' (<code>any</code>): depends on the usage ''(variadic)''


=== Returns ===

<code>any</code>

== sleep ==

sleeps the amount of seconds

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''duration''' (<code>number</code>): number of seconds to sleep


=== Returns ===

<code>bool</code>

== once ==

Creates a function wrapper that you can call multiple times but only gets executed once. The result value is cached and returned on a second call. You can add parameters to that resulting function that will be passed to the first run of the wrapped function.

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''f''' (<code>func</code>): function that produces the result value


=== Returns ===

<code>func(args:any...) -> any</code>

== mutex ==

Creates a context-aware mutex. The return value serializes calls to parameterless functions and stops waiting when the current request is cancelled.

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>func(fn:func) -> any</code>

== numcpu ==

Returns the number of logical CPUs available for parallel execution

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>number</code>

== memstats ==

Returns memory statistics as a dict with keys: alloc, total_alloc, sys, heap_alloc, heap_sys (all in bytes)

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>dict</code>


== setTimeout ==

Schedules a callback to run after the given delay in milliseconds (fractional values allowed for sub-millisecond precision).

'''Allowed number of parameters:''' 2–10000


=== Parameters ===

* '''callback''' (<code>func(args:any...) -> any</code>): function to execute once the timeout expires
* '''milliseconds''' (<code>number</code>): milliseconds until execution
* '''args...''' (<code>any</code>): optional arguments forwarded to the callback ''(variadic)''


=== Returns ===

<code>int</code>


== clearTimeout ==

Cancels a timeout created with setTimeout.

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''id''' (<code>number</code>): identifier returned by setTimeout


=== Returns ===

<code>bool</code>

Strings

2026-08-27T06:24:31Z

Carli:

= Strings =

The '''Strings''' module provides comprehensive string manipulation and processing functions for the SCM programming language. This module includes:

* '''String validation''': Functions to check string types and properties (string?, strlen)
* '''String manipulation''': Concatenation (concat), substring extraction (substr), and case conversion (toLower, toUpper)
* '''Pattern matching''': Wildcard pattern matching (strlike) and string replacement (replace)
* '''String parsing''': Splitting strings (split) and data type simplification (simplify)
* '''Encoding/Decoding''': URL encoding/decoding, HTML entities, JSON processing, and binary conversion
* '''Internationalization''': Collation support for natural sorting and language-specific comparisons

These functions provide the essential tools for text processing, data formatting, and string-based operations in SCM programs.

← Back to [[Full SCM API documentation]]


== string? ==

tells if the value is a string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value


=== Returns ===

<code>bool</code>

== concat ==

concatenates stringable values and returns a string

'''Allowed number of parameters:''' 1–10000


=== Parameters ===

* '''value''' (<code>any</code>): first value to concat
* '''more...''' (<code>any</code>): additional values to concat ''(variadic)''


=== Returns ===

<code>string</code>

== sql_concat ==

SQL CONCAT semantics: returns NULL if any argument is NULL

'''Allowed number of parameters:''' 1–10000


=== Parameters ===

* '''value''' (<code>any</code>): first value to concat
* '''more...''' (<code>any</code>): additional values to concat ''(variadic)''


=== Returns ===

<code>any</code>

== substr ==

returns a substring (0-based index)

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''value''' (<code>string</code>): string to cut
* '''start''' (<code>number</code>): first character index (0-based)
* '''len''' (<code>number</code>): optional length ''(optional)''


=== Returns ===

<code>string</code>

== sql_substr ==

SQL SUBSTR/SUBSTRING with 1-based index and bounds checking

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''value''' (<code>string</code>): string to cut
* '''start''' (<code>number</code>): first character position (1-based)
* '''len''' (<code>number</code>): optional length ''(optional)''


=== Returns ===

<code>string</code>

== simplify ==

turns a stringable input value in the easiest-most value (e.g. turn strings into numbers if they are numeric

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value to simplify


=== Returns ===

<code>any</code>

== strlen ==

returns the length of a string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): input string


=== Returns ===

<code>int</code>

== strlike ==

matches the string against a wildcard pattern using SQL NULL semantics

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''value''' (<code>string</code>): input string
* '''pattern''' (<code>string</code>): pattern with % and _ in them
* '''collation''' (<code>string</code>): collation in which to compare them ''(optional)''


=== Returns ===

<code>bool</code>

== strlike_cs ==

matches the string against a wildcard pattern case-sensitively using SQL NULL semantics

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''value''' (<code>string</code>): input string
* '''pattern''' (<code>string</code>): pattern with % and _ in them
* '''collation''' (<code>string</code>): ignored (present for parser compatibility) ''(optional)''


=== Returns ===

<code>bool</code>


== toLower ==

turns a string into lower case

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): input string


=== Returns ===

<code>string</code>


== toUpper ==

turns a string into upper case

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): input string


=== Returns ===

<code>string</code>

== replace ==

replaces all occurances in a string with another string

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''s''' (<code>string</code>): input string
* '''find''' (<code>string</code>): search string
* '''replace''' (<code>string</code>): replace string


=== Returns ===

<code>string</code>

== strtrim ==

trims whitespace from both ends of a string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): input string


=== Returns ===

<code>string</code>

== strltrim ==

trims whitespace from the left of a string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): input string


=== Returns ===

<code>string</code>

== strrtrim ==

trims whitespace from the right of a string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): input string


=== Returns ===

<code>string</code>

== sql_trim ==

SQL TRIM(): NULL-safe trim of whitespace from both ends

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): input string


=== Returns ===

<code>string</code>

== sql_ltrim ==

SQL LTRIM(): NULL-safe trim of whitespace from left

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): input string


=== Returns ===

<code>string</code>

== sql_rtrim ==

SQL RTRIM(): NULL-safe trim of whitespace from right

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): input string


=== Returns ===

<code>string</code>

== split ==

splits a string using a separator or space

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''value''' (<code>string</code>): input string
* '''separator''' (<code>string</code>): (optional) parameter, defaults to " " ''(optional)''


=== Returns ===

<code>list</code>

== string_repeat ==

repeats a string n times

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''value''' (<code>string</code>): string to repeat
* '''count''' (<code>number</code>): number of repetitions


=== Returns ===

<code>string</code>

== collate ==

returns a canonical order relation for a collation and direction. MemCP allows natural sorting of numeric literals.

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''collation''' (<code>string</code>): collation string of the form LANG or LANG_cs or LANG_ci where LANG is a BCP 47 code, for compatibility to MySQL, a CHARSET_ prefix is allowed and ignored as well as the aliases bin, danish, general, german1, german2, spanish and swedish are allowed for language codes
* '''reverse''' (<code>bool</code>): whether to reverse the order like in ORDER BY DESC ''(optional)''


=== Returns ===

<code>func(a:any, b:any) -> bool</code>

== htmlentities ==

escapes the string for use in HTML

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): input string


=== Returns ===

<code>string</code>

== urlencode ==

encodes a string according to URI coding schema

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): string to encode


=== Returns ===

<code>string</code>

== urldecode ==

decodes a string according to URI coding schema

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): string to decode


=== Returns ===

<code>string</code>

== json_encode ==

encodes a value in JSON, treats lists as lists

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value to encode


=== Returns ===

<code>string</code>

== json_quote ==

quotes a string as JSON

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_encode_assoc ==

encodes a value in JSON, treats lists as associative arrays

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value to encode


=== Returns ===

<code>string</code>

== json_decode ==

parses JSON into a map

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): string to decode


=== Returns ===

<code>any</code>

== json_decode_scmer ==

parses JSON produced by json_encode and preserves Scheme symbols and lists

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): Scmer JSON to decode


=== Returns ===

<code>any</code>

== base64_encode ==

encodes a string as Base64 (standard encoding)

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): binary string to encode


=== Returns ===

<code>string</code>

== base64_decode ==

decodes a Base64 string (standard encoding)

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): base64-encoded string


=== Returns ===

<code>string</code>

== sql_unescape ==

unescapes the inner part of a sql string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): string to decode


=== Returns ===

<code>string</code>

== bin2hex ==

turns binary data into hex with lowercase letters

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): string to encode


=== Returns ===

<code>string</code>


== bin2hex ==

turns binary data into hex with lowercase letters

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): string to encode


=== Returns ===

<code>string</code>

== hex2bin ==

decodes a hex string into binary data

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): hex string (even length)


=== Returns ===

<code>string</code>

== uuid ==

generates a new random UUID v4 string

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>string</code>


== randomBytes ==

returns a string with numBytes cryptographically secure random bytes

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''numBytes''' (<code>number</code>): number of random bytes


=== Returns ===

<code>string</code>

== regexp_replace ==

replaces matches of a regex pattern in a string

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''str''' (<code>string</code>): input string
* '''pattern''' (<code>string</code>): regex pattern
* '''replacement''' (<code>string</code>): replacement string


=== Returns ===

<code>string</code>

== fnv_hash ==

computes a fast non-cryptographic 64-bit FNV-1a hash of a string, returns a 16-character hex string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''str''' (<code>string</code>): input string to hash


=== Returns ===

<code>string</code>

== stable_structural_hash ==

streams the string or serialized representation of a Scheme value into stable FNV-1a without constructing the complete representation

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''value''' (<code>any</code>): value to hash
* '''serialize''' (<code>bool</code>): use the Scheme serializer instead of string rendering ''(optional)''


=== Returns ===

<code>string</code>

== sha1 ==

computes the SHA-1 digest of a string, returns a 40-character lowercase hex string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''str''' (<code>string</code>): input string to hash


=== Returns ===

<code>string</code>

== sha256 ==

computes the SHA-256 digest of a string, returns a 64-character lowercase hex string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''str''' (<code>string</code>): input string to hash


=== Returns ===

<code>string</code>

== regexp_test ==

tests if a string matches a regex pattern, returns true/false

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''str''' (<code>string</code>): input string
* '''pattern''' (<code>string</code>): regex pattern


=== Returns ===

<code>bool</code>

== json_parse_bson ==

validates JSON and returns a BSON-backed value

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_array ==

creates a MySQL-compatible JSON array

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_object ==

creates a MySQL-compatible JSON object

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>


== json_quote ==

quotes a string as JSON

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_unquote ==

unquotes a JSON string

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_valid ==

tests whether a value contains valid JSON

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_type ==

returns the MySQL JSON type name

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_depth ==

returns maximum JSON document depth

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_length ==

returns JSON object or array length

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_extract ==

extracts one or more JSON paths

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_keys ==

returns object member names

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_contains_path ==

tests whether one or all JSON paths exist

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_contains ==

tests JSON containment

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_overlaps ==

tests whether JSON values overlap

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_member_of ==

tests whether a JSON scalar is a member of an array

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_pretty ==

pretty-prints JSON

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_storage_size ==

returns BSON payload size

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_storage_free ==

returns unused BSON storage space

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_schema_valid ==

validates a document against a MySQL Draft 4 JSON schema

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_schema_validation_report ==

reports MySQL Draft 4 JSON schema validation

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_set ==

sets or inserts JSON path values

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_insert ==

inserts missing JSON path values

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_replace ==

replaces existing JSON path values

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_remove ==

removes JSON path values

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_array_append ==

appends values to JSON arrays

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_array_insert ==

inserts values into JSON arrays

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_merge_patch ==

merges documents using RFC 7396 semantics

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_merge_preserve ==

merges documents while preserving values

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_search ==

searches JSON string values

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_value ==

extracts a scalar JSON value

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_to_json ==

implements PostgreSQL to_json and to_jsonb

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_row_to_json ==

implements PostgreSQL row_to_json

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_json_build_array ==

implements PostgreSQL JSON array builders

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_json_array_absent ==

implements SQL/JSON ARRAY ABSENT ON NULL

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_json_build_object ==

implements PostgreSQL JSON object builders

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_json_object ==

implements PostgreSQL json_object and jsonb_object array forms

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_json_serialize ==

implements PostgreSQL json_serialize

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_json_array_length ==

implements PostgreSQL json_array_length

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_json_extract_path ==

implements PostgreSQL json_extract_path

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_json_extract_path_text ==

implements PostgreSQL json_extract_path_text

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_jsonb_set ==

implements PostgreSQL jsonb_set

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_jsonb_set_lax ==

implements PostgreSQL jsonb_set_lax

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_jsonb_insert ==

implements PostgreSQL jsonb_insert

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_json_strip_nulls ==

implements PostgreSQL json_strip_nulls

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_json_typeof ==

implements PostgreSQL json_typeof

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_jsonb_populate_record_valid ==

validates PostgreSQL jsonb_populate_record input shape

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_jsonb_concat ==

implements PostgreSQL jsonb concatenation

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_jsonb_delete ==

implements PostgreSQL jsonb deletion

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_jsonb_delete_path ==

implements PostgreSQL jsonb path deletion

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_subtract ==

dispatches PostgreSQL numeric and jsonb subtraction

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_jsonb_exists ==

implements PostgreSQL jsonb top-level existence

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_jsonb_path_exists ==

implements PostgreSQL jsonb_path_exists

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_jsonb_path_match ==

implements PostgreSQL jsonb_path_match

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_jsonb_path_query_array ==

implements PostgreSQL jsonb_path_query_array

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_jsonb_path_query_first ==

implements PostgreSQL jsonb_path_query_first

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_json_value ==

implements PostgreSQL SQL/JSON JSON_VALUE

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_get ==

implements PostgreSQL JSON object and array extraction

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_arrayagg_entry ==

wraps one JSON_ARRAYAGG input value

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_arrayagg_reduce ==

JSON_ARRAYAGG reducer

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_arrayagg_finalize ==

builds one BSON array from a collected JSON_ARRAYAGG value list

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_objectagg_entry ==

constructs a JSON_OBJECTAGG key/value entry

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== json_objectagg_reduce ==

JSON_OBJECTAGG reducer

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

== pg_json_table_rows ==

materializes a PostgreSQL JSON table function as rows

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''arguments''' (<code>any</code>): ''(variadic)''


=== Returns ===

<code>any</code>

Streams

2026-08-27T06:23:44Z

Carli:

= Streams =

The '''Streams''' module provides stream processing and compression functionality for the SCM programming language. This module includes:

* '''Stream creation''': Functions to create streams from string content (streamString)
* '''Compression''': Stream compression using gzip and xz algorithms
* '''Decompression''': Stream decompression using zcat (gzip) and xzcat (xz) functions

These functions enable efficient handling of data streams, file compression, and decompression operations in SCM programs, particularly useful for processing large datasets and file I/O operations.

← Back to [[Full SCM API documentation]]


== streamString ==

creates a stream that contains a string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''content''' (<code>string</code>): content to put into the stream


=== Returns ===

<code>stream</code>

== gzip ==

compresses a stream with gzip. Create streams with (stream filename)

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''stream''' (<code>stream</code>): input stream


=== Returns ===

<code>stream</code>

== xz ==

compresses a stream with xz. Create streams with (stream filename)

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''stream''' (<code>stream</code>): input stream


=== Returns ===

<code>stream</code>

== zcat ==

turns a compressed gzip stream into a stream of uncompressed data. Create streams with (stream filename)

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''stream''' (<code>stream</code>): input stream


=== Returns ===

<code>stream</code>

== xzcat ==

turns a compressed xz stream into a stream of uncompressed data. Create streams with (stream filename)

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''stream''' (<code>stream</code>): input stream


=== Returns ===

<code>stream</code>

Storage

2026-08-27T06:20:08Z

Carli:

= Storage =

The '''Storage''' module provides functions to manage databases, tables, columns, keys, partitions, and large data operations in SCM.

← Back to [[Full SCM API documentation]]

== table ==

resolves a schema+table name pair into a table handle

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''schema''' (<code>string</code>):
* '''table''' (<code>string</code>):


=== Returns ===

<code>table</code>

== scan_estimate ==

estimate output row count for a table scan

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''table''' (<code>table</code>):


=== Returns ===

<code>int</code>

== table_planner_statistics ==

return the immutable O(1) planner-statistics snapshot for a table

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''table''' (<code>table</code>):


=== Returns ===

<code>any</code>

== scan_selectivity_estimate ==

bounded estimate of visible rows matching a table filter; stops at max_rows and does not log scan telemetry

'''Allowed number of parameters:''' 5–5


=== Parameters ===

* '''tx''' (<code>any</code>): transaction context to use for visibility; usually ((context "session") "__memcp_tx")
* '''table''' (<code>table</code>):
* '''condition_cols''' (<code>list</code>):
* '''condition''' (<code>any</code>):
* '''max_rows''' (<code>int</code>):


=== Returns ===

<code>list</code>


== table_empty? ==

returns true if a table currently has no rows

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''table''' (<code>table</code>):


=== Returns ===

<code>bool</code>

== scan_recset ==

builds a query-local record-set handle from one table scan, or -- when given an existing recset instead of a table -- narrows that recset to the members which also satisfy filter, re-evaluating filter only over its existing membership. The latter is the cheap way to AND a further (possibly subscan-heavy) condition onto an already-narrowed recset without re-touching rows outside it (e.g. evaluating an expensive correlated check only over the rows a cheap selective filter already narrowed a table down to). The returned value is not persisted and can be scanned like a table

'''Allowed number of parameters:''' 4–4


=== Parameters ===

* '''tx''' (<code>any</code>): transaction context to use for visibility; usually ((context "session") "__memcp_tx")
* '''table''' (<code>any</code>): a table, or an existing recset to narrow further
* '''filterColumns''' (<code>list</code>):
* '''filter''' (<code>func(columns:any...) -> bool</code>): lambda function that decides whether a row enters the recset


=== Returns ===

<code>recset</code>

== recset_count ==

returns the number of currently stored recids in a query-local recset

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''recset''' (<code>recset</code>):


=== Returns ===

<code>int</code>

== recset_project_join ==

projects a source recset through key columns into a query-local target-table recset

'''Allowed number of parameters:''' 5–5


=== Parameters ===

* '''tx''' (<code>any</code>): transaction context to use for visibility; usually ((context "session") "__memcp_tx")
* '''source_recset''' (<code>recset</code>):
* '''source_key_columns''' (<code>list</code>):
* '''target_table''' (<code>table</code>):
* '''target_key_columns''' (<code>list</code>):


=== Returns ===

<code>recset</code>

== recset_key_index ==

builds an immutable lookup function for key columns of the rows contained in a query-local recset

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''tx''' (<code>any</code>): transaction context used while reading source keys
* '''source_recset''' (<code>recset</code>):
* '''source_key_columns''' (<code>list</code>):


=== Returns ===

<code>func(key:any...) -> bool</code>

== recset_union ==

combines query-local recsets from the same table and removes duplicate record IDs

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''recsets''' (<code>list</code>):


=== Returns ===

<code>recset</code>

== recset_intersect ==

intersects query-local recsets from the same table

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''recsets''' (<code>list</code>):


=== Returns ===

<code>recset</code>

== recset_difference ==

returns the records from the first query-local recset which occur in none of the following same-table recsets

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''recsets''' (<code>list</code>):


=== Returns ===

<code>recset</code>

== recset_not ==

returns the complement of a query-local recset relative to the currently visible rows of its base table

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''recset''' (<code>recset</code>):


=== Returns ===

<code>recset</code>

== scan_exists ==

returns true if a table contains at least one visible row matching the given filter; uses scan boundary analysis without map/reduce setup

'''Allowed number of parameters:''' 4–4


=== Parameters ===

* '''tx''' (<code>any</code>): transaction context to use for visibility; usually ((context "session") "__memcp_tx")
* '''table''' (<code>table|list|recset</code>):
* '''filterColumns''' (<code>list</code>):
* '''filter''' (<code>func(columns:any...) -> bool</code>): lambda function that decides whether a row exists


=== Returns ===

<code>bool</code>

== scan ==

does an unordered parallel filter-map-reduce pass on a single table and returns the reduced result

'''Allowed number of parameters:''' 6–10


=== Parameters ===

* '''tx''' (<code>any</code>): transaction context to use for visibility and mutations; usually ((context "session") "__memcp_tx")
* '''table''' (<code>table|list|recset</code>): table handle, query-local recset, or a list for temporary data
* '''filterColumns''' (<code>list</code>): physical columns passed to filter before map/reduce; $recset_contains supplies a row-bound RecSet membership closure
* '''filter''' (<code>func(columns:any...) -> bool</code>): lambda function that decides whether a dataset is passed to the map phase. You can use any column of that table as lambda parameter. You should structure your lambda with an (and) at the root element. Every equal? < > <= >= will possibly translated to an indexed scan
* '''mapColumns''' (<code>list</code>): physical columns passed to map after filtering; pseudo columns are $update (update/delete current row), $recset_contains (row-bound RecSet membership), $set:<column>, $increment:<column>, and $invalidate:<column> (computed-column maintenance), plus NEW.<column> in trigger plans
* '''map''' (<code>func(columns:any...) -> any</code>): lambda function to extract data from the dataset. You can use any column of that table as lambda parameter. You can return a value you want to extract and pass to reduce, but you can also directly call insert, print or resultrow functions. If you declare a parameter named '<math display="inline">update', this variable will hold a function that you can use to delete or update a row. Call (</math>update) to delete the dataset, call ($update '("field1" value1 "field2" value2)) to update certain columns.
* '''reduce''' (<code>func(any, any)</code>): (optional) lambda function to aggregate the map results. It takes two parameters (a b) where a is the accumulator and b the new value. The accumulator for the first reduce call is the neutral element. The return value will be the accumulator input for the next reduce call. There are two reduce phases: shard-local and shard-collect. In the shard-local phase, a starts with neutral and b is fed with the return values of each map call. In the shard-collect phase, a starts with neutral and b is fed with the result of each shard-local pass. ''(optional)''
* '''neutral''' (<code>any</code>): (optional) neutral element for the reduce phase, otherwise nil is assumed ''(optional)''
* '''reduce2''' (<code>func(any, any)</code>): (optional) second stage reduce function that will apply a result of reduce to the neutral element/accumulator ''(optional)''
* '''isOuter''' (<code>bool</code>): (optional) if true, in case of no hits, call map once anyway with NULL values ''(optional)''


=== Returns ===

<code>any</code>

== scan_batch ==

does an unordered parallel filter-map-reduce pass on a single table using batchdata-backed #N pseudo columns and returns the reduced result

'''Allowed number of parameters:''' 8–12


=== Parameters ===

* '''tx''' (<code>any</code>): transaction context to use for visibility and mutations; usually ((context "session") "__memcp_tx")
* '''table''' (<code>table|list|recset</code>): table handle, query-local recset, or a list for temporary data
* '''filterColumns''' (<code>list</code>): list of columns that are fed into filter; #0, #1, ... address batchdata slots
* '''filter''' (<code>func(columns:any...) -> bool</code>): lambda function that decides whether a dataset is passed to the map phase
* '''mapColumns''' (<code>list</code>): list of columns that are fed into map; #0, #1, ... address batchdata slots
* '''map''' (<code>func(columns:any...) -> any</code>): lambda function to extract data from the dataset
* '''stride''' (<code>int</code>): number of batchdata entries per batch row
* '''batchdata''' (<code>list</code>): flat batch buffer accessed via #N pseudo columns
* '''reduce''' (<code>func(any, any)</code>): (optional) lambda function to aggregate the map results ''(optional)''
* '''neutral''' (<code>any</code>): (optional) neutral element for the reduce phase, otherwise nil is assumed ''(optional)''
* '''reduce2''' (<code>func(any, any)</code>): (optional) second stage reduce function that will apply a result of reduce to the neutral element/accumulator ''(optional)''
* '''isOuter''' (<code>bool</code>): (optional) if true, in case of no hits, call map once anyway with NULL values ''(optional)''


=== Returns ===

<code>any</code>

== scan_order_batch_accept ==

incrementally scans a table or existing RecSet in scan_order order and applies a RecSet batch filter before OFFSET/LIMIT and map/reduce. The first candidate RecSet contains offset+limit rows; if too few rows are accepted, subsequent disjoint batches contain twice as many candidates until the accepted limit is satisfied or the input is exhausted. batchFilter is called as (batchFilter input_recset) and must return an exact subset RecSet of the same base table and transaction. A simple batchFilter may call (scan_recset tx input_recset filterColumns realFilter); complex filters may project input_recset to another table, apply search/ACL scans and project the result back to the input table. The returned RecSet is used only as a membership mask against the already ordered candidate vector, so output order is preserved without scanning the unordered RecSet again. For non-unique ORDER BY values, include an explicit unique tie-breaker. sortcols/sortdirs may both be empty; that path greedily collects candidates without sorting. limitPartitionCols is present for scan_order signature compatibility and currently must be 0

'''Allowed number of parameters:''' 10–14


=== Parameters ===

* '''tx''' (<code>any</code>): transaction context used consistently by the candidate scan and every batch filter operation; usually ((context "session") "__memcp_tx")
* '''table_or_recset''' (<code>table|recset</code>): base table or complete existing query-local RecSet from which ordered candidate batches are drawn
* '''batchFilter''' (<code>func(input_recset:recset) -> recset</code>): function (lambda (input_recset) accepted_recset). It may naively narrow input_recset with scan_recset, or run arbitrary RecSet projections/search/ACL operations and project back. It must return a same-table, same-transaction subset of input_recset
* '''sortcols''' (<code>list</code>): same as scan_order: columns or computed sort functions. Include a unique tie-breaker for a total repeatable order; use an empty list for greedy unsorted collection
* '''sortdirs''' (<code>list</code>): same as scan_order: one relation per sort column (<, > or collate relation); must also be empty when sortcols is empty
* '''limitPartitionCols''' (<code>number</code>): reserved for scan_order signature compatibility; currently must be 0
* '''offset''' (<code>number</code>): number of batch-filter-accepted rows to skip; it is not the number of driver candidates already examined
* '''limit''' (<code>number</code>): finite maximum number of accepted rows passed to map; the initial candidate batch size is offset+limit and doubles for every subsequent batch
* '''mapColumns''' (<code>list</code>): physical columns passed to map after filtering; pseudo columns are $update (update/delete current row), $recset_contains (row-bound RecSet membership), $set:<column>, $increment:<column>, and $invalidate:<column> (computed-column maintenance), plus NEW.<column> in trigger plans; $break is reserved for internal ORC convergence and must not implement SQL OFFSET/LIMIT, which belong in the native offset and limit arguments
* '''map''' (<code>func(columns:any...) -> any</code>): same map callback contract as scan_order; accepted record IDs are passed to its shard mapper in batches
* '''reduce''' (<code>func(acc:any, val:any) -> any</code>): optional serial reducer over mapped accepted rows, with the same accumulator contract as scan_order ''(optional)''
* '''neutral''' (<code>any</code>): optional neutral element for reduce; defaults to nil ''(optional)''
* '''isOuter''' (<code>bool</code>): optional scan_order-compatible outer behavior: map one NULL row when no accepted row reaches map ''(optional)''
* '''notFoundValue''' (<code>any</code>): optional result when no accepted row reaches map and isOuter is false; defaults to neutral ''(optional)''


=== Returns ===

<code>any</code>

== scan_order ==

does an ordered parallel filter and serial map-reduce pass on a single table and returns the reduced result

'''Allowed number of parameters:''' 11–17


=== Parameters ===

* '''tx''' (<code>any</code>): transaction context to use for visibility and mutations; usually ((context "session") "__memcp_tx")
* '''table''' (<code>table|list|recset</code>): table handle, query-local RecSet, or a list for temporary data
* '''filterColumns''' (<code>list</code>): physical columns passed to filter before map/reduce; $recset_contains supplies a row-bound RecSet membership closure
* '''filter''' (<code>func(columns:any...) -> bool</code>): lambda function that decides whether a dataset is passed to the map phase. You can use any column of that table as lambda parameter. You should structure your lambda with an (and) at the root element. Every equal? < > <= >= will possibly translated to an indexed scan
* '''sortcols''' (<code>list</code>): list of columns to sort. Each column is either a string to point to an existing column or a func(cols...)->any to compute a sortable value
* '''sortdirs''' (<code>list</code>): list of column directions to sort. Must be same length as sortcols. < means ascending, > means descending, (collate ...) will add collations
* '''limitPartitionCols''' (<code>number</code>): number of leading sort columns that form the partition key for per-partition offset/limit. 0 (default) means global offset/limit.
* '''offset''' (<code>number</code>): number of globally ordered, filter-accepted items to skip before map; apply SQL OFFSET here rather than in map
* '''limit''' (<code>number</code>): maximum globally ordered, filter-accepted items passed to map; -1 means unlimited; apply SQL LIMIT here so shard-local Top-K and the global merge can brake early
* '''mapColumns''' (<code>list</code>): physical columns passed to map after filtering; pseudo columns are $update (update/delete current row), $recset_contains (row-bound RecSet membership), $set:<column>, $increment:<column>, and $invalidate:<column> (computed-column maintenance), plus NEW.<column> in trigger plans; $break is reserved for internal ORC convergence and must not implement SQL OFFSET/LIMIT, which belong in the native offset and limit arguments
* '''map''' (<code>func(columns:any...) -> any</code>): lambda function to extract data from the dataset. You can use any column of that table as lambda parameter. You can return a value you want to extract and pass to reduce, but you can also directly call insert, print or resultrow functions. If you declare a parameter named '<math display="inline">update', this variable will hold a function that you can use to delete or update a row. Call (</math>update) to delete the dataset, call ($update '("field1" value1 "field2" value2)) to update certain columns.
* '''reduce''' (<code>func(acc:any, val:any) -> any</code>): (optional) lambda function to aggregate the map results. It takes two parameters (a b) where a is the accumulator and b the new value. The accumulator for the first reduce call is the neutral element. The return value will be the accumulator input for the next reduce call. There are two reduce phases: shard-local and shard-collect. In the shard-local phase, a starts with neutral and b is fed with the return values of each map call. In the shard-collect phase, a starts with neutral and b is fed with the result of each shard-local pass. ''(optional)''
* '''neutral''' (<code>any</code>): (optional) neutral element for the reduce phase, otherwise nil is assumed ''(optional)''
* '''isOuter''' (<code>bool</code>): (optional) if true, in case of no hits, call map once anyway with NULL values ''(optional)''
* '''notFoundValue''' (<code>any</code>): (optional) result for no hits when isOuter is false; defaults to neutral ''(optional)''
* '''postOrderFilterColumns''' (<code>list</code>): (optional) columns for a predicate evaluated in global order before OFFSET/LIMIT are counted; use for expensive acceptance checks that cannot participate in index boundaries ''(optional)''
* '''postOrderFilter''' (<code>func(columns:any...) -> bool</code>): (optional) late acceptance predicate. Rejected rows do not count toward OFFSET/LIMIT and never reach map. SQL plans use this instead of callback-driven $break control flow ''(optional)''


=== Returns ===

<code>any</code>

== scan_order_multi ==

does an ordered parallel filter and serial map-reduce pass across multiple tables simultaneously, merging results into a single sorted stream

'''Allowed number of parameters:''' 13–17


=== Parameters ===

* '''tx''' (<code>any</code>): transaction context
* '''tables''' (<code>list</code>): list of table handles
* '''filterColumns''' (<code>list</code>): list of filter column lists, one per table
* '''filterFns''' (<code>list</code>): list of filter lambdas, one per table
* '''sortcols''' (<code>list</code>): list of sort column lists, one per table
* '''sortdirs''' (<code>list</code>): list of sort direction comparators (shared)
* '''perTableOffset''' (<code>list</code>): per-table offset (list of int; -1 disables)
* '''perTableLimit''' (<code>list</code>): per-table limit (list of int; -1 disables)
* '''limitPartitionCols''' (<code>number</code>): number of leading sort columns forming partition key
* '''offset''' (<code>number</code>): number of items to skip (global)
* '''limit''' (<code>number</code>): max number of items to read (global; -1 = unlimited)
* '''mapColumns''' (<code>list</code>): list of map column lists, one per table
* '''mapFns''' (<code>list</code>): list of map lambdas, one per table
* '''reduce''' (<code>func</code>): (optional) aggregation function ''(optional)''
* '''neutral''' (<code>any</code>): (optional) neutral element for reduce ''(optional)''
* '''isOuter''' (<code>bool</code>): (optional) if true, emit null row when no hits ''(optional)''
* '''notFoundValue''' (<code>any</code>): (optional) result for no hits when isOuter is false; defaults to neutral ''(optional)''


=== Returns ===

<code>any</code>

== createdatabase ==

creates a new database

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''schema''' (<code>string</code>): name of the new database
* '''ignoreexists''' (<code>bool</code>): if true, return false instead of throwing an error ''(optional)''


=== Returns ===

<code>bool</code>

== dropdatabase ==

drops a database

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''schema''' (<code>string</code>): name of the database
* '''ifexists''' (<code>bool</code>): if true, don't throw an error if it doesn't exist ''(optional)''


=== Returns ===

<code>bool</code>

== checktablemaintenance ==

checks whether a user-initiated maintenance operation is allowed for a table

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''schema''' (<code>string</code>):
* '''table''' (<code>string</code>):
* '''operation''' (<code>string</code>):


=== Returns ===

<code>bool</code>

== maintenance_capabilities ==

returns the server-side maintenance capabilities for a database or table

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''schema''' (<code>string</code>):
* '''table''' (<code>string</code>): ''(optional)''


=== Returns ===

<code>list</code>

== createtable ==

creates a table, runs its oninit option and registered after-create-table lifecycle triggers synchronously, and returns only after initialization completes; concurrent if-not-exists callers wait for that same completion

'''Allowed number of parameters:''' 4–5


=== Parameters ===

* '''schema''' (<code>string</code>): name of the existing database that will contain the table
* '''table''' (<code>string</code>): name of the table to create
* '''cols''' (<code>list</code>): column and constraint definitions: ("column" name type dimensions typeparams), ("unique" name columns), or ("foreign" name local_columns referenced_table referenced_columns update_mode delete_mode). dimensions is a list of integer type dimensions. typeparams is an alternating key/value list supporting primary (bool), unique (bool), auto_increment (bool), null (bool), default (any), default_expression (string), update (expression), comment (string), collate (string), temp (bool), filtercols (string list), filter (function), sortcols (string list), sortdirs (bool list), partitioncount (integer), mapcols (string list), mapfn (function), reducefn (function), and reduceinit (any). Column lists are string lists; foreign-key modes are restrict, cascade, or set null
* '''options''' (<code>list</code>): alternating key/value list; supported keys are engine (safe, logged, sloppy, memory, or cache), collation (string), charset (string), comment (string), auto_increment (non-negative integer), and oninit (closed zero-argument function run synchronously once per data generation; concurrent if-not-exists callers wait for it, and memory/cache tables persist the callback so the first idempotent createtable after restart repopulates their empty data)
* '''ifnotexists''' (<code>bool</code>): when true, return false instead of failing if the table exists; if another caller is still creating it, wait for that caller's after-create-table initialization before returning false ''(optional)''


=== Returns ===

<code>bool</code>

== createcolumn ==

creates a new column in table

'''Allowed number of parameters:''' 5–7


=== Parameters ===

* '''table''' (<code>table</code>):
* '''colname''' (<code>string</code>): name of the new column
* '''type''' (<code>string</code>): name of the basetype
* '''dimensions''' (<code>list</code>): dimensions of the type (e.g. for decimal)
* '''options''' (<code>list</code>): assoc list: primary, unique, auto_increment, null, comment, default, default_expression, collate; ORC: sortcols, sortdirs, partitioncount, mapcols, mapfn, reducefn, reduceinit
* '''computorCols''' (<code>list</code>): list of columns that is passed into params of computor ''(optional)''
* '''computor''' (<code>func(columns:any...) -> any</code>): lambda expression that can take other column values and computes the value of that column ''(optional)''


=== Returns ===

<code>bool</code>

== createkey ==

creates a new key on a table

'''Allowed number of parameters:''' 4–4


=== Parameters ===

* '''table''' (<code>table</code>):
* '''keyname''' (<code>string</code>): name of the new key
* '''unique''' (<code>bool</code>): whether the key is unique
* '''columns''' (<code>list</code>): list of columns to include


=== Returns ===

<code>bool</code>

== createforeignkey ==

creates a new foreign key on a table

'''Allowed number of parameters:''' 7–7


=== Parameters ===

* '''table1''' (<code>table</code>):
* '''keyname''' (<code>string</code>): name of the new key
* '''columns1''' (<code>list</code>): list of columns to include
* '''table2''' (<code>table</code>):
* '''columns2''' (<code>list</code>): list of columns to include
* '''updatemode''' (<code>string</code>): restrict|cascade|set null
* '''deletemode''' (<code>string</code>): restrict|cascade|set null


=== Returns ===

<code>bool</code>

== shardcolumn ==

tells us how it would partition a column according to their values. Returns a list of pivot elements.

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''table''' (<code>table</code>):
* '''colname''' (<code>string</code>): name of the column
* '''numpartitions''' (<code>number</code>): number of partitions; optional. leave 0 if you want to detect the partiton number automatically or copy the partition schema of the table ''(optional)''


=== Returns ===

<code>list</code>

== partitiontable ==

suggests a partition scheme for a table. If the table has no partition scheme yet, it will immediately apply that scheme and return true. If the table already has a partition scheme, it will alter the partitioning score such that the partitioning scheme is considered in the next repartitioning and return false.

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''table''' (<code>table</code>):
* '''columns''' (<code>list</code>): associative list of string -> list representing column name -> pivots. You can compute pivots by (shardcolumn ...)


=== Returns ===

<code>bool</code>

== altertable ==

alters a table

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''table''' (<code>table</code>):
* '''operation''' (<code>string</code>): one of owner|drop|engine|collation|auto_increment
* '''parameter''' (<code>any</code>): name of the column to drop or value of the parameter


=== Returns ===

<code>bool</code>

== altercolumn ==

alters a column

'''Allowed number of parameters:''' 4–4


=== Parameters ===

* '''table''' (<code>table</code>):
* '''column''' (<code>string</code>): name of the column
* '''operation''' (<code>string</code>): one of drop|type|collation|auto_increment|comment
* '''parameter''' (<code>any</code>): name of the column to drop or value of the parameter


=== Returns ===

<code>bool</code>

== droptable ==

removes a table

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''schema''' (<code>string</code>):
* '''table''' (<code>string</code>):
* '''ifexists''' (<code>bool</code>): if true, don't throw an error if it already exists ''(optional)''


=== Returns ===

<code>bool</code>

== dropcolumn ==

drops a column from a table

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''table''' (<code>table</code>):
* '''column''' (<code>string</code>): name of the column to drop


=== Returns ===

<code>bool</code>

== migratedropcolumn ==

drops a legacy system column during startup migration

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''table''' (<code>table</code>):
* '''column''' (<code>string</code>): legacy column name


=== Returns ===

<code>bool</code>

== invalidatecolumn ==

marks all values of a computed column as stale

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''table''' (<code>table</code>):
* '''column''' (<code>string</code>): name of the computed column


=== Returns ===

<code>bool</code>

== invalidateorc ==

invalidates ORC column rows from a sort key onwards via validMask scan

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''table''' (<code>table</code>):
* '''column''' (<code>string</code>): name of the ORC column
* '''sortkeys''' (<code>list</code>): composite sort key values from which to invalidate


=== Returns ===

<code>bool</code>

== register_keytable_cleanup ==

registers triggers on a base table to maintain keytable entries (insert/delete group keys)

'''Allowed number of parameters:''' 4–4


=== Parameters ===

* '''base_table''' (<code>table</code>):
* '''kt_table''' (<code>table</code>):
* '''tblvar''' (<code>string</code>): table alias used in scan column prefixes
* '''key_pairs''' (<code>list</code>): list of (base_col kt_col) pairs


=== Returns ===

<code>bool</code>

== initialize_cache_table ==

registers maintenance, locks source tables for a consistent snapshot, and runs a canonical planner-cache initializer exactly once

'''Allowed number of parameters:''' 5–6


=== Parameters ===

* '''transaction''' (<code>any</code>): explicit transaction context carrying query-session ownership
* '''table''' (<code>table</code>):
* '''source_tables''' (<code>list</code>):
* '''register_maintenance''' (<code>func</code>):
* '''initializer''' (<code>func</code>):
* '''finalizer''' (<code>func</code>): optional zero-argument finalizer run under the same source-table locks after initialization ''(optional)''


=== Returns ===

<code>bool</code>

== touch_keytable ==

extends the lease on a keytable so CacheManager defers eviction

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''table''' (<code>table</code>):


=== Returns ===

<code>bool</code>

== locktables ==

acquires WRITE or READ user-level locks on a list of tables (LOCK TABLES); implicitly releases any previously held locks

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''locks''' (<code>list</code>): flat list of schema, table, write? triples


=== Returns ===

<code>bool</code>

== unlocktables ==

releases all user-level table locks held by this session

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>bool</code>

== get_fk_target ==

returns (ref_table ref_column) if a single-column FK exists for the given column, nil otherwise

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''table''' (<code>table</code>):
* '''column''' (<code>string</code>): column name


=== Returns ===

<code>any</code>

== renametable ==

renames a table

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''schema''' (<code>string</code>): name of the database
* '''oldname''' (<code>string</code>): current name of the table
* '''newname''' (<code>string</code>): new name of the table


=== Returns ===

<code>bool</code>

== insert ==

inserts a new dataset into table and returns the number of successful items

'''Allowed number of parameters:''' 3–7


=== Parameters ===

* '''table''' (<code>table</code>):
* '''columns''' (<code>list</code>): list of column names, e.g. '("ID", "value")
* '''datasets''' (<code>list</code>): list of list of column values, e.g. '('(1 10) '(2 15))
* '''onCollisionCols''' (<code>list</code>): list of columns of the old dataset that have to be passed to onCollision. Can also request $update, $set:<computed-column>, or NEW.<insert-column>. ''(optional)''
* '''onCollision''' (<code>func</code>): function called for each collision. Its positional parameters are the values requested by onCollisionCols, in the same order. If omitted, collisions raise an error. ''(optional)''
* '''mergeNull''' (<code>bool</code>): if true, it will handle NULL values as equal according to SQL 2003's definition of DISTINCT (https://en.wikipedia.org/wiki/Null_(SQL)#When_two_nulls_are_equal:_grouping,_sorting,_and_some_set_operations) ''(optional)''
* '''onInsertid''' (<code>func(id:number) -> any</code>): (optional) callback (id)->any; called once with the first auto_increment id assigned for this INSERT ''(optional)''


=== Returns ===

<code>number</code>

== stat ==

return system statistics as assoc: mem_available, mem_total, process_memory, shard_memory, shard_budget, persisted_memory, persisted_budget, cache_entry_count, cache_entry_size. (stat schema) and (stat schema tbl) return a string with detailed memory usage.

'''Allowed number of parameters:''' 0–2


=== Parameters ===

* '''schema''' (<code>string</code>): (optional) database name for detailed string output ''(optional)''
* '''table''' (<code>string</code>): (optional) table name for detailed string output ''(optional)''


=== Returns ===

<code>any</code>

== totalmem ==

Returns total physical memory in bytes (from /proc/meminfo)

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>number</code>

== resolve_column_name ==

resolve a physical column name from immutable table metadata

'''Allowed number of parameters:''' 4–4


=== Parameters ===

* '''schema''' (<code>string</code>): database name
* '''table''' (<code>string</code>): table name
* '''column''' (<code>string</code>): column name
* '''ignorecase''' (<code>bool</code>): whether identifier case is ignored


=== Returns ===

<code>string|nil</code>

== show ==

show databases/tables/columns/shards

(show) lists database names (show schema) lists table names (show table_handle) lists the memoized column defs (show table_handle true) returns table metadata (show table_handle "statistics") returns index statistics (show schema true) lists tables with full info: [{name,engine,row_count,size_bytes,collation,comment},...] (show schema tbl) lists column defs (show schema tbl true) returns assoc {columns,meta,shards} (show schema tbl N) returns shard N overview assoc {shard,state,main_count,delta,deletions,size_bytes} (show schema tbl N true) returns shard N full assoc adding columns and indexes (show schema tbl "statistics") returns index statistics (used by INFORMATION_SCHEMA)

'''Allowed number of parameters:''' 0–4


=== Parameters ===

* '''schema_or_table''' (<code>string|table|recset</code>): (optional) database name or resolved table/recset handle ''(optional)''
* '''table_or_property''' (<code>string|bool</code>): (optional) table name, true for full info, or "statistics" for a handle ''(optional)''
* '''property''' (<code>int|bool|string</code>): (optional) shard index (int), true for full table info, or "statistics" ''(optional)''
* '''full''' (<code>bool</code>): (optional) true to include columns and indexes in shard detail ''(optional)''


=== Returns ===

<code>any</code>

== show_triggers ==

show triggers for a given table

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''schema''' (<code>string</code>): database name
* '''table''' (<code>string</code>): (optional) table name, if omitted shows all triggers in schema ''(optional)''


=== Returns ===

<code>any</code>

== rebuild ==

rebuilds main storages and returns the amount of time it took; with a table handle, rebuilds only that table

'''Allowed number of parameters:''' 0–3


=== Parameters ===

* '''table_or_all''' (<code>bool|table</code>): table handle for a table-local rebuild; otherwise whether to rebuild unchanged shards globally (default: false) ''(optional)''
* '''all_or_repartition''' (<code>bool</code>): with a table: whether to rebuild unchanged shards; globally: whether to repartition (default: true) ''(optional)''
* '''repartition''' (<code>bool</code>): with a table handle, whether to repartition that table (default: true) ''(optional)''


=== Returns ===

<code>string</code>


== loadCSV ==

loads a CSV stream into a table and returns the amount of time it took. The first line of the file must be the headlines. The headlines must match the table's columns exactly.

'''Allowed number of parameters:''' 3–5


=== Parameters ===

* '''schema''' (<code>string</code>): name of the database
* '''table''' (<code>string</code>): name of the table
* '''stream''' (<code>stream</code>): CSV file, load with: (stream filename)
* '''delimiter''' (<code>string</code>): (optional) delimiter defaults to ";" ''(optional)''
* '''firstline''' (<code>bool</code>): (optional) if the first line contains the column names (otherwise, the tables column order is used) ''(optional)''


=== Returns ===

<code>string</code>


== loadJSON ==

loads a .jsonl file from stream into a database and returns the amount of time it took. JSONL is a linebreak separated file of JSON objects. Each JSON object is one dataset in the database. Before you add rows, you must declare the table in a line '#table <tablename>'. All other lines starting with # are comments. Columns are created dynamically as soon as they occur in a json object.

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''schema''' (<code>string</code>): name of the database where you want to put the tables in
* '''stream''' (<code>stream</code>): stream of the .jsonl file, read with: (stream filename)


=== Returns ===

<code>string</code>

== settings ==

reads or writes a global settings value. This modifies your data/settings.json.

'''Allowed number of parameters:''' 0–2


=== Parameters ===

* '''key''' (<code>string</code>): name of the key to set or get (for reference, rts) ''(optional)''
* '''value''' (<code>any</code>): new value of that setting ''(optional)''


=== Returns ===

<code>any</code>

== createcreatetabletrigger ==

registers a lifecycle trigger that fires synchronously after a future createtable for the given schema/table succeeds

'''Allowed number of parameters:''' 6–6


=== Parameters ===

* '''schema''' (<code>string</code>): name of the database
* '''table''' (<code>string</code>): name of the table to watch for creation
* '''name''' (<code>string</code>): name of the trigger
* '''source_sql''' (<code>string</code>): original SQL body text (for diagnostics)
* '''body''' (<code>any</code>): trigger body (Scheme procedure or deferred trigger expression)
* '''visible''' (<code>bool</code>): true = user trigger, false = internal trigger


=== Returns ===

<code>bool</code>

== dropcreatetabletrigger ==

removes a registered create-table lifecycle trigger

'''Allowed number of parameters:''' 4–4


=== Parameters ===

* '''schema''' (<code>string</code>): name of the database
* '''table''' (<code>string</code>): name of the table watched for creation
* '''name''' (<code>string</code>): name of the trigger
* '''ifexists''' (<code>bool</code>): don't throw error if trigger doesn't exist


=== Returns ===

<code>bool</code>

== createtrigger ==

creates a new trigger on a table

'''Allowed number of parameters:''' 6–6


=== Parameters ===

* '''table''' (<code>table</code>):
* '''name''' (<code>string</code>): name of the trigger
* '''timing''' (<code>string</code>): one of: before_insert, after_insert, before_update, after_update, before_delete, after_delete
* '''source_sql''' (<code>string</code>): original SQL body text (for SHOW TRIGGERS)
* '''body''' (<code>any</code>): trigger body (parsed Scheme expression)
* '''visible''' (<code>bool</code>): true = user trigger (shown in SHOW TRIGGERS), false = internal trigger (hidden)


=== Returns ===

<code>bool</code>

== droptrigger ==

removes a trigger from a table

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''schema''' (<code>string</code>): name of the database
* '''name''' (<code>string</code>): name of the trigger
* '''ifexists''' (<code>bool</code>): don't throw error if trigger doesn't exist


=== Returns ===

<code>bool</code>

== mysql_import ==

imports schema+data from a MySQL server into MemCP

'''Allowed number of parameters:''' 4–8


=== Parameters ===

* '''host''' (<code>string|nil</code>): MySQL host (nil => 127.0.0.1)
* '''port''' (<code>int|nil</code>): MySQL port (nil => 3306)
* '''username''' (<code>string</code>): MySQL username
* '''password''' (<code>string</code>): MySQL password
* '''sourcedb''' (<code>string|nil</code>): source database (omit/nil => all non-system dbs) ''(optional)''
* '''targetdb''' (<code>string|nil</code>): target database (omit/nil => sourcedb) ''(optional)''
* '''sourcetable''' (<code>string|nil</code>): source table (omit/nil => all tables in sourcedb) ''(optional)''
* '''targettable''' (<code>string|nil</code>): target table (omit/nil => sourcetable) ''(optional)''


=== Returns ===

<code>bool</code>

== psql_import ==

imports schema+data from a PostgreSQL server into MemCP

'''Allowed number of parameters:''' 4–9


=== Parameters ===

* '''host''' (<code>string|nil</code>): PostgreSQL host (nil => 127.0.0.1)
* '''port''' (<code>int|nil</code>): PostgreSQL port (nil => 5432)
* '''username''' (<code>string</code>): PostgreSQL username
* '''password''' (<code>string</code>): PostgreSQL password
* '''sourcedb''' (<code>string|nil</code>): source database (omit/nil => all non-system dbs) ''(optional)''
* '''sourceschema''' (<code>string|nil</code>): source schema (omit/nil => all non-system schemas in sourcedb) ''(optional)''
* '''targetdb''' (<code>string|nil</code>): target database (omit/nil => sourcedb) ''(optional)''
* '''sourcetable''' (<code>string|nil</code>): source table (omit/nil => all tables in sourceschema) ''(optional)''
* '''targettable''' (<code>string|nil</code>): target table (omit/nil => sourcetable) ''(optional)''


=== Returns ===

<code>bool</code>

SCM Builtins

2026-08-27T06:19:26Z

Carli:

= SCM Builtins =

The '''SCM Builtins''' module provides the core foundational functions for the SCM (Scheme) programming language implementation. These built-in functions handle essential operations including:

* '''Language constructs''': Control flow (if, and, or), variable management (define, set), and code evaluation (eval, quote)
* '''Function operations''': Lambda creation, function application, and error handling (try, error)
* '''Data manipulation''': Type conversion (string, symbol), list operations, and pattern matching
* '''Development tools''': Code optimization, timing measurements, and debugging support
* '''Execution control''': Parallel processing, variable scoping, and source code annotation

This module forms the foundation upon which all other SCM functionality is built, providing the essential primitives needed for Scheme programming.

← Back to [[Full SCM API documentation]]

== quote ==

returns a symbol or list without evaluating it

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''symbol''' (<code>symbol</code>): symbol to quote


=== Returns ===

<code>symbol</code>

== eval ==

executes the given scheme program in the current environment

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''code''' (<code>list</code>): list with head and optional parameters


=== Returns ===

<code>any</code>

== size ==

compute the memory size of a value

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value to examine


=== Returns ===

<code>int</code>

== optimize ==

optimize the given scheme program and optionally report telemetry after completion

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''code''' (<code>list</code>): list with head and optional parameters
* '''telemetry_callback''' (<code>func(assoc) -> any</code>): optional callback invoked once with optimizer telemetry ''(optional)''


=== Returns ===

<code>any</code>

== time ==

measures the time it takes to compute the first argument

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''code''' (<code>any</code>): code to execute
* '''label''' (<code>string</code>): label to print in the log or trace ''(optional)''


=== Returns ===

<code>any</code>

== if ==

checks a condition and then conditionally evaluates code branches; there might be multiple condition+true-branch clauses

'''Allowed number of parameters:''' 2–10000


=== Parameters ===

* '''condition...''' (<code>any</code>): condition to evaluate
* '''true-branch...''' (<code>returntype</code>): code to evaluate if condition is true
* '''false-branch''' (<code>any</code>): code to evaluate if condition is false ''(variadic)''


=== Returns ===

<code>returntype</code>

== and ==

lazily combines conditions using SQL three-valued logic; returns false on the first false value, nil for UNKNOWN, otherwise true

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''condition''' (<code>bool</code>): condition to evaluate ''(variadic)''


=== Returns ===

<code>bool</code>

== or ==

lazily combines conditions using SQL three-valued logic; returns true on the first true value, nil for UNKNOWN, otherwise false

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''condition''' (<code>any</code>): condition to evaluate ''(variadic)''


=== Returns ===

<code>bool</code>

== coalesce ==

returns the first value that has a non-zero value

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value''' (<code>returntype</code>): value to examine ''(variadic)''


=== Returns ===

<code>returntype</code>


== coalesceNil ==

returns the first value that has a non-nil value

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value''' (<code>returntype</code>): value to examine ''(variadic)''


=== Returns ===

<code>returntype</code>

== define ==

defines or sets a variable in the current environment

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''variable''' (<code>symbol</code>): variable to set
* '''value''' (<code>returntype</code>): value to set the variable to


=== Returns ===

<code>bool</code>

== set ==

defines or sets a variable in the current environment

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''variable''' (<code>symbol</code>): variable to set
* '''value''' (<code>returntype</code>): value to set the variable to


=== Returns ===

<code>bool</code>

== error ==

halts the whole execution thread and throws an error message

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>any</code>): value or message to throw ''(variadic)''


=== Returns ===

<code>string</code>

== try ==

tries to execute a function and returns its result. In case of a failure, the error is fed to the second function and its result value will be used

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''func''' (<code>func</code>): function with no parameters that will be called
* '''errorhandler''' (<code>func(error:any) -> any</code>): function that takes the error as parameter


=== Returns ===

<code>any</code>

== apply ==

runs the function with its arguments

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''function''' (<code>func</code>): function to execute
* '''arguments''' (<code>list</code>): list of arguments to apply


=== Returns ===

<code>any</code>

== apply_assoc ==

runs the function with its arguments but arguments is a assoc list

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''function''' (<code>func</code>): function to execute (must be a lambda)
* '''arguments''' (<code>list</code>): assoc list of arguments to apply


=== Returns ===

<code>symbol</code>

== symbol ==

returns a symbol built from that string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): string value that will be converted into a symbol


=== Returns ===

<code>symbol</code>

== list ==

constructs a list from its arguments

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''items''' (<code>any</code>): items to put into the list ''(variadic)''


=== Returns ===

<code>list</code>

== for ==

Sequential loop over a list state; applies a condition and step function and returns the final state list. Use only when iterations have strong data dependencies and must run sequentially.

Examples:

* Count to 10: (for '(0) (lambda (x) (< x 10)) (lambda (x) (list (+ x 1)))) => '(10)
* Sum 0..9: (for '(0 0) (lambda (x sum) (< x 10)) (lambda (x sum) (list (+ x 1) (+ sum x)))) => '(10 45)

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''init''' (<code>list</code>): initial state as a list
* '''condition''' (<code>func(state:any...) -> bool</code>): func that receives the current state as parameters and must return true if the loop shall be continued
* '''step''' (<code>func(state:any...) -> list</code>): step func that returns the next state as a list


=== Returns ===

<code>list</code>

== string ==

converts the given value into string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): any value


=== Returns ===

<code>string</code>

== match ==

takes a value evaluates the branch that first matches the given pattern Patterns can be any of:

* symbol matches any value and stores is into a variable
* "string" (matches only this string)
* number (matches only this value)
* (symbol "something") will only match the symbol 'something'
* '(subpattern subpattern...) matches a list with exactly these subpatterns
* (concat str1 str2 str3) will decompose a string into one of the following patterns: "prefix" variable, variable "postfix", variable "infix" variable
* (cons a b) will reverse the cons function, so it will match the head of the list with a and the rest with b
* (regex "pattern" text var1 var2...) will match the given regex pattern, store the whole string into text and all capture groups into var1, var2...

'''Allowed number of parameters:''' 3–10000


=== Parameters ===

* '''value''' (<code>any</code>): value to evaluate
* '''pattern...''' (<code>any</code>): pattern
* '''result...''' (<code>returntype</code>): result value when the pattern matches; this code can use the variables matched in the pattern
* '''default''' (<code>any</code>): (optional) value that is returned when no pattern matches ''(variadic)''


=== Returns ===

<code>any</code>

== lambda ==

returns a function (func) constructed from the given code

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''parameters''' (<code>symbol|list|nil</code>): if you provide a parameter list, you will have named parameters. If you provide a single symbol, the list of parameters will be provided in that symbol
* '''code''' (<code>any</code>): value that is evaluated when the lambda is called. code can use the parameters provided in the declaration as well es the scope above
* '''numvars''' (<code>number</code>): number of unnamed variables that can be accessed via (var 0) (var 1) etc. ''(optional)''


=== Returns ===

<code>func</code>

== begin ==

creates a own variable scope, evaluates all sub expressions and returns the result of the last one

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''expression...''' (<code>any</code>): expressions to evaluate ''(variadic)''


=== Returns ===

<code>any</code>

== parallel ==

executes all parameters in parallel and returns nil if they are finished

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''expression...''' (<code>any</code>): expressions to evaluate in parallel ''(variadic)''


=== Returns ===

<code>any</code>

== source ==

annotates the node with filename and line information for better backtraces

'''Allowed number of parameters:''' 4–4


=== Parameters ===

* '''filename''' (<code>string</code>): Filename of the code
* '''line''' (<code>number</code>): Line of the code
* '''column''' (<code>number</code>): Column of the code
* '''code''' (<code>returntype</code>): code


=== Returns ===

<code>returntype</code>

== source_coverage_report ==

returns Scheme source coverage statistics, optionally filtered by source path prefix

'''Allowed number of parameters:''' 0–1


=== Parameters ===

* '''prefix''' (<code>string</code>): source path prefix ''(optional)''


=== Returns ===

<code>assoc</code>

== scheme ==

parses a scheme expression into a list

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''code''' (<code>string</code>): Scheme code
* '''filename''' (<code>string</code>): optional filename ''(optional)''


=== Returns ===

<code>any</code>

== serialize ==

serializes a piece of code into a (hopefully) reparsable string; you shall be able to send that code over network and reparse with (scheme)

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''code''' (<code>list</code>): Scheme code


=== Returns ===

<code>string</code>

== pretty_print ==

formats Scheme code as an indented, human-readable string; expressions up to width characters are kept on one line, longer ones are expanded with one argument per line

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''code''' (<code>list</code>): Scheme code to format
* '''width''' (<code>int</code>): max characters before expanding (default 20) ''(optional)''


=== Returns ===

<code>string</code>

Parsers

2026-08-27T06:17:42Z

Carli:

= Parsers =

The '''Parsers''' module provides parsing functionality for the SCM programming language. This module includes:

* '''Parser creation''': Functions to create custom parsers using grammar syntax (parser)
* '''Grammar support''': Support for various parser types including AtomParser, RegexParser, AndParser, OrParser, KleeneParser, ManyParser, MaybeParser, and more
* '''Packrat parsing''': Implementation of packrat parsing algorithms for efficient parsing
* '''Whitespace handling''': Configurable whitespace skipping mechanisms

These functions provide essential tools for creating custom parsers and processing structured text data in SCM programs.

← Back to [[Full SCM API documentation]]

== parser ==

creates a parser

<pre>Scm parsers work this way:
(parser syntax scmerresult) -> func

syntax can be one of:
(parser syntax scmerresult) will execute scmerresult after parsing syntax
(parser syntax scmerresult "skipper") will add a different whitespace skipper regex to the root parser
(define var syntax) valid inside (parser...), stores the result of syntax into var for use in scmerresult
"str" AtomParser
(atom "str" caseinsensitive skipws) AtomParser
(regex "asdf" caseinsensitive skipws) RegexParser
'(a b c) AndParser
(or a b c) OrParser
(* sub separator noMemo) KleeneParser
(+ sub separator) ManyParser
(? xyz) MaybeParser (if >1 AndParser)
(not mainparser parser1 parser2 parser3 ...) a parser that matches mainparser but not parser1...
(capture subparser) wraps a parser and returns (matched_text parsed_result)
$ EndParser
empty EmptyParser
symbol -> use other parser defined in env

for further details on packrat parsers, take a look at https://github.com/launix-de/go-packrat</pre>
'''Allowed number of parameters:''' 1–3


=== Parameters ===

* '''syntax''' (<code>any</code>): syntax of the grammar (see docs)
* '''generator''' (<code>any</code>): (optional) expressions to evaluate. All captured variables are available in the scope. ''(optional)''
* '''skipper''' (<code>string</code>): (optional) string that defines the skip mechanism for whitespaces as regexp ''(optional)''


=== Returns ===

<code>func</code>

Lists

2026-08-27T06:15:56Z

Carli:

= Lists =

The '''Lists''' module provides comprehensive list manipulation and processing functions for the SCM programming language. This module includes:

* '''List operations''': Basic operations like counting (count), accessing elements (nth), and type checking (list?)
* '''List construction''': Building lists with append, cons, and unique operations (append_unique, merge_unique)
* '''List deconstruction''': Extracting parts with car (head), cdr (tail), and filtering operations
* '''Functional programming''': Higher-order functions like map, filter, reduce, and produce for advanced list processing
* '''List utilities''': Searching (has?, contains?), merging, zipping, and flattening operations
* '''List generation''': Creating sequences and ranges with produce and produceN functions

These functions provide the essential tools for working with lists as the primary data structure in functional programming with SCM.

← Back to [[Full SCM API documentation]]

== list ==

constructs a list from its arguments

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''items''' (<code>any</code>): items to put into the list ''(variadic)''


=== Returns ===

<code>list</code>

== count ==

counts the number of elements in the list

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''list''' (<code>list</code>): base list


=== Returns ===

<code>int</code>

== nth ==

get the nth item of a list

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''list''' (<code>list</code>): base list
* '''index''' (<code>number</code>): index beginning from 0


=== Returns ===

<code>any</code>

== slice ==

extract a sublist from start (inclusive) to end (exclusive). (slice list start end) returns elements list[start..end).

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''list''' (<code>list</code>): base list
* '''start''' (<code>number</code>): start index (inclusive)
* '''end''' (<code>number</code>): end index (exclusive)


=== Returns ===

<code>list</code>

== reverse ==

returns a new list with elements in reversed order.

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''list''' (<code>list</code>): list to reverse


=== Returns ===

<code>list</code>

== append ==

appends items to a list and return the extended list. The original list stays unharmed.

'''Allowed number of parameters:''' 1–10000


=== Parameters ===

* '''list''' (<code>list</code>): base list
* '''item...''' (<code>any</code>): items to add ''(variadic)''


=== Returns ===

<code>list</code>

== append_unique ==

appends items to a list but only if they are new. The original list stays unharmed.

'''Allowed number of parameters:''' 1–10000


=== Parameters ===

* '''list''' (<code>list</code>): base list
* '''item...''' (<code>any</code>): items to add ''(variadic)''


=== Returns ===

<code>list</code>

== cons ==

constructs a list from a head and a tail list

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''car''' (<code>any</code>): new head element
* '''cdr''' (<code>list</code>): tail that is appended after car


=== Returns ===

<code>list</code>

== car ==

extracts the head of a list

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''list''' (<code>list</code>): list


=== Returns ===

<code>any</code>

== cdr ==

extracts the tail of a list The tail of a list is a list with all items except the head.

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''list''' (<code>list</code>): list


=== Returns ===

<code>list</code>

== cadr ==

extracts the second element of a list. Equivalent to (car (cdr x)).

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''list''' (<code>list</code>): list


=== Returns ===

<code>any</code>

== zip ==

swaps the dimension of a list of lists. If one parameter is given, it is a list of lists that is flattened. If multiple parameters are given, they are treated as the components that will be zipped into the sub list

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''list''' (<code>any</code>): list of lists of items ''(variadic)''


=== Returns ===

<code>list</code>

== merge ==

flattens a list of lists into a list containing all the subitems. If one parameter is given, it is a list of lists that is flattened. If multiple parameters are given, they are treated as lists that will be merged into one

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''list''' (<code>any</code>): list of lists of items ''(variadic)''


=== Returns ===

<code>list</code>

== merge_unique ==

flattens a list of lists into a list containing all the subitems. Duplicates are filtered out.

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''list''' (<code>list</code>): list of lists of items ''(variadic)''


=== Returns ===

<code>list</code>


== has? ==

checks if a list has a certain item (equal?)

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''haystack''' (<code>list</code>): list to search in
* '''needle''' (<code>any</code>): item to search for


=== Returns ===

<code>bool</code>

== filter ==

returns a list that only contains elements that pass the filter function

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''list''' (<code>list</code>): list that has to be filtered
* '''condition''' (<code>func(item:any) -> bool</code>): filter condition func(item)->bool


=== Returns ===

<code>list</code>

== find ==

returns the first list element that passes the condition function, or nil/default if none matches

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''list''' (<code>list</code>): list to search
* '''condition''' (<code>func(item:any) -> bool</code>): predicate func(any)->bool that is applied until the first match
* '''default''' (<code>any</code>): optional default value if nothing matches ''(optional)''


=== Returns ===

<code>any</code>

== map ==

returns a list that contains the results of a map function that is applied to the list

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''list''' (<code>list</code>): list that has to be mapped
* '''map''' (<code>func(item:any) -> any</code>): map function func(any)->any that is applied to each item


=== Returns ===

<code>list</code>

== parallel_map ==

like map, but applies fn to each element in parallel using a worker pool limited to runtime.NumCPU()

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''list''' (<code>list</code>): list to map over in parallel
* '''fn''' (<code>func(item:any) -> any</code>): function applied to each element


=== Returns ===

<code>list</code>

== parallel_map_mut ==

like parallel_map, but signals the optimizer that fn may have side effects

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''list''' (<code>list</code>): list to map over in parallel
* '''fn''' (<code>func(item:any) -> any</code>): function with side effects applied to each element


=== Returns ===

<code>list</code>


== mapIndex ==

returns a list that contains the results of a map function that is applied to the list

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''list''' (<code>list</code>): list that has to be mapped
* '''map''' (<code>func(index:int, item:any) -> any</code>): map function func(i, any)->any that is applied to each item


=== Returns ===

<code>list</code>

== reduce ==

returns a list that contains the result of a map function

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''list''' (<code>list</code>): list that has to be reduced
* '''reduce''' (<code>func(acc:any, item:any) -> any</code>): reduce function func(any any)->any where the first parameter is the accumulator, the second is a list item
* '''neutral''' (<code>any</code>): (optional) initial value of the accumulator, defaults to nil ''(optional)''


=== Returns ===

<code>any</code>

== produce ==

returns a list that contains produced items - it works like for(state = startstate, condition(state), state = iterator(state)) {yield state}

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''startstate''' (<code>any</code>): start state to begin with
* '''condition''' (<code>func(state:any) -> bool</code>): func that returns true whether the state will be inserted into the result or the loop is stopped
* '''iterator''' (<code>func(state:any) -> any</code>): func that produces the next state


=== Returns ===

<code>list</code>


== produceN ==

returns a list with numbers from 0..n-1, optionally mapped through a function

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''n''' (<code>number</code>): number of elements to produce
* '''fn''' (<code>func(index:int) -> any</code>): (optional) map function applied to each index ''(optional)''


=== Returns ===

<code>list</code>


== parallelN ==

returns a list with numbers from 0..n-1 mapped in parallel through a function

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''n''' (<code>number</code>): number of elements to produce
* '''fn''' (<code>func(index:int) -> any</code>): map function applied to each index in parallel


=== Returns ===

<code>list</code>


== list? ==

checks if a value is a list

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value to check


=== Returns ===

<code>bool</code>


== contains? ==

checks if a value is in a list; uses the equal?? operator

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''list''' (<code>list</code>): list to check
* '''value''' (<code>any</code>): value to check


=== Returns ===

<code>bool</code>

== sql_in ==

tests SQL IN-list membership and returns nil when NULL makes the result UNKNOWN

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''values''' (<code>list</code>): SQL IN-list values
* '''value''' (<code>any</code>): value to find


=== Returns ===

<code>bool</code>

JIT in Scheme

2026-08-27T06:15:13Z

Carli: Created page with "= JIT Compilation = Our scheme dialect allows to JIT-compile SCM code into real machine code. ← Back to Full SCM API documentation == jit == compiles a lambda to optimized native code when this build enables JIT '''Allowed number of parameters:''' 1–1 === Parameters === * '''fn''' (<code>any</code>): the function to compile === Returns === <code>any</code> == jit? == tel..."

= JIT Compilation =

Our scheme dialect allows to JIT-compile SCM code into real machine code.

← Back to [[Full SCM API documentation]]

== jit ==

compiles a lambda to optimized native code when this build enables JIT

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''fn''' (<code>any</code>): the function to compile


=== Returns ===

<code>any</code>


== jit? ==

tells whether a value is a JIT-compiled function descriptor

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value to inspect


=== Returns ===

<code>bool</code>

== jit-warn-if-fallback ==

prints a diagnostic warning when an enabled JIT build kept a procedure interpreted and returns the procedure unchanged

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''procedure''' (<code>any</code>): procedure expected to be a native compilation candidate
* '''label''' (<code>string</code>): optional diagnostic label ''(optional)''


=== Returns ===

<code>any</code>


== jit-enabled? ==

tells whether this binary was built with the patched Go JIT runtime

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>bool</code>

Main Page

2026-08-27T06:14:58Z

Carli: /* Scheme documentation */

= MemCP – A Modern In-Memory Columnar Database =

<div style="padding:2.5rem 2rem; margin:0 0 1.5rem; border-radius:14px; background:linear-gradient(135deg,#3d6208 0%,#64910c 52%,#76b512 100%); color:#fff; text-align:center;">
<div style="font-size:2.6rem; line-height:1.1; font-weight:700; margin-bottom:.7rem;">MemCP</div>
<div style="font-size:1.45rem; line-height:1.35; font-weight:600; margin:0 auto .9rem; max-width:56rem;">A fast, compressed, MySQL-compatible columnar database for modern OLTP and OLAP workloads</div>
<div style="font-size:1.05rem; line-height:1.6; margin:0 auto 1.4rem; max-width:54rem;">MemCP combines persistent in-memory operation, adaptive column compression, parallel query execution and direct application APIs in a lightweight database written in Go.</div>
<div style="margin-bottom:1.5rem;">Beta Open Source MySQL Protocol + HTTP APIs</div>
<div class="plainlinks">[[Install MemCP with Docker|Get started with Docker]] [[Supported SQL|Explore SQL support]] [https://github.com/launix-de/memcp View on GitHub]</div>
</div>

[[File:Memcp-Load.png|center|frameless|1000px|alt=MemCP database dashboard and workload overview]]

<div style="padding:.85rem 1rem; margin:1rem 0 2rem; border-left:5px solid #d99b00; background:#fff7d6; color:#332600;">
'''Development status: Beta.''' MemCP is under active development. Check [[Supported SQL]], [[Current Status and Open Issues]] and the durability requirements of your workload before migrating production data.
</div>

== Choose your path ==

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(220px,1fr)); gap:1rem; margin:1rem 0 2rem;">
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Evaluate MemCP</div>
Understand the workload model, current status, hardware needs and differences from MySQL.

[[What is OLTP and OLAP|OLTP and OLAP]] · [[Comparison: MemCP vs. MySQL|Compare with MySQL]] · [[Hardware Requirements|Hardware requirements]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Build an application</div>
Use the MySQL protocol, SQL over HTTP, RDF or application-specific endpoints inside MemCP.

[[Advanced SQL Tutorial|SQL tutorial]] · [[SQL over REST|SQL over HTTP]] · [[Database Tools compatibility with MemCP|Client tooling]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Operate MemCP</div>
Plan deployment, migration, persistence, storage backends, settings and performance measurement.

[[Deployment]] · [[Persistency and Performance Guarantees|Durability]] · [[Migration from MySQL and PostgreSQL|Migration]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Understand and contribute</div>
Explore the storage engine, optimizer, embedded Scheme runtime and project internals.

[[Query Planner and Physical Lowering|Query planner]] · [[Columnar Storage]] · [[Contributing]]
</div>
</div>

== Quickstart ==

=== Run with Docker ===

<syntaxhighlight lang="bash">
docker run --name memcp \
-e ROOT_PASSWORD='choose-a-password' \
-p 4321:4321 -p 3307:3307 \
carli2/memcp:latest
</syntaxhighlight>

Then open <code>http://localhost:4321</code> and connect a MySQL application to <code>127.0.0.1</code> on port '''3307'''. Add a volume mounted at <code>/data</code> when the container must retain persistent data. See [[Install MemCP with Docker|the complete Docker guide]] before deploying it as a service.

=== Build from source ===

<syntaxhighlight lang="bash">
git clone https://github.com/launix-de/memcp
cd memcp
go mod download
make
./memcp --api-port=4321 --mysql-port=3307 lib/main.scm
</syntaxhighlight>

Connect with MySQL tooling:

<syntaxhighlight lang="bash">
mysql -h 127.0.0.1 -u root -p -P 3307
# Enter the password selected for this data directory.
</syntaxhighlight>

The development default for a fresh data directory is <code>admin</code>. Change it before exposing MemCP to another machine. MemCP can also be supervised with PM2:

<syntaxhighlight lang="bash">
pm2 start ./memcp --name memcp -- \
--no-repl -data ./data --api-port=4321 --mysql-port=3307 lib/main.scm
</syntaxhighlight>

Background deployments must use <code>--no-repl</code> so that closing standard input does not terminate the interactive console and stop the server. See [[Compile MemCP from Source|Build from Source]], [[Deployment]] and [[MemCP Console]].

== Key features ==

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(260px,1fr)); gap:.8rem; margin:1rem 0;">
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''High performance''' NUMA-aware, parallelized query execution is optimized for multicore CPUs, large caches and NVMe SSDs, serving both OLTP and OLAP workloads.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Columnar storage''' Data is stored by column for improved compression, a smaller memory footprint and fast analytical access.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Persistent in-memory operation''' MemCP is designed to keep active data in memory while offering configurable per-table durability and persistence backends.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Built-in APIs''' SQL over HTTP and in-database services can remove an extra middleware hop for suitable applications.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Adaptive compression''' Bit-packing, dictionary encoding and sequence compression can reduce suitable datasets by up to 80% compared with their MySQL/MariaDB representation.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Simple deployment''' Start with Docker, PM2 or the native binary. The compact application has historically had an installation footprint of approximately 10 MB.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Extensible frontends''' The Go storage engine and embedded Scheme environment support SQL, RDF, REST and custom application interfaces.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #9ad32d; background:#f5f8f0; color:#17202a;">'''Cost-based query execution''' Logical decorrelation and join optimization are separated from cost-based physical selection of scans, indexes, RecSets, reusable caches and execution pipelines. [[Query Planner and Physical Lowering|How MemCP plans and lowers queries →]]</div>
</div>

== Why MemCP? ==

Traditional relational databases were designed around spinning disks and comparatively small numbers of CPU cores. MemCP rethinks storage and query execution for modern multicore systems, large caches, fast storage and mixed workloads.

Typical use cases include:

* real-time dashboards and analytics;
* data-heavy SaaS platforms;
* embedded systems with limited resources;
* high-throughput OLTP/OLAP hybrids.

Whether MemCP is a good fit depends on the required SQL compatibility, durability, query mix, data size and operational environment. Review [[Supported SQL]], [[Persistency and Performance Guarantees]] and [[Performance Measurement]] rather than treating benchmark figures as universal guarantees.

== MemCP vs. MySQL ==

{| class="wikitable" style="width:100%;"
! Feature
! MySQL
! MemCP
|-
| Storage model
| Primarily row-based
| Column-based and compressed
|-
| Performance focus
| General-purpose relational workloads
| NUMA-aware, in-memory execution for mixed operational and analytical workloads
|-
| In-memory capability
| Available through selected engines and caching
| Central design goal and default operating model
|-
| REST API integration
| Normally external
| Built in
|-
| Installation footprint
| Common server installations are approximately 150 MB or larger
| The native application has historically been approximately 10 MB
|-
| Open source
| ✅
| ✅
|}

MemCP provides MySQL protocol compatibility but does not claim to implement every MySQL feature. See [[Comparison: MemCP vs. MySQL]] for a detailed comparison and [[Database Tools compatibility with MemCP]] for client compatibility.

== Architecture overview ==

* '''Tables, schemas and columns''': Familiar SQL structures use a compressed columnar physical layout. See [[Databases, Tables and Columns]] and [[Columnar Storage]].
* '''Transaction model''': Delta and main storage support mixed OLTP and OLAP semantics. See [[Transactions and Isolation]] and [[Shards, RecordIDs, Main Storage, Delta Storage]].
* '''Query planning''': Logical optimization is separated from physical execution decisions. See [[Query Planner and Physical Lowering]].
* '''Persistence''': Per-table durability can use filesystem, S3 or Ceph/RADOS storage. See [[Persistency and Performance Guarantees]].
* '''Frontends''': MemCP provides multiple query and application interfaces:
** SQL through the MySQL wire protocol and SQL over HTTP;
** an RDF/graph query frontend;
** custom APIs through in-database web applications.

== Documentation ==

The following navigation remains on the start page so users and search engines can reach every major documentation area directly.

=== Introduction and evaluation ===

* [[What is OLTP and OLAP]]
* [[History of the MemCP project]]
* [[Hardware Requirements]]
* [[Persistency and Performance Guarantees]]
* [[Current Status and Open Issues]]
* [[Comparison: MemCP vs. MySQL]]

=== Getting started ===

* [[Install MemCP with Docker|Install with Docker]]
* [[With Singularity|Install with Singularity/Apptainer]]
* [[Compile MemCP from Source|Build from Source]]
* [[Contributing]]
* [[Introduction to Scheme]]
* [[Full SCM API documentation]]

=== Administration ===

* [[Deployment]]
* [[Migration from MySQL and PostgreSQL]]
* [[Settings]]
* [[Process Hibernation]]
* [[Performance Measurement]]
* [[MemCP Console]]

=== Frontends ===

==== SQL frontend ====

* [[Supported SQL]]
* [[Advanced SQL Tutorial]]
* [[SQL over REST]]
* [[Database Tools compatibility with MemCP|Supported Tooling]]
* [[How SQL Operators are implemented on MemCP]]
* [[Add custom SQL operators to MemCP]]

==== RDF frontend ====

* [[Introduction to RDF]]
* [[Advanced Graph Querying]]
* [[RDF templating and model driven development]]

==== Custom frontends ====

* [[In-Database WebApps|In-Database WebApps and REST Services]]
* [[MemCP for Microservices]]
* [[Websockets in MemCP]]

=== Persistence backends and storage ===

* [[File System]]
* [[S3 Buckets]]
* [[Ceph/Rados]]
* [[Cluster Monitor]]

=== Internals ===

==== How MemCP works ====

* [[Databases, Tables and Columns]]
* [[Shards, RecordIDs, Main Storage, Delta Storage]]
* [[Columnar Storage]]
* [[Transactions and Isolation]]
* [[Query Planner and Physical Lowering]]
* [[Full SCM API documentation]]

==== Scheme documentation ====

* [[SCM Builtins]]
* [[Arithmetic / Logic]]
* [[Strings]]
* [[Streams]]
* [[Lists]]
* [[Associative Lists / Dictionaries]]
* [[Date]]
* [[Vectors]]
* [[Parsers]]
* [[Sync]]
* [[IO]]
* [[Storage]]
* [[JIT in Scheme|JIT]]

==== Optimizations ====

* [[Query Planner and Physical Lowering]]
* [[In-Memory Compression, Columnar Compression Techniques]]
* [[Temporary Columns]]
* [[Data Auto Sharding and Auto Indexing]]
* [[Parallel Computing]]

== Further reading ==

* [https://github.com/launix-de/memcp MemCP on GitHub]
* [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]
* [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]
* [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]

Additional blog posts on design decisions, compression techniques and performance optimization are available on the [https://launix.de/launix/ Launix blog].

== Community ==

MemCP is an open-source project maintained by developers for developers. Contributions are welcome in the form of bug reports, feature requests and pull requests.

See [[Contributing]] and the [https://github.com/launix-de/memcp GitHub repository].

Full SCM API documentation

2026-08-27T06:13:34Z

Carli:

= Documentation =

* [[SCM Builtins]]
* [[Arithmetic / Logic]]
* [[Strings]]
* [[Streams]]
* [[Lists]]
* [[Associative Lists / Dictionaries]]
* [[Date]]
* [[Vectors]]
* [[Parsers]]
* [[Sync]]
* [[IO]]
* [[Storage]]
* [[JIT in Scheme|JIT]]

IO

2026-08-27T06:12:23Z

Carli:

= IO =

The '''IO''' module provides functions for input and output operations, environment handling, file streaming, server control, and argument parsing in SCM.

← Back to [[Full SCM API documentation]]

== print ==

Prints values to stdout (only in IO environment)

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>any</code>): values to print ''(variadic)''


=== Returns ===

<code>bool</code>

== env ==

returns the content of a environment variable

'''Allowed number of parameters:''' 1–2


=== Parameters ===

* '''var''' (<code>string</code>): envvar
* '''default''' (<code>string</code>): default if the env is not found ''(optional)''


=== Returns ===

<code>string</code>

== help ==

Lists all functions or returns help for a specific function as a string

'''Allowed number of parameters:''' 0–1


=== Parameters ===

* '''topic''' (<code>string</code>): function to get help about ''(optional)''


=== Returns ===

<code>string</code>

== import ==

Imports a file .scm file into current namespace

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''filename''' (<code>string</code>): filename relative to folder of source file or absolute path


=== Returns ===

<code>any</code>

== load ==

Loads a file or stream and returns the string or iterates line-wise

'''Allowed number of parameters:''' 1–3


=== Parameters ===

* '''filenameOrStream''' (<code>string|stream</code>): filename relative to folder of source file, absolute path, or stream to read from
* '''linehandler''' (<code>func(line:string) -> any</code>): handler that reads each line; each line may end with delimiter ''(optional)''
* '''delimiter''' (<code>string</code>): delimiter to extract; if no delimiter is given, the file is read as whole and returned or passed to linehandler ''(optional)''


=== Returns ===

<code>string|bool</code>

== stream ==

Opens a file readonly as stream

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''filename''' (<code>string</code>): filename relative to folder of source file or absolute path


=== Returns ===

<code>stream</code>

== watch ==

Loads a file and calls the callback. Whenever the file changes on disk, the file is load again.

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''filename''' (<code>string</code>): filename relative to folder of source file or absolute path
* '''updatehandler''' (<code>func(content:string) -> any</code>): handler that receives the file content func(content)


=== Returns ===

<code>bool</code>

== serve ==

Opens a HTTP server at a given port

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''port''' (<code>number</code>): port number for HTTP server
* '''handler''' (<code>func(req:any, res:any) -> any</code>): handler: lambda(req res) that handles the http request


=== Returns ===

<code>bool</code>


== serveStatic ==

creates a static handler for use as a callback in (serve) - returns a handler lambda(req res)

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''directory''' (<code>string</code>): folder with the files to serve


=== Returns ===

<code>func(req:any, res:any) -> any</code>

== mysql ==

Imports a file .scm file into current namespace

'''Allowed number of parameters:''' 4–4


=== Parameters ===

* '''port''' (<code>number</code>): port number for MySQL server
* '''getPassword''' (<code>func(username:string) -> string|nil</code>): lambda(username string) string|nil has to return the password for a user or nil to deny login
* '''schemacallback''' (<code>func(username:string, schema:string) -> bool</code>): lambda(username schema) bool handler check whether user is allowed to schema - you should check access rights here
* '''handler''' (<code>func(schema:string, sql:string, resultrow:func, session:func) -> any</code>): lambda(schema sql resultrow session) handler to process sql query in schema. resultrow is a lambda(list)


=== Returns ===

<code>bool</code>

== mysql_socket ==

Listen on a Unix domain socket for MySQL protocol

'''Allowed number of parameters:''' 4–4


=== Parameters ===

* '''socketpath''' (<code>string</code>): path to the Unix domain socket
* '''getPassword''' (<code>func(username:string) -> string|nil</code>): lambda(username string) string|nil has to return the password for a user or nil to deny login
* '''schemacallback''' (<code>func(username:string, schema:string) -> bool</code>): lambda(username schema) bool handler check whether user is allowed to schema - you should check access rights here
* '''handler''' (<code>func(schema:string, sql:string, resultrow:func, session:func) -> any</code>): lambda(schema sql resultrow session) handler to process sql query in schema. resultrow is a lambda(list)


=== Returns ===

<code>bool</code>

== password ==

Hashes a password with sha1 (for mysql user authentication)

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''password''' (<code>string</code>): plain text password to hash


=== Returns ===

<code>string</code>

== shutdown ==

Initiates a graceful shutdown of memcp after a short delay

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>bool</code>

== crash ==

Hard process exit with no cleanup (kill -9 equivalent) for crash testing. SCM only, not exposed to SQL.

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>bool</code>

== path ==

Joins path segments using the OS path separator and cleans the result

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''segments''' (<code>string</code>): path segments to join ''(variadic)''


=== Returns ===

<code>string</code>

== args ==

Returns command line arguments

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>list</code>

== arg ==

Gets a command line argument value

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''longname''' (<code>string</code>): long argument name (without --)
* '''shortname''' (<code>string|any</code>): short argument name (without -) or default value if only 2 args
* '''default''' (<code>any</code>): default value if argument not found ''(optional)''


=== Returns ===

<code>any</code>

Date

2026-08-27T06:11:21Z

Carli:

= Date =

The '''Date''' module provides date and time functionality for the SCM programming language. This module includes:

* '''Current time''': Functions to get the current Unix timestamp (now)
* '''Date parsing''': Functions to parse date strings into Unix timestamps (parse_date)

These functions provide essential tools for working with dates and timestamps in SCM programs, enabling time-based operations and date string processing.

← Back to [[Full SCM API documentation]]

== sql_temporal_output ==

formats a temporal SQL result according to its compiler-tracked declared type

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''value''' (<code>any</code>): type-flexible temporal value
* '''sql_type''' (<code>string</code>): declared SQL temporal type


=== Returns ===

<code>any</code>

== now ==

returns the current date/time

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>date</code>

== nanotime ==

returns a monotonic nanosecond timestamp for benchmarking (not wall-clock)

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>int</code>

== current_date ==

returns the current date (midnight in session timezone)

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>date</code>

== parse_date ==

parses a date from a string

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>string</code>): values to parse


=== Returns ===

<code>date</code>

== format_date ==

formats a unix timestamp, date, or datetime string into a date string

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''timestamp''' (<code>any</code>): unix timestamp, date, or datetime string
* '''format''' (<code>string</code>): MySQL-style format string (e.g. %Y-%m-%d %H:%i:%s)


=== Returns ===

<code>string</code>

== extract_date ==

extracts a date field (YEAR, MONTH, DAY, HOUR, MINUTE, SECOND, QUARTER, WEEK, DAYOFWEEK, WEEKDAY) from a date value

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''value''' (<code>any</code>): date value
* '''field''' (<code>string</code>): field name: YEAR, MONTH, DAY, HOUR, MINUTE, SECOND, QUARTER, WEEK, DAYOFWEEK, WEEKDAY


=== Returns ===

<code>int</code>

== date_add ==

adds an interval to a date value

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''value''' (<code>any</code>): date value
* '''amount''' (<code>int</code>): interval amount
* '''unit''' (<code>string</code>): interval unit: DAY, WEEK, MONTH, YEAR, HOUR, MINUTE, SECOND


=== Returns ===

<code>date</code>

== date_sub ==

subtracts an interval from a date value

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''value''' (<code>any</code>): date value
* '''amount''' (<code>int</code>): interval amount
* '''unit''' (<code>string</code>): interval unit: DAY, WEEK, MONTH, YEAR, HOUR, MINUTE, SECOND


=== Returns ===

<code>date</code>

== date_trunc_day ==

truncates a datetime to date (midnight UTC)

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): date/datetime value


=== Returns ===

<code>date</code>

== timestampdiff ==

returns the difference between two datetimes in the given unit

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''unit''' (<code>string</code>): SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, YEAR
* '''dt1''' (<code>any</code>): first datetime
* '''dt2''' (<code>any</code>): second datetime


=== Returns ===

<code>int</code>

== datediff ==

returns number of days between two dates (date1 - date2)

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''date1''' (<code>any</code>): first date
* '''date2''' (<code>any</code>): second date


=== Returns ===

<code>int</code>

== str_to_date ==

parses a string with MySQL format specifiers to a date

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''value''' (<code>string</code>): date string
* '''format''' (<code>string</code>): MySQL format string (e.g. %Y-%m-%d)


=== Returns ===

<code>date</code>

Associative Lists / Dictionaries

2026-08-27T06:10:09Z

Carli:

= Associative Lists / Dictionaries =

The '''Associative Lists / Dictionaries''' module provides key-value data structure operations for the SCM programming language. This module includes:

* '''Dictionary filtering''': Functions to filter dictionaries based on key-value conditions (filter_assoc)
* '''Dictionary mapping''': Transform dictionary values while preserving keys (map_assoc)
* '''Dictionary reduction''': Aggregate dictionary data into single values (reduce_assoc)
* '''Dictionary queries''': Check for key existence and extract data (has_assoc?, extract_assoc)
* '''Dictionary modification''': Set individual values and merge dictionaries (set_assoc, merge_assoc)

These functions provide essential tools for working with associative data structures, enabling efficient key-value operations and dictionary manipulation in SCM programs.

← Back to [[Full SCM API documentation]]

== filter_assoc ==

returns a filtered dictionary according to a filter function

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''dict''' (<code>list</code>): dictionary that has to be filtered
* '''condition''' (<code>func(key:string, value:any) -> bool</code>): filter function func(string any)->bool where the first parameter is the key, the second is the value


=== Returns ===

<code>list</code>

== find_assoc ==

returns the first key/value pair that passes the condition function, or nil/default if none matches

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''dict''' (<code>list</code>): dictionary to search
* '''condition''' (<code>func(key:string, value:any) -> bool</code>): predicate func(string any)->bool that is applied until the first match
* '''default''' (<code>any</code>): optional default value if nothing matches ''(optional)''


=== Returns ===

<code>any</code>

== map_assoc ==

returns a mapped dictionary according to a map function Keys will stay the same but values are mapped.

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''dict''' (<code>list</code>): dictionary that has to be mapped
* '''map''' (<code>func(key:string, value:any) -> any</code>): map function func(string any)->any where the first parameter is the key, the second is the value. It must return the new value.


=== Returns ===

<code>list</code>

== reduce_assoc ==

reduces a dictionary according to a reduce function

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''dict''' (<code>list</code>): dictionary that has to be reduced
* '''reduce''' (<code>func(acc:any, key:any, value:any) -> any</code>): reduce function func(any string any)->any where the first parameter is the accumulator, second is key, third is value. It must return the new accumulator.
* '''neutral''' (<code>any</code>): initial value for the accumulator


=== Returns ===

<code>any</code>

== make_structural_index ==

Builds an immutable structural-expression index. It eagerly hashes every key and every node under roots, then returns a parallel-safe lookup function that maps an equal expression to its zero-based key position or nil.

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''keys''' (<code>list</code>): immutable structural expressions to index
* '''roots''' (<code>list</code>): immutable expression roots whose descendant hashes are precomputed


=== Returns ===

<code>func(expression:any) -> int|nil</code>

== make_structural_catalog ==

Creates an atomic compile-local structural catalog. Look up with (catalog key), insert with (catalog key value), or freeze with (catalog) for parallel-safe read-only lookup.

'''Allowed number of parameters:''' 0–1


=== Parameters ===

* '''mode''' (<code>bool|symbol</code>): true forces collisions for tests; ast selects type-stable compiler equality ''(optional)''


=== Returns ===

<code>func</code>


== has_assoc? ==

checks if a dictionary has a key present

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''dict''' (<code>list</code>): dictionary that has to be checked
* '''key''' (<code>string</code>): key to test


=== Returns ===

<code>bool</code>

== get_assoc ==

gets a value from a dictionary by key, returns nil if not found

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''dict''' (<code>list</code>): dictionary to look up
* '''key''' (<code>any</code>): key to look up
* '''default''' (<code>any</code>): optional default value if key not found ''(optional)''


=== Returns ===

<code>any</code>

== get_assoc_pairlist ==

gets a value from a list of key/value rows without flattening the rows

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''rows''' (<code>list</code>): list whose rows contain a key followed by one or more values
* '''key''' (<code>any</code>): key compared with the first item of each row
* '''default''' (<code>any</code>): value returned when no row contains the key


=== Returns ===

<code>any</code>

== extract_assoc ==

applies a function (key value) on the dictionary and returns the results as a flat list

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''dict''' (<code>list</code>): dictionary that has to be checked
* '''map''' (<code>func(key:string, value:any) -> any</code>): func(key, value)->any that extracts one element per key-value pair


=== Returns ===

<code>list</code>

== set_assoc ==

returns a new dictionary where a single value has been changed. The original dictionary is not modified.

'''Allowed number of parameters:''' 3–4


=== Parameters ===

* '''dict''' (<code>list</code>): input dictionary
* '''key''' (<code>string</code>): key that has to be set
* '''value''' (<code>any</code>): new value to set
* '''merge''' (<code>func(old:any, new:any) -> any</code>): (optional) func(any any)->any that is called when a value is overwritten. The first parameter is the old value, the second is the new value. It must return the merged value that shall be physically stored in the new dictionary. ''(optional)''


=== Returns ===

<code>list</code>

== merge_assoc ==

returns a dictionary where all keys from dict1 and all keys from dict2 are present. If a key is present in both inputs, the second one will be dominant so the first value will be overwritten unless you provide a merge function

'''Allowed number of parameters:''' 2–3


=== Parameters ===

* '''dict1''' (<code>list</code>): first input dictionary that has to be changed. You must not use this value again.
* '''dict2''' (<code>list</code>): input dictionary that contains the new values that have to be added
* '''merge''' (<code>func(old:any, new:any) -> any</code>): (optional) func(any any)->any that is called when a value is overwritten. The first parameter is the old value, the second is the new value from dict2. It must return the merged value that shall be pysically stored in the new dictionary. ''(optional)''


=== Returns ===

<code>list</code>

== sort ==

returns a sorted copy of a list using a comparator (lambda (a b) truthy/falsy)

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''list''' (<code>list</code>):
* '''comparator''' (<code>func(any, any) -> bool</code>):


=== Returns ===

<code>list</code>

== sort_mut ==

sorts a list in-place using a comparator (lambda (a b) truthy/falsy)

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''list''' (<code>list</code>):
* '''comparator''' (<code>func(any, any) -> bool</code>):


=== Returns ===

<code>list</code>

== mapkey_assoc ==

returns a mapped dictionary according to a map function Values stay the same but keys are mapped.

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''dict''' (<code>list</code>): dictionary whose keys have to be mapped
* '''map''' (<code>func(key:string, value:any) -> any</code>): map function func(key, value)->key where the first parameter is the old key, the second is the value. It must return the new key.


=== Returns ===

<code>list</code>

Arithmetic / Logic

2026-08-27T06:09:39Z

Carli:

= Arithmetic / Logic =

The '''Arithmetic / Logic''' module provides essential mathematical operations and logical comparisons for the SCM programming language. This module includes:

* '''Type checking''': Functions to verify data types (int?, number?, nil?)
* '''Basic arithmetic''': Addition (+), subtraction (-), multiplication (*), and division (/)
* '''Comparison operations''': All standard comparison operators (>, <, >=, <=, equal?)
* '''Logical operations''': Boolean negation (!, not) and advanced equality checks
* '''Mathematical functions''': Min/max operations and rounding functions (floor, ceil, round)

These functions form the mathematical foundation for calculations and logical decision-making in SCM programs.

← Back to [[Full SCM API documentation]]

== int? ==

tells if the value is a integer

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value


=== Returns ===

<code>bool</code>


== number? ==

tells if the value is a number

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value


=== Returns ===

<code>bool</code>


== symbol? ==

tells if the value is a symbol

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value


=== Returns ===

<code>bool</code>

== + ==

adds two or more numbers

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>number</code>): values to add ''(variadic)''


=== Returns ===

<code>number</code>

== sql_sum_reduce ==

adds two SQL SUM values while treating NULL as the empty aggregate identity

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''left''' (<code>number|nil</code>): partial sum
* '''right''' (<code>number|nil</code>): next value


=== Returns ===

<code>number|nil</code>

== - ==

subtracts two or more numbers from the first one

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>number</code>): values ''(variadic)''


=== Returns ===

<code>number</code>

== sql_add_numeric_literals ==

adds two SQL numeric literals using their exact decimal spellings

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>number</code>): left literal
* '''b''' (<code>number</code>): right literal


=== Returns ===

<code>number</code>

== sql_sub_numeric_literals ==

subtracts two SQL numeric literals using their exact decimal spellings

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>number</code>): left literal
* '''b''' (<code>number</code>): right literal


=== Returns ===

<code>number</code>


== * ==

multiplies two or more numbers

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>number</code>): values ''(variadic)''


=== Returns ===

<code>number</code>


== / ==

divides two or more numbers from the first one

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>number</code>): values ''(variadic)''


=== Returns ===

<code>number</code>

== mod ==

returns the remainder of integer division (modulo)

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>number</code>): dividend
* '''b''' (<code>number</code>): divisor


=== Returns ===

<code>number</code>


== <= ==

compares two numbers or strings; returns nil if either value is nil

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>any</code>): first value
* '''b''' (<code>any</code>): second value


=== Returns ===

<code>bool</code>


== < ==

compares two numbers or strings; returns nil if either value is nil

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>any</code>): first value
* '''b''' (<code>any</code>): second value


=== Returns ===

<code>bool</code>


== > ==

compares two numbers or strings; returns nil if either value is nil

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>any</code>): first value
* '''b''' (<code>any</code>): second value


=== Returns ===

<code>bool</code>


== >= ==

compares two numbers or strings; returns nil if either value is nil

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>any</code>): first value
* '''b''' (<code>any</code>): second value


=== Returns ===

<code>bool</code>


== equal? ==

compares two values of the same type, (equal? nil nil) is true

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>any</code>): first value
* '''b''' (<code>any</code>): second value


=== Returns ===

<code>bool</code>


== equal?? ==

performs a SQL compliant sloppy equality check on primitive values (number, int, string, bool. nil), strings are compared case insensitive, (equal? nil nil) is nil

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>any</code>): first value
* '''b''' (<code>any</code>): second value


=== Returns ===

<code>bool</code>

== sql_not ==

negates a SQL predicate while preserving nil as UNKNOWN

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): SQL predicate value


=== Returns ===

<code>bool|nil</code>

== equal_collate ==

performs SQL equality with a specified collation (e.g. *_ci case-insensitive, *_bin case-sensitive); returns nil if either arg is nil

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''a''' (<code>any</code>): left side
* '''b''' (<code>any</code>): right side
* '''collation''' (<code>string</code>): collation name


=== Returns ===

<code>bool</code>

== notequal_collate ==

performs SQL inequality with a specified collation; returns nil if either arg is nil

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''a''' (<code>any</code>): left side
* '''b''' (<code>any</code>): right side
* '''collation''' (<code>string</code>): collation name


=== Returns ===

<code>bool</code>


== ! ==

negates the boolean value

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>bool</code>): value


=== Returns ===

<code>bool</code>

== not ==

negates the boolean value

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>bool</code>): value


=== Returns ===

<code>bool</code>


== nil? ==

returns true if value is nil

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value


=== Returns ===

<code>bool</code>

== min ==

returns the smallest value

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>number|string</code>): value ''(variadic)''


=== Returns ===

<code>number|string</code>

== max ==

returns the highest value

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>number|string</code>): value ''(variadic)''


=== Returns ===

<code>number|string</code>

== floor ==

rounds the number down

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>number</code>): value


=== Returns ===

<code>number</code>

== ceil ==

rounds the number up

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>number</code>): value


=== Returns ===

<code>number</code>

== round ==

rounds the number

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>number</code>): value


=== Returns ===

<code>number</code>

== sql_decimal_output ==

normalizes a DECIMAL-derived result to its declared output scale

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''value''' (<code>number|nil</code>): DECIMAL-derived value
* '''scale''' (<code>int</code>): declared decimal scale


=== Returns ===

<code>number|nil</code>

== sql_abs ==

SQL ABS(): returns absolute value, NULL-safe

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>number</code>): value


=== Returns ===

<code>number</code>

== sqrt ==

returns the square root of a number

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>number</code>): value


=== Returns ===

<code>number</code>

== sql_rand ==

SQL RAND(): returns a random float in [0,1)

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>number</code>

== parameterize_sql_select_literals ==

replaces safe literals in a top-level MySQL SELECT and returns normalized SQL, positional runtime bindings, and its stable shape hash

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''query''' (<code>string</code>):


=== Returns ===

<code>list</code>

Arithmetic / Logic

2026-08-27T06:08:03Z

Carli:

= Arithmetic / Logic =


== int? ==

tells if the value is a integer

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value


=== Returns ===

<code>bool</code>


== number? ==

tells if the value is a number

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value


=== Returns ===

<code>bool</code>


== symbol? ==

tells if the value is a symbol

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value


=== Returns ===

<code>bool</code>

== + ==

adds two or more numbers

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>number</code>): values to add ''(variadic)''


=== Returns ===

<code>number</code>

== sql_sum_reduce ==

adds two SQL SUM values while treating NULL as the empty aggregate identity

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''left''' (<code>number|nil</code>): partial sum
* '''right''' (<code>number|nil</code>): next value


=== Returns ===

<code>number|nil</code>

== - ==

subtracts two or more numbers from the first one

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>number</code>): values ''(variadic)''


=== Returns ===

<code>number</code>

== sql_add_numeric_literals ==

adds two SQL numeric literals using their exact decimal spellings

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>number</code>): left literal
* '''b''' (<code>number</code>): right literal


=== Returns ===

<code>number</code>

== sql_sub_numeric_literals ==

subtracts two SQL numeric literals using their exact decimal spellings

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>number</code>): left literal
* '''b''' (<code>number</code>): right literal


=== Returns ===

<code>number</code>


== * ==

multiplies two or more numbers

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>number</code>): values ''(variadic)''


=== Returns ===

<code>number</code>


== / ==

divides two or more numbers from the first one

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>number</code>): values ''(variadic)''


=== Returns ===

<code>number</code>

== mod ==

returns the remainder of integer division (modulo)

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>number</code>): dividend
* '''b''' (<code>number</code>): divisor


=== Returns ===

<code>number</code>


== <= ==

compares two numbers or strings; returns nil if either value is nil

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>any</code>): first value
* '''b''' (<code>any</code>): second value


=== Returns ===

<code>bool</code>


== < ==

compares two numbers or strings; returns nil if either value is nil

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>any</code>): first value
* '''b''' (<code>any</code>): second value


=== Returns ===

<code>bool</code>


== > ==

compares two numbers or strings; returns nil if either value is nil

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>any</code>): first value
* '''b''' (<code>any</code>): second value


=== Returns ===

<code>bool</code>


== >= ==

compares two numbers or strings; returns nil if either value is nil

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>any</code>): first value
* '''b''' (<code>any</code>): second value


=== Returns ===

<code>bool</code>


== equal? ==

compares two values of the same type, (equal? nil nil) is true

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>any</code>): first value
* '''b''' (<code>any</code>): second value


=== Returns ===

<code>bool</code>


== equal?? ==

performs a SQL compliant sloppy equality check on primitive values (number, int, string, bool. nil), strings are compared case insensitive, (equal? nil nil) is nil

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''a''' (<code>any</code>): first value
* '''b''' (<code>any</code>): second value


=== Returns ===

<code>bool</code>

== sql_not ==

negates a SQL predicate while preserving nil as UNKNOWN

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): SQL predicate value


=== Returns ===

<code>bool|nil</code>

== equal_collate ==

performs SQL equality with a specified collation (e.g. *_ci case-insensitive, *_bin case-sensitive); returns nil if either arg is nil

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''a''' (<code>any</code>): left side
* '''b''' (<code>any</code>): right side
* '''collation''' (<code>string</code>): collation name


=== Returns ===

<code>bool</code>

== notequal_collate ==

performs SQL inequality with a specified collation; returns nil if either arg is nil

'''Allowed number of parameters:''' 3–3


=== Parameters ===

* '''a''' (<code>any</code>): left side
* '''b''' (<code>any</code>): right side
* '''collation''' (<code>string</code>): collation name


=== Returns ===

<code>bool</code>


== ! ==

negates the boolean value

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>bool</code>): value


=== Returns ===

<code>bool</code>

== not ==

negates the boolean value

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>bool</code>): value


=== Returns ===

<code>bool</code>


== nil? ==

returns true if value is nil

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>any</code>): value


=== Returns ===

<code>bool</code>

== min ==

returns the smallest value

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>number|string</code>): value ''(variadic)''


=== Returns ===

<code>number|string</code>

== max ==

returns the highest value

'''Allowed number of parameters:''' 0–10000


=== Parameters ===

* '''value...''' (<code>number|string</code>): value ''(variadic)''


=== Returns ===

<code>number|string</code>

== floor ==

rounds the number down

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>number</code>): value


=== Returns ===

<code>number</code>

== ceil ==

rounds the number up

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>number</code>): value


=== Returns ===

<code>number</code>

== round ==

rounds the number

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>number</code>): value


=== Returns ===

<code>number</code>

== sql_decimal_output ==

normalizes a DECIMAL-derived result to its declared output scale

'''Allowed number of parameters:''' 2–2


=== Parameters ===

* '''value''' (<code>number|nil</code>): DECIMAL-derived value
* '''scale''' (<code>int</code>): declared decimal scale


=== Returns ===

<code>number|nil</code>

== sql_abs ==

SQL ABS(): returns absolute value, NULL-safe

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>number</code>): value


=== Returns ===

<code>number</code>

== sqrt ==

returns the square root of a number

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''value''' (<code>number</code>): value


=== Returns ===

<code>number</code>

== sql_rand ==

SQL RAND(): returns a random float in [0,1)

'''Allowed number of parameters:''' 0–0


=== Parameters ===

''This function has no parameters.''


=== Returns ===

<code>number</code>

== parameterize_sql_select_literals ==

replaces safe literals in a top-level MySQL SELECT and returns normalized SQL, positional runtime bindings, and its stable shape hash

'''Allowed number of parameters:''' 1–1


=== Parameters ===

* '''query''' (<code>string</code>):


=== Returns ===

<code>list</code>

Supported SQL

2026-08-26T07:31:35Z

Carli:

== Statements ==
The following SQL statements are supported:

* SELECT FROM GROUP BY HAVING ORDER BY LIMIT OFFSET
* UPDATE SET WHERE
* DELETE FROM WHERE
* INSERT INTO VALUES ON DUPLICATE KEY UPDATE / INSERT ... ON CONFLICT (PostgreSQL)
* INSERT IGNORE
* CREATE TABLE
* ALTER TABLE
* CREATE DATABASE
* CREATE USER
* ALTER USER
* SHOW DATABASES
* SHOW TABLES
* SHOW TABLE STATUS
* SHOW VARIABLES
* SET NAMES (no function)
* DROP DATABASE
* DROP TABLE
* SET SESSION
* LOCK TABLES (no function)
* UNLOCK TABLES (no function)

== Expressions ==
The following functions are supported:
* CASE WHEN THEN ELSE END
* DATABASE
* PASSWORD (hashes a password for user authentication)
* FLOOR
* CEIL
* CEILING
* ROUND
* UPPER
* LOWER
* CAST AS UNSIGNED
* CAST AS CHAR CHARACTER SET utf8
* CONCAT
* COALESCE
* UNIX_TIMESTAMP
* LIKE
* ILIKE

== Aggregates ==
The following aggregates are available:

* COUNT
* SUM
* AVG
* MIN
* MAX

== Modifiers ==

* COLLATE

== JSON ==
JSON support is documented separately in the [[JSON|MemCP MySQL- and Postgres-compatible JSON SQL]]

JSON

2026-08-26T07:22:04Z

Carli: Created page with "= JSON and SQL/JSON in MemCP = '''MemCP provides native JSON and SQL/JSON support on its high-performance columnar SQL engine.''' JSON documents can be filtered, joined, grouped, sorted, modified and assembled into deeply nested application responses without leaving SQL. The same JSON engine is exposed through both MemCP SQL dialects: applications get MySQL/MariaDB-compatible JSON functions such as <code>JSON_EXTRACT</code>, <code>JSON_VALUE</code>, <code>JSON_OBJECT</..."

= JSON and SQL/JSON in MemCP =

'''MemCP provides native JSON and SQL/JSON support on its high-performance columnar SQL engine.''' JSON documents can be filtered, joined, grouped, sorted, modified and assembled into deeply nested application responses without leaving SQL.

The same JSON engine is exposed through both MemCP SQL dialects: applications get MySQL/MariaDB-compatible JSON functions such as <code>JSON_EXTRACT</code>, <code>JSON_VALUE</code>, <code>JSON_OBJECT</code>, <code>JSON_ARRAYAGG</code> and <code>JSON_TABLE</code>, as well as PostgreSQL-compatible <code>json</code>/<code>jsonb</code> operators, constructors, aggregates and JSONPath functions. This makes it possible to migrate JSON-heavy SQL queries or run MemCP beside an existing MySQL or PostgreSQL application. See [[Migration from MySQL and PostgreSQL]] for connection options and [[Supported SQL]] for the wider SQL feature set.

JSON columns validate incoming documents and convert them to MemCP's native tagged BSON value. Objects and arrays therefore remain structured while they are filtered, joined, sorted or aggregated; they are serialized to JSON only at the SQL/API boundary.

JSON functions also accept ordinary character strings containing JSON. Such strings are parsed when the function needs a JSON value. Use a <code>JSON</code> or <code>JSONB</code> column for repeatedly queried documents so that parsing happens on write instead of on every read.

== MemCP compared with MySQL JSON, MariaDB JSON and PostgreSQL JSONB ==

{| class="wikitable"
! SQL dialect
! Familiar JSON syntax in MemCP
! Internal representation in MemCP
! Typical use
|-
| MySQL
| <code>JSON_EXTRACT</code>, <code>JSON_VALUE</code>, <code>-></code>, <code>->></code>, <code>JSON_TABLE</code>
| Native immutable BSON
| Existing MySQL applications and document columns
|-
| MariaDB
| MySQL-style JSON functions and JSONPath
| Native immutable BSON instead of LONGTEXT
| MariaDB-compatible SQL and mixed relational/document workloads
|-
| PostgreSQL
| <code>json</code>/<code>jsonb</code> casts, operators, JSONPath and set-returning functions
| One native BSON representation for both <code>json</code> and <code>jsonb</code>
| PostgreSQL JSONB queries, containment and relational projection
|}

Unlike a database that reparses a text document for every JSON expression, a declared MemCP <code>JSON</code> or <code>JSONB</code> column is converted when it is written. Frequently used paths can participate in deterministic computed expressions and computed indexes.

== Quick start ==

<pre>
CREATE TABLE events (
id INT PRIMARY KEY,
category VARCHAR(40),
payload JSON,
created_at DATETIME
);

INSERT INTO events VALUES
(1, 'customer', '{"customer":{"name":"Ada","rank":2},"tags":["sql","go"]}', NOW());

SELECT
id,
JSON_VALUE(payload, '$.customer.name') AS customer,
JSON_EXTRACT(payload, '$.customer.rank') + 1 AS next_rank
FROM events
WHERE JSON_CONTAINS(payload, '"sql"', '$.tags')
ORDER BY JSON_VALUE(payload, '$.customer.rank' RETURNING SIGNED);
</pre>

JSON scalars participate in normal SQL coercion. A numeric result of <code>JSON_EXTRACT</code>, for example, can be added, compared and sorted numerically. <code>NULL</code> in SQL and the JSON literal <code>null</code> remain distinct where the selected dialect distinguishes them.

== Common MySQL JSON and SQL JSON queries ==

=== Extract a value from JSON in SQL ===

<pre>
SELECT JSON_EXTRACT(payload, '$.customer.rank') AS rank
FROM events;

SELECT payload->>'$.customer.name' AS customer_name
FROM events;
</pre>

=== Filter SQL rows by a JSON attribute ===

<pre>
SELECT id
FROM events
WHERE JSON_VALUE(payload, '$.customer.name') = 'Ada'
AND JSON_CONTAINS(payload, '"sql"', '$.tags');
</pre>

=== Order SQL results by a JSON value ===

<pre>
SELECT id, payload
FROM events
ORDER BY JSON_VALUE(payload, '$.customer.rank' RETURNING SIGNED), id;
</pre>

The numeric <code>RETURNING</code> type prevents lexical ordering such as <code>1, 10, 2</code>. Eligible deterministic path expressions can be backed by a computed index.

=== Aggregate SQL rows into a JSON array ===

<pre>
SELECT category, JSON_ARRAYAGG(
JSON_OBJECT('id', id, 'payload', payload)
) AS documents
FROM events
GROUP BY category;
</pre>

=== Convert a JSON array into SQL rows ===

<pre>
SELECT item.ord, item.name
FROM JSON_TABLE(
'[{"name":"Ada"},{"name":"Bob"}]',
'$[*]' COLUMNS (
ord FOR ORDINALITY,
name TEXT PATH '$.name'
)
) AS item;
</pre>

== MySQL and MariaDB syntax ==

MemCP implements the following MySQL/MariaDB-style JSON surface:

{| class="wikitable"
! Area
! Functions and operators
|-
| Construction and aggregation
| <code>JSON_ARRAY</code>, <code>JSON_OBJECT</code>, <code>JSON_ARRAYAGG</code>, <code>JSON_OBJECTAGG</code>
|-
| Inspection and extraction
| <code>JSON_EXTRACT</code>, <code>JSON_VALUE</code>, <code>JSON_KEYS</code>, <code>JSON_LENGTH</code>, <code>JSON_DEPTH</code>, <code>JSON_TYPE</code>, <code>JSON_VALID</code>, <code>-></code>, <code>->></code>
|-
| Search and comparison
| <code>JSON_CONTAINS</code>, <code>JSON_CONTAINS_PATH</code>, <code>JSON_OVERLAPS</code>, <code>JSON_SEARCH</code>, <code>MEMBER OF</code>
|-
| Modification
| <code>JSON_SET</code>, <code>JSON_INSERT</code>, <code>JSON_REPLACE</code>, <code>JSON_REMOVE</code>, <code>JSON_ARRAY_APPEND</code>, <code>JSON_ARRAY_INSERT</code>, <code>JSON_MERGE_PATCH</code>, <code>JSON_MERGE_PRESERVE</code>, <code>JSON_MERGE</code>
|-
| Conversion and formatting
| <code>JSON_QUOTE</code>, <code>JSON_UNQUOTE</code>, <code>JSON_PRETTY</code>, <code>JSON_STORAGE_SIZE</code>, <code>JSON_STORAGE_FREE</code>
|-
| Validation
| <code>JSON_SCHEMA_VALID</code>, <code>JSON_SCHEMA_VALIDATION_REPORT</code>
|-
| Relational projection
| <code>JSON_TABLE</code>, including path columns and <code>FOR ORDINALITY</code>
|}

Paths start at <code>$</code>. Object members, zero-based array indexes, wildcards, recursive descent and <code>last</code>/<code>last-N</code> array indexes are supported.

<pre>
SELECT JSON_EXTRACT(
'{"orders":[{"total":10},{"total":25}]}',
'$.orders[last].total'
); -- 25

SELECT *
FROM JSON_TABLE(
'[{"name":"Ada"},{"name":"Bob"}]',
'$[*]' COLUMNS (
ord FOR ORDINALITY,
name TEXT PATH '$.name'
)
) AS people;
</pre>

The upstream references are MariaDB's [https://mariadb.com/docs/server/reference/sql-functions/special-functions/json-functions JSON functions index] and [https://mariadb.com/docs/server/reference/sql-functions/special-functions/json-functions/jsonpath-expressions JSONPath reference]. MariaDB adds functions over time; the table above is the supported MemCP surface, not a claim that every function in every MariaDB release is available.

== PostgreSQL syntax ==

The PostgreSQL-compatible parser accepts both <code>json</code> and <code>jsonb</code> casts and column declarations. Internally both use the same native BSON representation.

{| class="wikitable"
! Area
! Functions and operators
|-
| Extraction
| <code>-></code>, <code>->></code>, <code>#></code>, <code>#>></code>, <code>json_extract_path</code>, <code>jsonb_extract_path</code>, and their <code>_text</code> variants
|-
| jsonb operators
| <code>@></code>, <code><@</code>, <code>?</code>, <code>?|</code>, <code>?&</code>, <code>||</code>, <code>-</code>, <code>#-</code>, <code>@?</code>, <code>@@</code>
|-
| Construction and conversion
| <code>json</code>, <code>to_json</code>, <code>to_jsonb</code>, <code>array_to_json</code>, <code>row_to_json</code>, <code>json_build_array</code>, <code>jsonb_build_array</code>, <code>json_build_object</code>, <code>jsonb_build_object</code>, <code>json_object</code>, <code>jsonb_object</code>, <code>json_scalar</code>, <code>json_serialize</code>
|-
| Inspection and modification
| <code>json_array_length</code>, <code>jsonb_array_length</code>, <code>json_typeof</code>, <code>jsonb_typeof</code>, <code>jsonb_set</code>, <code>jsonb_set_lax</code>, <code>jsonb_insert</code>, <code>json_strip_nulls</code>, <code>jsonb_strip_nulls</code>, <code>jsonb_pretty</code>
|-
| Set-returning functions
| <code>json[b]_array_elements[_text]</code>, <code>json[b]_each[_text]</code>, <code>json[b]_object_keys</code>, <code>json[b]_populate_record[set]</code>, <code>jsonb_populate_record_valid</code>, <code>json[b]_to_record[set]</code>
|-
| JSONPath
| <code>jsonb_path_exists</code>, <code>jsonb_path_match</code>, <code>jsonb_path_query</code>, <code>jsonb_path_query_array</code>, <code>jsonb_path_query_first</code>, plus their <code>_tz</code> variants
|-
| Aggregation
| <code>json_agg</code>, <code>jsonb_agg</code>, strict variants, <code>json_object_agg</code>/<code>jsonb_object_agg</code> and their <code>_strict</code>, <code>_unique</code> and <code>_unique_strict</code> variants
|-
| SQL/JSON
| <code>JSON_ARRAY</code>, <code>JSON_OBJECT</code>, <code>JSON_ARRAYAGG</code>, <code>JSON_OBJECTAGG</code>, <code>JSON_EXISTS</code>, <code>JSON_QUERY</code>, <code>JSON_VALUE</code>, <code>JSON_TABLE</code>
|}

<pre>
SELECT
payload->'customer'->>'name' AS name,
payload @> '{"tags":["sql"]}'::jsonb AS has_sql
FROM events
ORDER BY payload->'customer'->'rank';

SELECT jsonb_path_query_array(
'{"values":[1,2,3,4]}'::jsonb,
'$.values[*] ? (@ > 2)'
); -- [3,4]
</pre>

See PostgreSQL's official [https://www.postgresql.org/docs/current/functions-json.html JSON functions and operators] and [https://www.postgresql.org/docs/current/functions-aggregate.html aggregate functions] documentation for the source syntax and semantics. The tables above identify the forms currently accepted by MemCP.

== Building nested application documents ==

JSON constructors and aggregates compose with correlated subqueries. This allows a relational schema to emit complete API documents, including arrays nested several levels deep:

<pre>
SELECT JSON_OBJECT(
'id', delivery_note.id,
'number', delivery_note.note_number,
'date', delivery_note.delivery_date,
'items', (
SELECT JSON_ARRAYAGG(JSON_OBJECT(
'id', delivery_item.id,
'sku', delivery_item.sku,
'quantity', delivery_item.quantity,
'serialNumbers', (
SELECT JSON_ARRAYAGG(JSON_OBJECT(
'id', serial_number.id,
'value', serial_number.serial_number
))
FROM delivery_serial_numbers AS serial_number
WHERE serial_number.delivery_item_id = delivery_item.id
)
))
FROM delivery_items AS delivery_item
WHERE delivery_item.delivery_note_id = delivery_note.id
)
) AS document
FROM delivery_notes AS delivery_note
ORDER BY delivery_note.delivery_date DESC;
</pre>

<code>JSON_ARRAYAGG</code> collects values first and emits one exact-sized BSON array at aggregate finalization. Nested aggregation therefore avoids repeatedly copying a growing JSON string. The same finalization is applied independently to scalar subqueries, groups and set-operation branches.

== Storage and indexing ==

* A declared <code>JSON</code> or <code>JSONB</code> column rejects invalid documents on <code>INSERT</code> and <code>UPDATE</code>.
* MemCP stores one native BSON value, not both a JSON string and a parsed tree. BSON is a normal tagged SCM value and needs no JSON-specific storage engine.
* Serialization is canonical rather than lexical: insignificant whitespace is not retained, object key order is deterministic, and duplicate object keys do not remain separate.
* JSON extraction expressions can be used in <code>WHERE</code>, <code>GROUP BY</code> and <code>ORDER BY</code>. Eligible deterministic expressions can be materialized as computed columns and served by computed indexes, so document-style tables do not require reparsing every row for each query.

For frequently filtered or sorted attributes, keep the JSON payload for flexibility and expose the hot path as a deterministic expression used consistently by queries. Fully relational columns remain preferable for keys, high-selectivity joins and attributes with strong schema constraints.

== Compatibility notes ==

MemCP aims at practical query compatibility, but it does not preserve PostgreSQL's textual <code>json</code> representation separately from <code>jsonb</code>, nor MariaDB's LONGTEXT representation. Native BSON is used for both dialects. Applications that depend on original whitespace, duplicate-key preservation or byte-for-byte round trips should store the original document in a separate text column.

Unknown or newly introduced upstream JSON functions should be treated as unsupported until they appear in the supported tables above and in MemCP's compatibility tests.

== Frequently asked questions about SQL JSON in MemCP ==

=== Does MemCP support MySQL JSON functions? ===

Yes. MemCP supports commonly used MySQL JSON functions and operators for construction, extraction, search, modification, aggregation, schema validation and <code>JSON_TABLE</code>. The complete currently supported surface is listed under [[#MySQL and MariaDB syntax]].

=== Can MemCP run JSON_EXTRACT and JSON_VALUE on a string? ===

Yes. Both functions accept valid JSON text. A declared <code>JSON</code> column is more efficient for repeated queries because MemCP validates and converts the document when it is written.

=== Does MemCP support PostgreSQL JSONB? ===

Yes. The PostgreSQL-compatible parser supports <code>json</code>/<code>jsonb</code> casts, extraction and containment operators, JSONPath, constructors, aggregates and set-returning JSON functions. MemCP represents both SQL types with the same immutable BSON value internally.

=== Can MemCP index a field inside a JSON document? ===

JSON extraction can be used as a deterministic computed expression. When eligible, MemCP can materialize that expression and use a computed index for filters or ordering by the extracted JSON attribute.

=== Should all application data be stored in one JSON column? ===

It is possible to use a document-style table containing an ID, timestamps and a JSON payload. Relational columns are still preferable for primary keys, frequently joined keys and strongly constrained attributes. A hybrid schema normally gives the optimizer more options while retaining JSON flexibility.

[[Category:SQL]]
[[Category:JSON]]
[[Category:MySQL]]
[[Category:PostgreSQL]]

Current Status and Open Issues

2026-08-21T15:11:54Z

Carli:

<div style="padding:1rem 1.2rem; margin:0 0 1.5rem; border-left:5px solid #76b512; background:#f5f8f0; color:#17202a;">
'''Status: Beta · Verified against commit <code>e90d6ce7fd</code> on 21 August 2026.''' MemCP is usable for tested workloads, but SQL compatibility and complex planner cases continue to evolve. Validate every application-critical query, durability requirement and client before a production migration.
</div>

This page describes the current implementation and the remaining publicly tracked work. It replaces the old static TODO list from November 2024. Active tasks and priorities belong in the [https://github.com/launix-de/memcp/issues GitHub issue tracker]; completed work belongs in the changelog and repository history.

== Current status ==

{| class="wikitable" style="width:100%;"
! Area
! Status
! Notes
|-
| SQL frontend
| Broad support, Beta compatibility
| DDL, DML, joins, transactions, views, triggers, window functions, UNION and extensively tested subqueries are available. MemCP does not claim complete MySQL or PostgreSQL compatibility. See [[Supported SQL]].
|-
| Storage and durability
| Operational
| Per-table <code>safe</code>, <code>logged</code>, <code>sloppy</code>, <code>memory</code> and <code>cache</code> engines are implemented. See [[Persistency and Performance Guarantees]] before choosing or changing an engine.
|-
| Query planner
| New architecture implemented, actively optimized
| Logical decorrelation and join ordering are separated from cost-based physical lowering. RecSets, indexes, direct scans, group caches, ORC columns and pipeline variants are physical alternatives. See [[Query Planner and Physical Lowering]].
|-
| Transactions and constraints
| Implemented for tested semantics
| Cursor-stability transactions, an explicit ACID/OCC mode, rollback, unique and NOT NULL constraints, and enforced CREATE TABLE foreign keys are covered by tests. XA and complete MySQL isolation-level compatibility are not claimed.
|-
| Operations
| Implemented, still maturing
| Dashboard, metrics, processlist, query/connection cancellation, graceful shutdown, memory budgets and eviction are available.
|-
| Storage backends
| Implemented
| Local files, S3-compatible object storage and optional Ceph/RADOS are supported. Backend-specific outage, backup and latency behavior must be validated operationally.
|-
| RDF/SPARQL
| Supported subset
| Query, FILTER, OPTIONAL, update forms and Turtle loading are tested. This is not a claim of complete SPARQL standard coverage.
|-
| Scheme runtime
| Operational
| The functional runtime, concurrency primitives and native x86-64 JIT are integrated. Unsupported JIT shapes fall back to interpreted execution.
|}

== What changed since the old roadmap ==

The previous page was a development checklist. The table below preserves every old item and records what happened to it.

=== Storage engine ===

{| class="wikitable" style="width:100%;"
! Original roadmap item
! Current result
|-
| <code>ALTER TABLE … ENGINE</code>
| '''Implemented.''' Tables can transition between the five MemCP engines. Some transitions change durability; persisted → <code>memory</code> permanently removes on-disk data.
|-
| LRU garbage collection for temporary columns
| '''Implemented and expanded.''' Global memory budgets cover persisted columns, indexes, temporary/computed columns, keytables, cache tables and dictionaries. Reloadable data can be evicted; memory-engine rows cannot.
|-
| Triggers and change hooks on computed columns
| '''Implemented.''' Persistent SQL triggers and hidden dependency/invalidation triggers maintain computed columns, group caches, prejoins and order-dependent caches.
|-
| Respect foreign keys
| '''Implemented for foreign keys declared with CREATE TABLE.''' RESTRICT, CASCADE and SET NULL paths are tested for inserts, deletes and key updates. Some ALTER TABLE foreign-key forms remain compatibility-only syntax.
|-
| Serialize into memory-mapped database-sized key/value files
| '''Not adopted as the persistence architecture.''' MemCP retains versioned per-column storage and WAL files and now also supports object backends. Selected storage/JIT implementations use mmap internally, but there is no single 100-GiB database blob contract.
|-
| Sort index delta storage and merge it with the main index
| '''Implemented.''' Ordered iteration merges the compressed main permutation with an ordered index-local delta structure.
|-
| Multi-table <code>scan_star</code>/merge-join operator
| '''Superseded by the current planner.''' Logical join trees are reordered and lowered into costed nested, parallel, indexed, ordered, RecSet or cached physical paths instead of one mandatory combined operator.
|-
| Process IDs, request context and kill switch
| '''Implemented.''' <code>SHOW [FULL] PROCESSLIST</code>, dashboard process controls, <code>KILL QUERY</code>, <code>KILL CONNECTION</code> and cancellation-aware table waits are available.
|-
| Transaction insertion/deletion overlays and conflict handling
| '''Implemented and expanded.''' The storage engine provides cursor-stability undo handling plus snapshot/OCC transaction infrastructure, shard visibility, commit conflict checks, rollback and WAL synchronization at commit.
|-
| Indexes for LIKE queries
| '''Implemented.''' LIKE-prefix and adaptive match-set boundaries participate in scan and physical cost decisions. Residual predicates remain when a boundary is not an exact proof.
|}

=== RDF frontend ===

The three former RDF items are implemented and covered by integration tests:

* SPARQL update with <code>DELETE { … } INSERT { … } WHERE { … }</code>;
* update templates driven by SELECT/WHERE bindings;
* <code>OPTIONAL { … }</code> with left-join behavior, including unmatched and multiple optional blocks.

RDF support remains a tested subset. Unsupported SPARQL syntax should not be inferred from the implementation of these constructs.

=== SQL frontend ===

{| class="wikitable" style="width:100%;"
! Original roadmap item
! Current result
|-
| AUTO_INCREMENT
| '''Implemented and persisted''', including restart and ALTER TABLE behavior.
|-
| Convert subqueries into joins
| '''Implemented far beyond the original item.''' The logical planner decorrelates scalar, IN/NOT IN and EXISTS/NOT EXISTS shapes using Neumann-style domains before join reordering and physical lowering.
|-
| Shard group tables like their group keys
| '''Implemented as a physical option.''' Group keytables can be partitioned from group-key information when the cost model selects reusable materialization.
|-
| Prejoin complex plans
| '''Implemented as one of several alternatives.''' Prejoins, FK/PK group reuse, group caches, direct scans and RecSets compete according to semantics and cost.
|-
| Restrict users to databases
| '''Implemented.''' Users, grants, revokes and database access policies are represented in system metadata and enforced by the SQL/API frontends.
|-
| Test DBeaver, phpMyAdmin and metadata compatibility
| '''Ongoing compatibility work.''' MySQL protocol, prepared statements, SHOW metadata and selected INFORMATION_SCHEMA relations are implemented, but individual client releases still require dated compatibility tests.
|}

=== Scheme language and infrastructure ===

* '''Native JIT: implemented.''' Supported hot Scheme procedures can be compiled to x86-64 machine code, including an expanding set of calls, control flow and planner-generated lambdas. Unsupported procedures continue in the interpreter.
* '''HTTP/IPFS stream filenames: not generally implemented.''' Local files and supported virtual archive paths are available; arbitrary <code>http://</code> and <code>ipfs://</code> loading should not be documented as supported.
* '''General native plugin system: not implemented as a stable public API.''' MemCP has extension hooks such as storage boundary matchers and embedded Scheme modules, but no promised ABI for arbitrary C++/GPU plugins.

== Current roadmap ==

There is currently no published release milestone with a fixed feature schedule. As of the verification date, the public issue tracker contains one open feature issue:

* [https://github.com/launix-de/memcp/issues/9 #9 – Vector extension: helper functions]

=== Next SQL roadmap: JSON compatibility ===

JSON support is planned as a compatibility and integration feature, not as a replacement for relational modelling. Frequently queried fields should normally remain typed columns with constraints, statistics and indexes. JSON is useful for sparse external attributes, application payloads and migration compatibility; the SQL interface must still make extraction into relational rows and typed values straightforward.

MemCP already has JSONL import, JSON serialization for HTTP and Scheme, and type mapping for MySQL <code>json</code> and PostgreSQL <code>json</code>/<code>jsonb</code> imports. These facilities do not yet constitute MySQL, PostgreSQL or SQL/JSON query compatibility.

{| class="wikitable" style="width:100%;"
! Phase
! Planned scope
! Required semantic decisions
|-
| 1. JSON value contract
| A documented SQL JSON type/storage contract; validation and canonical serialization; <code>JSON_VALID</code>, <code>JSON_TYPE</code>, <code>JSON_QUOTE</code> and <code>JSON_UNQUOTE</code>; casts between text and JSON.
| Distinguish SQL NULL, JSON <code>null</code> and a missing path; preserve numeric precision; define duplicate-object-key behavior, invalid-input errors and equality/comparison rules before adding optimizer shortcuts.
|-
| 2. Paths and read access
| MySQL-compatible <code>JSON_EXTRACT</code>, <code>JSON_VALUE</code>, <code>JSON_QUERY</code>, <code>JSON_EXISTS</code>, <code>-></code> and <code>->></code>; <code>JSON_LENGTH</code>, <code>JSON_DEPTH</code>, <code>JSON_KEYS</code>, <code>JSON_CONTAINS</code>, <code>JSON_CONTAINS_PATH</code> and <code>JSON_SEARCH</code> where compatible semantics are defined.
| Define one parsed path representation shared by functions and operators. Constant paths should be parsed once, not once per row. Dynamic paths must have explicit cost and error behavior.
|-
| 3. Construction and aggregation
| <code>JSON_OBJECT</code>, <code>JSON_ARRAY</code>, <code>JSON_OBJECTAGG</code> and <code>JSON_ARRAYAGG</code> with deterministic documented handling of NULLs, duplicate keys and aggregate ordering.
| Constructors must not silently conflate Scheme association lists, SQL rows and JSON objects. Aggregate results need stable semantics across shards and parallel execution.
|-
| 4. Mutation
| <code>JSON_SET</code>, <code>JSON_INSERT</code>, <code>JSON_REPLACE</code>, <code>JSON_REMOVE</code>, array append/insert functions and merge-patch behavior.
| Functions remain pure expressions; an SQL UPDATE writes the resulting JSON value. Partial-update storage optimization may follow later but must not change transactional, trigger or durability semantics.
|-
| 5. Relational projection
| SQL/JSON <code>JSON_TABLE</code> or an equivalent table-producing operator, with typed columns, nested paths, defaults and explicit error/empty behavior.
| JSON expansion belongs in the logical relational plan and must participate correctly in joins, correlation, NULL extension, cardinality estimation and cancellation. It must not be hidden as an uncosted per-row fallback.
|-
| 6. Indexing and physical optimization
| Reusable computed columns and adaptive indexes over stable JSON path expressions; statistics for extracted scalar values; batch extraction for scans.
| Optimize declared path expressions rather than pretending an opaque document is relationally indexed. Invalidating or updating a JSON document must invalidate every dependent extracted value safely.
|-
| 7. Dialect compatibility
| Map PostgreSQL <code>json</code>/<code>jsonb</code> extraction and containment operators such as <code>-></code>, <code>->></code>, <code>#></code>, <code>#>></code> and <code>@></code> onto the same semantic core where behavior is genuinely compatible.
| MySQL, PostgreSQL and SQL/JSON differ in path grammar, scalar return types, ordering, containment and errors. Dialect spellings may share internals, but must not be advertised as aliases when their observable semantics differ.
|}

Definition of done for every JSON feature:

* successful and must-fail SQL tests for SQL NULL, JSON null, missing paths, malformed JSON and malformed paths;
* MySQL and PostgreSQL dialect tests where the feature is advertised for both;
* persistence/restart, trigger, transaction and prepared-statement coverage;
* batch and allocation benchmarks before enabling path extraction in scan hot loops;
* EXPLAIN visibility for table expansion, computed extraction and index use;
* explicit documentation of deviations from MySQL, PostgreSQL or the SQL/JSON standard.

No phase above has a promised release date. Before implementation starts, the agreed compatibility slice should be represented by one or more GitHub issues and tests, rather than treating every vendor-specific JSON function as automatically in scope.

Recent development has concentrated on correctness and cost calibration of the planner, RecSet representations and bulk scans, JIT coverage, shard rebuild concurrency and query cancellation. Git history shows completed changes; it does not constitute a delivery promise for future work.

Areas that continue to need testing and incremental improvement include:

* MySQL/PostgreSQL syntax and metadata compatibility required by real applications;
* rare combinations of correlated subqueries, outer joins, UNION, grouping, windows and SQL three-valued logic;
* transaction contention, long-running rebuilds and crash recovery across engine/backends;
* ORM, administration-tool and connector compatibility by concrete version;
* non-x86-64 platform behavior where native JIT acceleration is unavailable;
* reproducible performance measurements across representative workloads.

These are quality areas, not commitments to a particular release date. Concrete work should be represented by a GitHub issue before this page calls it scheduled.

== Reporting an issue ==

Please open a [https://github.com/launix-de/memcp/issues/new GitHub issue] with:

* MemCP version or commit;
* frontend used: MySQL protocol, <code>/sql</code>, <code>/psql</code>, RDF or embedded Scheme;
* minimal schema, setup data and query;
* expected and actual result;
* table ENGINE and storage backend;
* whether the problem reproduces after a clean restart;
* for performance reports: row counts, EXPLAIN output, hardware, concurrency, cache state, successful response validation and raw samples.

See [[Contributing]] for the development and test workflow.

Main Page

2026-08-21T14:23:48Z

Carli:

= MemCP – A Modern In-Memory Columnar Database =

<div style="padding:2.5rem 2rem; margin:0 0 1.5rem; border-radius:14px; background:linear-gradient(135deg,#3d6208 0%,#64910c 52%,#76b512 100%); color:#fff; text-align:center;">
<div style="font-size:2.6rem; line-height:1.1; font-weight:700; margin-bottom:.7rem;">MemCP</div>
<div style="font-size:1.45rem; line-height:1.35; font-weight:600; margin:0 auto .9rem; max-width:56rem;">A fast, compressed, MySQL-compatible columnar database for modern OLTP and OLAP workloads</div>
<div style="font-size:1.05rem; line-height:1.6; margin:0 auto 1.4rem; max-width:54rem;">MemCP combines persistent in-memory operation, adaptive column compression, parallel query execution and direct application APIs in a lightweight database written in Go.</div>
<div style="margin-bottom:1.5rem;">Beta Open Source MySQL Protocol + HTTP APIs</div>
<div class="plainlinks">[[Install MemCP with Docker|Get started with Docker]] [[Supported SQL|Explore SQL support]] [https://github.com/launix-de/memcp View on GitHub]</div>
</div>

[[File:Memcp-Load.png|center|frameless|1000px|alt=MemCP database dashboard and workload overview]]

<div style="padding:.85rem 1rem; margin:1rem 0 2rem; border-left:5px solid #d99b00; background:#fff7d6; color:#332600;">
'''Development status: Beta.''' MemCP is under active development. Check [[Supported SQL]], [[Current Status and Open Issues]] and the durability requirements of your workload before migrating production data.
</div>

== Choose your path ==

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(220px,1fr)); gap:1rem; margin:1rem 0 2rem;">
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Evaluate MemCP</div>
Understand the workload model, current status, hardware needs and differences from MySQL.

[[What is OLTP and OLAP|OLTP and OLAP]] · [[Comparison: MemCP vs. MySQL|Compare with MySQL]] · [[Hardware Requirements|Hardware requirements]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Build an application</div>
Use the MySQL protocol, SQL over HTTP, RDF or application-specific endpoints inside MemCP.

[[Advanced SQL Tutorial|SQL tutorial]] · [[SQL over REST|SQL over HTTP]] · [[Database Tools compatibility with MemCP|Client tooling]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Operate MemCP</div>
Plan deployment, migration, persistence, storage backends, settings and performance measurement.

[[Deployment]] · [[Persistency and Performance Guarantees|Durability]] · [[Migration from MySQL and PostgreSQL|Migration]]
</div>
<div style="padding:1rem 1.1rem; border:1px solid #b7c99a; border-radius:9px;">
<div style="font-size:1.15rem; font-weight:700; margin-bottom:.4rem;">Understand and contribute</div>
Explore the storage engine, optimizer, embedded Scheme runtime and project internals.

[[Query Planner and Physical Lowering|Query planner]] · [[Columnar Storage]] · [[Contributing]]
</div>
</div>

== Quickstart ==

=== Run with Docker ===

<syntaxhighlight lang="bash">
docker run --name memcp \
-e ROOT_PASSWORD='choose-a-password' \
-p 4321:4321 -p 3307:3307 \
carli2/memcp:latest
</syntaxhighlight>

Then open <code>http://localhost:4321</code> and connect a MySQL application to <code>127.0.0.1</code> on port '''3307'''. Add a volume mounted at <code>/data</code> when the container must retain persistent data. See [[Install MemCP with Docker|the complete Docker guide]] before deploying it as a service.

=== Build from source ===

<syntaxhighlight lang="bash">
git clone https://github.com/launix-de/memcp
cd memcp
go mod download
make
./memcp --api-port=4321 --mysql-port=3307 lib/main.scm
</syntaxhighlight>

Connect with MySQL tooling:

<syntaxhighlight lang="bash">
mysql -h 127.0.0.1 -u root -p -P 3307
# Enter the password selected for this data directory.
</syntaxhighlight>

The development default for a fresh data directory is <code>admin</code>. Change it before exposing MemCP to another machine. MemCP can also be supervised with PM2:

<syntaxhighlight lang="bash">
pm2 start ./memcp --name memcp -- \
--no-repl -data ./data --api-port=4321 --mysql-port=3307 lib/main.scm
</syntaxhighlight>

Background deployments must use <code>--no-repl</code> so that closing standard input does not terminate the interactive console and stop the server. See [[Compile MemCP from Source|Build from Source]], [[Deployment]] and [[MemCP Console]].

== Key features ==

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(260px,1fr)); gap:.8rem; margin:1rem 0;">
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''High performance''' NUMA-aware, parallelized query execution is optimized for multicore CPUs, large caches and NVMe SSDs, serving both OLTP and OLAP workloads.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Columnar storage''' Data is stored by column for improved compression, a smaller memory footprint and fast analytical access.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Persistent in-memory operation''' MemCP is designed to keep active data in memory while offering configurable per-table durability and persistence backends.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Built-in APIs''' SQL over HTTP and in-database services can remove an extra middleware hop for suitable applications.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Adaptive compression''' Bit-packing, dictionary encoding and sequence compression can reduce suitable datasets by up to 80% compared with their MySQL/MariaDB representation.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Simple deployment''' Start with Docker, PM2 or the native binary. The compact application has historically had an installation footprint of approximately 10 MB.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #76b512; background:#f5f8f0; color:#17202a;">'''Extensible frontends''' The Go storage engine and embedded Scheme environment support SQL, RDF, REST and custom application interfaces.</div>
<div style="padding:.9rem 1rem; border-top:4px solid #9ad32d; background:#f5f8f0; color:#17202a;">'''Cost-based query execution''' Logical decorrelation and join optimization are separated from cost-based physical selection of scans, indexes, RecSets, reusable caches and execution pipelines. [[Query Planner and Physical Lowering|How MemCP plans and lowers queries →]]</div>
</div>

== Why MemCP? ==

Traditional relational databases were designed around spinning disks and comparatively small numbers of CPU cores. MemCP rethinks storage and query execution for modern multicore systems, large caches, fast storage and mixed workloads.

Typical use cases include:

* real-time dashboards and analytics;
* data-heavy SaaS platforms;
* embedded systems with limited resources;
* high-throughput OLTP/OLAP hybrids.

Whether MemCP is a good fit depends on the required SQL compatibility, durability, query mix, data size and operational environment. Review [[Supported SQL]], [[Persistency and Performance Guarantees]] and [[Performance Measurement]] rather than treating benchmark figures as universal guarantees.

== MemCP vs. MySQL ==

{| class="wikitable" style="width:100%;"
! Feature
! MySQL
! MemCP
|-
| Storage model
| Primarily row-based
| Column-based and compressed
|-
| Performance focus
| General-purpose relational workloads
| NUMA-aware, in-memory execution for mixed operational and analytical workloads
|-
| In-memory capability
| Available through selected engines and caching
| Central design goal and default operating model
|-
| REST API integration
| Normally external
| Built in
|-
| Installation footprint
| Common server installations are approximately 150 MB or larger
| The native application has historically been approximately 10 MB
|-
| Open source
| ✅
| ✅
|}

MemCP provides MySQL protocol compatibility but does not claim to implement every MySQL feature. See [[Comparison: MemCP vs. MySQL]] for a detailed comparison and [[Database Tools compatibility with MemCP]] for client compatibility.

== Architecture overview ==

* '''Tables, schemas and columns''': Familiar SQL structures use a compressed columnar physical layout. See [[Databases, Tables and Columns]] and [[Columnar Storage]].
* '''Transaction model''': Delta and main storage support mixed OLTP and OLAP semantics. See [[Transactions and Isolation]] and [[Shards, RecordIDs, Main Storage, Delta Storage]].
* '''Query planning''': Logical optimization is separated from physical execution decisions. See [[Query Planner and Physical Lowering]].
* '''Persistence''': Per-table durability can use filesystem, S3 or Ceph/RADOS storage. See [[Persistency and Performance Guarantees]].
* '''Frontends''': MemCP provides multiple query and application interfaces:
** SQL through the MySQL wire protocol and SQL over HTTP;
** an RDF/graph query frontend;
** custom APIs through in-database web applications.

== Documentation ==

The following navigation remains on the start page so users and search engines can reach every major documentation area directly.

=== Introduction and evaluation ===

* [[What is OLTP and OLAP]]
* [[History of the MemCP project]]
* [[Hardware Requirements]]
* [[Persistency and Performance Guarantees]]
* [[Current Status and Open Issues]]
* [[Comparison: MemCP vs. MySQL]]

=== Getting started ===

* [[Install MemCP with Docker|Install with Docker]]
* [[With Singularity|Install with Singularity/Apptainer]]
* [[Compile MemCP from Source|Build from Source]]
* [[Contributing]]
* [[Introduction to Scheme]]
* [[Full SCM API documentation]]

=== Administration ===

* [[Deployment]]
* [[Migration from MySQL and PostgreSQL]]
* [[Settings]]
* [[Process Hibernation]]
* [[Performance Measurement]]
* [[MemCP Console]]

=== Frontends ===

==== SQL frontend ====

* [[Supported SQL]]
* [[Advanced SQL Tutorial]]
* [[SQL over REST]]
* [[Database Tools compatibility with MemCP|Supported Tooling]]
* [[How SQL Operators are implemented on MemCP]]
* [[Add custom SQL operators to MemCP]]

==== RDF frontend ====

* [[Introduction to RDF]]
* [[Advanced Graph Querying]]
* [[RDF templating and model driven development]]

==== Custom frontends ====

* [[In-Database WebApps|In-Database WebApps and REST Services]]
* [[MemCP for Microservices]]
* [[Websockets in MemCP]]

=== Persistence backends and storage ===

* [[File System]]
* [[S3 Buckets]]
* [[Ceph/Rados]]
* [[Cluster Monitor]]

=== Internals ===

==== How MemCP works ====

* [[Databases, Tables and Columns]]
* [[Shards, RecordIDs, Main Storage, Delta Storage]]
* [[Columnar Storage]]
* [[Transactions and Isolation]]
* [[Query Planner and Physical Lowering]]
* [[Full SCM API documentation]]

==== Scheme documentation ====

* [[SCM Builtins]]
* [[Arithmetic / Logic]]
* [[Strings]]
* [[Streams]]
* [[Lists]]
* [[Associative Lists / Dictionaries]]
* [[Date]]
* [[Vectors]]
* [[Parsers]]
* [[Sync]]
* [[IO]]
* [[Storage]]

==== Optimizations ====

* [[Query Planner and Physical Lowering]]
* [[In-Memory Compression, Columnar Compression Techniques]]
* [[Temporary Columns]]
* [[Data Auto Sharding and Auto Indexing]]
* [[Parallel Computing]]

== Further reading ==

* [https://github.com/launix-de/memcp MemCP on GitHub]
* [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]
* [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]
* [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]

Additional blog posts on design decisions, compression techniques and performance optimization are available on the [https://launix.de/launix/ Launix blog].

== Community ==

MemCP is an open-source project maintained by developers for developers. Contributions are welcome in the form of bug reports, feature requests and pull requests.

See [[Contributing]] and the [https://github.com/launix-de/memcp GitHub repository].

Main Page

2026-08-21T11:27:06Z

Carli:

= MemCP – A Modern In-Memory Columnar Database =

'''MemCP is a high-performance, in-memory, column-oriented database designed for modern workloads.'''
It provides a lightweight, developer-friendly alternative to traditional relational databases such as MySQL, with a focus on speed, compression, and direct API integration.
[[File:Memcp-Load.png|center|frameless|1600x1600px]]

----

== Quickstart ==
docker run --name memcp \
-e ROOT_PASSWORD='choose-a-password' \
-p 4321:4321 -p 3307:3307 \
carli2/memcp:latest
Then open http://localhost:4321 and connect your MySQL application via 127.0.0.1 port '''3307'''.

== Key Features ==

* '''High Performance''': NUMA-aware, parallelized query execution optimized for multicore CPUs, large caches, and NVMe SSDs. Handles both OLTP and OLAP workloads efficiently.
* '''Columnar Storage''': Data is stored by column for improved compression, reduced memory footprint, and faster analytical queries.
* '''In-Memory Operation''': Designed to keep data in memory, with configurable persistence backends for durability.
* '''Built-in APIs''': Exposes RESTful endpoints directly from the database, reducing middleware overhead.
* '''Compression''': Multiple strategies (bit-packing, dictionary encoding, sequence compression) reduce storage by up to 80% compared to MySQL/MariaDB.
* '''Simple Deployment''': Start with a single <code>docker run</code> or <code>pm2 start</code> command. Lightweight footprint (~10MB).
* '''Extensible''': Written in Go, with pluggable storage backends and custom frontend support (SQL, RDF, REST).

----

== Why MemCP? ==

Traditional relational databases were designed decades ago, optimized for spinning disks and single-core CPUs.
MemCP rethinks the core design for today’s hardware and workloads:

* Real-time dashboards and analytics
* Data-heavy SaaS platforms
* Embedded systems with limited resources
* High-throughput OLTP/OLAP hybrids

----

== Quick Start ==

Clone and build MemCP from source:

<pre>
git clone https://github.com/launix-de/memcp
cd memcp
go get
make
pm2 start ./memcp ./data/
</pre>

Connect with MySQL tooling:

<pre>
mysql -u root -p -P 3307
Enter password: admin
</pre>

----

== MemCP vs. MySQL ==

{| class="wikitable"
! Feature
! MySQL
! MemCP
|-
| Storage Model
| Row-based
| Column-based (compressed)
|-
| Performance
| Good
| NUMA-optimized, in-memory
|-
| In-Memory Capable
| Limited
| Yes (default)
|-
| REST API Integration
| External
| Built-in
|-
| Installation Footprint
| ~150MB+
| ~10MB
|-
| Open Source
| ✅
| ✅
|}

----

== Architecture Overview ==

* '''Tables, Schemas, Columns''': Familiar SQL-style structures with a columnar physical layout.
* '''Transaction Model''': Supports both OLTP and OLAP semantics with delta + main storage.
* '''Persistence''': Configurable storage backends (filesystem, S3, Ceph).
* '''Frontends''': Multiple query interfaces:
- SQL frontend (MySQL wire protocol + SQL over REST)
- RDF/graph query engine
- Custom APIs via in-database web apps

----

== Documentation ==

* [[What is OLTP and OLAP]]
* [[History of the MemCP project]]
* [[Hardware Requirements]]
* [[Persistency and Performance Guarantees]]
* [[Comparison: MemCP vs. MySQL]]
* [[Install MemCP with Docker|Install with Docker]]
* [[Compile MemCP from Source|Build from Source]]
* [[Contributing]]
* [[SQL over REST]]
* [[In-Database WebApps|REST & Microservices]]

----

===Navigation===

====Introduction====
*[[What is OLTP and OLAP]]
*[[History of the MemCP project]]
*[[Hardware Requirements]]
*[[Persistency and Performance Guarantees]]
*[[Current Status and Open Issues]]
*[[Comparison: MemCP vs. MySQL]]

====Getting Started====
*[[Install MemCP with Docker|With Docker]]
*[[With Singularity]]
*[[Compile MemCP from Source|Build from Source]]
*[[Contributing]]
*[[Introduction to Scheme]]
*[[Full SCM API documentation]]

====Administration====

* [[Deployment]]
* [[Migration from MySQL and PostgreSQL]]
* [[Settings]]
*[[Process Hibernation]]
*[[Performance Measurement]]
*[[MemCP Console]]
*[[Performance Measurement]]

====Frontends====

=====SQL Frontend=====
*[[Supported SQL]]
*[[Advanced SQL Tutorial]]
*[[SQL over REST]]
*[[Database Tools compatibility with MemCP|Supported Tooling]]
*[[How SQL Operators are implemented on MemCP]]
*[[Add custom SQL operators to MemCP]]

=====RDF Frontend=====
*[[Introduction to RDF]]
*[[Advanced Graph Querying]]
*[[RDF templating and model driven development]]

=====Custom Frontends=====

*[[In-Database WebApps|In-Database WebApps and REST Services]]
*[[MemCP for Microservices]]
*[[Websockets in MemCP]]

==== Persistency Backends (= Storage) ====

* [[File System]]
* [[S3 Buckets]]
* [[Ceph/Rados]]
* [[Cluster Monitor]]

====Internals====

=====How things work in MemCP=====

*[[Databases, Tables and Columns]]
*[[Shards, RecordIDs, Main Storage, Delta Storage]]
*[[Columnar Storage]]
*[[Transactions and Isolation]]
*[[Full SCM API documentation]]
*[[Query Planner and Physical Lowering]]

===== SCM Documentation =====

* [[SCM Builtins]]
* [[Arithmetic / Logic]]
* [[Strings]]
* [[Streams]]
* [[Lists]]
* [[Associative Lists / Dictionaries]]
* [[Date]]
* [[Vectors]]
* [[Parsers]]
* [[Sync]]
* [[IO]]
* [[Storage]]

=====Optimizations=====
*[[In-Memory Compression, Columnar Compression Techniques]]
*[[Temporary Columns]]
*[[Data Auto Sharding and Auto Indexing]]
* [[Parallel Computing]]

----

== Further Reading ==

* [https://github.org/launix-de/memcp MemCP on GitHub]
* [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]
* [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]
* [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]

Additional blog posts on design decisions, compression techniques, and performance optimization are available on the [https://launix.de/launix/ Launix blog].

----

== Community ==

MemCP is an open-source project maintained by developers for developers.
Contributions are welcome — whether in the form of bug reports, feature requests, or pull requests.

See: [[Contributing]]

Main Page

2026-08-21T11:18:36Z

Carli: /* Navigation */

= MemCP – A Modern In-Memory Columnar Database =

'''MemCP is a high-performance, in-memory, column-oriented database designed for modern workloads.'''
It provides a lightweight, developer-friendly alternative to traditional relational databases such as MySQL, with a focus on speed, compression, and direct API integration.
[[File:Memcp-Load.png|center|frameless|1600x1600px]]

----

== Key Features ==

* '''High Performance''': NUMA-aware, parallelized query execution optimized for multicore CPUs, large caches, and NVMe SSDs. Handles both OLTP and OLAP workloads efficiently.
* '''Columnar Storage''': Data is stored by column for improved compression, reduced memory footprint, and faster analytical queries.
* '''In-Memory Operation''': Designed to keep data in memory, with configurable persistence backends for durability.
* '''Built-in APIs''': Exposes RESTful endpoints directly from the database, reducing middleware overhead.
* '''Compression''': Multiple strategies (bit-packing, dictionary encoding, sequence compression) reduce storage by up to 80% compared to MySQL/MariaDB.
* '''Simple Deployment''': Start with a single <code>docker run</code> or <code>pm2 start</code> command. Lightweight footprint (~10MB).
* '''Extensible''': Written in Go, with pluggable storage backends and custom frontend support (SQL, RDF, REST).

----

== Why MemCP? ==

Traditional relational databases were designed decades ago, optimized for spinning disks and single-core CPUs.
MemCP rethinks the core design for today’s hardware and workloads:

* Real-time dashboards and analytics
* Data-heavy SaaS platforms
* Embedded systems with limited resources
* High-throughput OLTP/OLAP hybrids

----

== Quick Start ==

Clone and build MemCP from source:

<pre>
git clone https://github.com/launix-de/memcp
cd memcp
go get
make
pm2 start ./memcp ./data/
</pre>

Connect with MySQL tooling:

<pre>
mysql -u root -p -P 3307
Enter password: admin
</pre>

----

== MemCP vs. MySQL ==

{| class="wikitable"
! Feature
! MySQL
! MemCP
|-
| Storage Model
| Row-based
| Column-based (compressed)
|-
| Performance
| Good
| NUMA-optimized, in-memory
|-
| In-Memory Capable
| Limited
| Yes (default)
|-
| REST API Integration
| External
| Built-in
|-
| Installation Footprint
| ~150MB+
| ~10MB
|-
| Open Source
| ✅
| ✅
|}

----

== Architecture Overview ==

* '''Tables, Schemas, Columns''': Familiar SQL-style structures with a columnar physical layout.
* '''Transaction Model''': Supports both OLTP and OLAP semantics with delta + main storage.
* '''Persistence''': Configurable storage backends (filesystem, S3, Ceph).
* '''Frontends''': Multiple query interfaces:
- SQL frontend (MySQL wire protocol + SQL over REST)
- RDF/graph query engine
- Custom APIs via in-database web apps

----

== Documentation ==

* [[What is OLTP and OLAP]]
* [[History of the MemCP project]]
* [[Hardware Requirements]]
* [[Persistency and Performance Guarantees]]
* [[Comparison: MemCP vs. MySQL]]
* [[Install MemCP with Docker|Install with Docker]]
* [[Compile MemCP from Source|Build from Source]]
* [[Contributing]]
* [[SQL over REST]]
* [[In-Database WebApps|REST & Microservices]]

----

===Navigation===

====Introduction====
*[[What is OLTP and OLAP]]
*[[History of the MemCP project]]
*[[Hardware Requirements]]
*[[Persistency and Performance Guarantees]]
*[[Current Status and Open Issues]]
*[[Comparison: MemCP vs. MySQL]]

====Getting Started====
*[[Install MemCP with Docker|With Docker]]
*[[With Singularity]]
*[[Compile MemCP from Source|Build from Source]]
*[[Contributing]]
*[[Introduction to Scheme]]
*[[Full SCM API documentation]]

====Administration====

* [[Deployment]]
* [[Migration from MySQL and PostgreSQL]]
* [[Settings]]
*[[Process Hibernation]]
*[[Performance Measurement]]
*[[MemCP Console]]
*[[Performance Measurement]]

====Frontends====

=====SQL Frontend=====
*[[Supported SQL]]
*[[Advanced SQL Tutorial]]
*[[SQL over REST]]
*[[Database Tools compatibility with MemCP|Supported Tooling]]
*[[How SQL Operators are implemented on MemCP]]
*[[Add custom SQL operators to MemCP]]

=====RDF Frontend=====
*[[Introduction to RDF]]
*[[Advanced Graph Querying]]
*[[RDF templating and model driven development]]

=====Custom Frontends=====

*[[In-Database WebApps|In-Database WebApps and REST Services]]
*[[MemCP for Microservices]]
*[[Websockets in MemCP]]

==== Persistency Backends (= Storage) ====

* [[File System]]
* [[S3 Buckets]]
* [[Ceph/Rados]]
* [[Cluster Monitor]]

====Internals====

=====How things work in MemCP=====

*[[Databases, Tables and Columns]]
*[[Shards, RecordIDs, Main Storage, Delta Storage]]
*[[Columnar Storage]]
*[[Transactions and Isolation]]
*[[Full SCM API documentation]]
*[[Query Planner and Physical Lowering]]

===== SCM Documentation =====

* [[SCM Builtins]]
* [[Arithmetic / Logic]]
* [[Strings]]
* [[Streams]]
* [[Lists]]
* [[Associative Lists / Dictionaries]]
* [[Date]]
* [[Vectors]]
* [[Parsers]]
* [[Sync]]
* [[IO]]
* [[Storage]]

=====Optimizations=====
*[[In-Memory Compression, Columnar Compression Techniques]]
*[[Temporary Columns]]
*[[Data Auto Sharding and Auto Indexing]]
* [[Parallel Computing]]

----

== Further Reading ==

* [https://github.org/launix-de/memcp MemCP on GitHub]
* [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]
* [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]
* [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]

Additional blog posts on design decisions, compression techniques, and performance optimization are available on the [https://launix.de/launix/ Launix blog].

----

== Community ==

MemCP is an open-source project maintained by developers for developers.
Contributions are welcome — whether in the form of bug reports, feature requests, or pull requests.

See: [[Contributing]]

Dashboard and Operations

2026-08-21T11:04:48Z

Carli: Created page with "The <code>/dashboard</code> interface provides CPU/memory/request metrics, database and table browsing, shard/compression details, SQL/Scheme consoles, users and grants, runtime settings, process inspection, service state, and configured logs. <code>SHOW [FULL] PROCESSLIST</code>, dashboard process controls, <code>KILL QUERY</code>, and <code>KILL CONNECTION</code> expose active work and cancellation. Graceful shutdown stops new requests and waits up to <code>ShutdownDr..."

The <code>/dashboard</code> interface provides CPU/memory/request metrics, database and table browsing, shard/compression details, SQL/Scheme consoles, users and grants, runtime settings, process inspection, service state, and configured logs.

<code>SHOW [FULL] PROCESSLIST</code>, dashboard process controls, <code>KILL QUERY</code>, and <code>KILL CONNECTION</code> expose active work and cancellation. Graceful shutdown stops new requests and waits up to <code>ShutdownDrainSeconds</code> for in-flight work.

Optional <code>MetricsTracing</code> writes periodic samples. <code>ErrorQueryLog</code> and <code>PrintLog</code> store failed SQL and Scheme print/time output with configurable retention limits. These logs may contain SQL text or application data; restrict access and set appropriate limits.

Protect the dashboard with a strong password and network controls. Do not use it as a substitute for backups, external health checks, alerting, or backend monitoring. Watch memory budgets, errors, active requests, WAL/rebuild behavior, remote-storage latency, and restart validation.

Triggers

2026-08-21T11:03:47Z

Carli: Created page with "MemCP supports persistent BEFORE and AFTER triggers for INSERT, UPDATE, and DELETE. Trigger bodies can read <code>OLD</code> and <code>NEW<</code>, assign permitted <code>NEW</code> values, run conditional BEGIN/END bodies, and execute supported DML including <code>INSERT … SELECT</code>. Multiple triggers execute by registered priority. AFTER work may be configured as asynchronous, in which case its completion and failure are not identical to the originating statemen..."

MemCP supports persistent BEFORE and AFTER triggers for INSERT, UPDATE, and DELETE. Trigger bodies can read <code>OLD</code> and <code>NEW<</code>, assign permitted <code>NEW</code> values, run conditional BEGIN/END bodies, and execute supported DML including <code>INSERT … SELECT</code>.

Multiple triggers execute by registered priority. AFTER work may be configured as asynchronous, in which case its completion and failure are not identical to the originating statement's synchronous path. Trigger changes participate in transaction/savepoint recovery.

MemCP also uses hidden system triggers to enforce constraints and maintain computed columns, group caches, prejoins, and invalidation dependencies. User DROP/CASCADE behavior must not delete unrelated data; lifecycle callbacks are not general garbage collectors.

Test trigger persistence across restart, multi-row statements, NULL/default handling, errors, ordering, cross-table writes, and rollback. Recursive trigger graphs should be treated with particular care.

Memory Management and Eviction

2026-08-21T11:02:20Z

Carli: Created page with "MemCP tracks persisted shard columns, indexes, temporary columns, keytables, cache-engine objects, and string dictionaries in a global memory manager. The overall budget defaults to 50% of system RAM. A separate persisted-data budget defaults to 30%. Exact byte limits override percentages. If system available memory falls too low, MemCP can evict before its own configured budget is exhausted. Eviction is LRU-like and type-aware. Persistent shard eviction releases only..."

MemCP tracks persisted shard columns, indexes, temporary columns, keytables, cache-engine objects, and string dictionaries in a global memory manager.

The overall budget defaults to 50% of system RAM. A separate persisted-data budget defaults to 30%. Exact byte limits override percentages. If system available memory falls too low, MemCP can evict before its own configured budget is exhausted.

Eviction is LRU-like and type-aware. Persistent shard eviction releases only in-memory representations; files remain and are reloaded on demand. Shards with unflushed deltas cannot be evicted. <code>memory</code> tables are never evicted because that would destroy their only copy. <code>cache</code> tables are explicitly reconstructible and may be discarded under pressure.

Monitor both current/budgeted memory and reload behavior. A workload whose hot set exceeds the budget remains correct for persistent tables but may become storage-I/O bound. See [[Settings]] and [[Dashboard_and_Operations|Dashboard and Operations]].

Transactions and Isolation

2026-08-21T11:01:00Z

Carli: Created page with "Every SQL statement runs in an implicit transaction unless the session already has an explicit <code>BEGIN</code>/<code>COMMIT</code>/<code>ROLLBACK</code> transaction. The default cursor-stability mode applies writes directly with undo masks, gives statement atomicity, and rolls back on errors. MemCP also contains an ACID mode with snapshot visibility and optimistic commit conflict detection. Savepoints are used internally for nested trigger/error recovery. Transactio..."

Every SQL statement runs in an implicit transaction unless the session already has an explicit <code>BEGIN</code>/<code>COMMIT</code>/<code>ROLLBACK</code> transaction.

The default cursor-stability mode applies writes directly with undo masks, gives statement atomicity, and rolls back on errors. MemCP also contains an ACID mode with snapshot visibility and optimistic commit conflict detection. Savepoints are used internally for nested trigger/error recovery.

Transaction state is tracked per shard. Visibility includes main rows, delta rows, insert/delete overlays, transaction snapshot, and the active shard generation. Commit locks and publishes touched shards; rollback reverses staged or directly applied changes. Nested shard work has a bounded fanout to avoid deadlock and goroutine explosion.

For <code>safe</code> tables, WAL synchronization occurs at commit. <code>logged</code> writes WAL without fsync; <code>sloppy</code>, <code>memory</code>, and <code>cache</code> skip WAL durability. A transaction spanning tables therefore inherits the guarantees and risks of each table's ENGINE.

MemCP does not claim every MySQL isolation level, XA, or distributed transaction feature. Test conflicts and failure recovery required by the application.

Query Planner and Physical Lowering

2026-08-21T10:58:02Z

Carli: Created page with "MemCP compiles SQL into executable Scheme code. The planner first converts SQL into a logical representation, decorrelates subqueries, and chooses a join order. Only after those semantic decisions are complete does the physical lowerer select scans, indexes, caches, RecSets, computed columns, and other storage-engine operators. This separation is important: two queries that mean the same thing should reach the same logical representation even when they use different..."

MemCP compiles SQL into executable Scheme code. The planner first converts SQL into a logical representation, decorrelates subqueries, and chooses a join order. Only after those semantic decisions are complete does the physical lowerer select scans, indexes, caches, [[RecSets]], computed columns, and other storage-engine operators.

This separation is important: two queries that mean the same thing should reach the same logical representation even when they use different SQL spelling. The physical plan can then be chosen from statistics, parameters, ordering, limits, and the state of reusable intermediate relations.
==Planner pipeline==
-> MySQL or PostgreSQL parser AST
-> normalize_sql_syntax
-> decorrelate_logical_query / untangle_query
-> logical predicate placement and join_reorder / optimize
-> build_queryplan
-> Scheme optimizer and optional native JIT
-> storage-engine scans</syntaxhighlight>
Each phase owns a different kind of decision:
{| class="wikitable"
|-
!Phase
!Responsibility
|-
|Parser
|Preserve SQL structure and produce neutral query terms.
|-
|Normalization
|Remove parser-specific spelling and safe syntactic sugar while preserving SQL three-valued logic.
|-
|Decorrelation
|Replace dependent subqueries with explicit domains, stages, keys, and joins.
|-
|Logical optimization
|Place predicates across proven-safe boundaries, estimate cardinalities and selectivities, and choose the join tree.
|-
|Physical lowering
|Select concrete scan sources and operators, bind predicates to scans, and emit Scheme code.
|-
|Scheme optimization/JIT
|Optimize the emitted functional program and compile supported hot paths to native code.
|}Physical objects such as <code>scan</code>, <code>scan_order</code>, RecSets, keytables, ORC columns, and temporary tables must not leak into the logical representation. Conversely, the physical lowerer must not redo decorrelation or silently choose a different join order.
==Logical representation==
MemCP deliberately uses three coarse combined operator shapes instead of a long chain of textbook relational operators:
*<code>query-block</code> represents SELECT, JOIN, filter, projection, order, limit, and offset work.
*<code>group-stage</code> represents grouping, aggregates, correlated domains, EXISTS/IN/scalar helpers, HAVING, and window-partition work.
*<code>union-block</code> represents <code>UNION</code> and <code>UNION ALL</code>.
A <code>stage-output</code> describes the logical relation produced by a group or union stage. It is not a physical table and does not prescribe how the result will be stored or scanned.

This combined model keeps related operations together so the physical engine can fuse them into one scan where possible. Splitting every filter, projection, and limit into a separate logical node would introduce artificial materialization boundaries and make later rewrites more expensive.
==Subquery decorrelation==
MemCP does not execute a correlated subquery once for every outer row as a fallback. Supported correlated forms are decorrelated into relational stages. Unsupported shapes fail explicitly instead of silently selecting a slow or semantically weaker execution path.

The planner first tries simple unnesting:
#collect equality classes;
#choose safe representative expressions;
#pull predicates and projections across valid boundaries;
#convert trivial dependent joins into ordinary joins.
If dependencies remain, the planner builds Neumann's '''Domain D''': the duplicate-free projection of the outer values actually read by the dependent subtree. Domain keys are carried through the inner stages and joined back to the outer query. This supports correlated scalar subqueries, <code>EXISTS</code>, <code>IN</code>, grouped subqueries, HAVING, derived tables, and window expressions without per-row recursive SQL execution.

The logical stage also preserves semantic details that must survive lowering:
*scalar subquery cardinality: ordinary relation, first row, or single row with an error on the second row;
*SQL NULL behavior for <code>IN</code> and <code>NOT IN</code>;
*empty correlated aggregate domains, including <code>COUNT = 0</code> versus nullable aggregates;
*outer-join null-extension boundaries;
*ordering, limit, and window requirements.
==Join ordering==
Join ordering is a logical decision with one owner. After decorrelation exposes the complete join graph, <code>join_reorder</code> uses cardinality and selectivity facts to choose a join tree. The search can use exact DPHyp-style enumeration within the configured budget and fall back to bounded strategies for larger join graphs.

The selected tree can be bushy. Physical lowering consumes that tree structurally; it may not flatten the leaves and make a second order decision. Each join node retains:
*its left and right subtrees;
*join kind;
*bound aliases;
*ON-clause ownership;
*null-extension boundary;
*predicates already classified and costed for a leaf.
Independent subtrees may run in parallel. Dependent subtrees remain ordered by their bindings. This execution choice does not change the logical join order.

The exact-search budget can be inspected or changed through:

<code>(settings "JoinReorderDPBudget") (settings "JoinReorderDPBudget" 256)</code>
==Cost-based physical lowering==
The physical lowerer answers a different question from join ordering: given the chosen logical stage and join tree, which concrete storage-engine representation and operator is cheapest while preserving its semantics?

The cost model can consider:
*base-table, stage, and probe cardinality;
*filter selectivity and distinct key counts;
*scan and probe work estimates;
*order compatibility, offset, and limit;
*available and adaptive indexes;
*warm or cold state of reusable group caches;
*expected reuse count;
*build cost versus probe cost;
*memory footprint;
*compile-time cost;
*previous scan and intermediate-relation telemetry.
Depending on these facts, the same logical stage may lower differently:
{| class="wikitable"
|-
!Physical choice
!Typical use
|-
|Fused <code>scan</code>
|Unordered filter, projection, and aggregation in one pass.
|-
|<code>scan_order</code>
|Ordered access, top-k, offset/limit ownership, or a bounded scalar probe.
|-
|<code>scan_order_multi</code>
|Merge already streamable ordered inputs, especially <code>UNION ALL</code>.
|-
|<code>scan_exists</code>
|Stop after proving that at least one matching row exists.
|-
|Direct nested scan
|Cheap selective lookup driven by already bound join keys.
|-
|[[RecSet]]
|Query-local set of matching record IDs for membership and boolean combinations.
|-
|[[RecSet]] projection
|Move a selective candidate set through join keys to another base relation.
|-
|Group keytable
|Reusable grouped key domain with computed aggregate columns.
|-
|FK-backed cached column
|Reuse a foreign-key-shaped lookup or aggregate result on the referencing relation.
|-
|ORC computed column
|Reusable order-dependent computation, including window work with dependency ranges.
|-
|Query-local temporary table
|Last-resort relational barrier when streaming or reusable representations do not fit.
|}Relational results stay inside the storage engine. They are not copied into Scheme lists merely to join, sort, limit, or probe them. Keeping them as scan sources preserves indexing, statistics, batch reads, range braking, late materialization, concurrency, and bounded memory behavior.
==RecSets and boolean predicates==
A RecSet is a query-local set of physical record IDs for one base relation and visibility snapshot. RecSets can represent exact truth sets or safe candidate sets that still require a residual SQL predicate.

For truth-filtering contexts, the planner can use ordinary set identities:

<code>T(false) = empty T(true) = all visible rows T(p OR q) = T(p) union T(q) T(p AND q) = T(p) intersect T(q)</code> Current physical representations include sparse IDs and ranges, with optimized union, intersection, complement, difference, and cross-relation projection. The planner still compares RecSet work with direct scans, indexes, keytables, and ordered drivers; deriving a valid set expression does not force RecSet execution.

SQL three-valued logic remains authoritative. In particular, complementing the TRUE rows of a nullable predicate is not generally equivalent to SQL <code>NOT</code>, and <code>NOT IN</code> must retain its distinction between a match, a NULL probe, and NULL on the right-hand side.
==LIMIT, UNION, and window pipelines==
<code>LIMIT</code> is owned by a physical scan boundary. A plan does not first materialize all qualifying rows into a Scheme list and slice it later. <code>scan_order</code> can own LIMIT even without an explicit <code>ORDER BY</code>, allowing early termination and top-k behavior.

For set operations, the preferred lowering is:
*unordered <code>UNION ALL</code>: emit branches successively;
*unordered <code>UNION</code>: use a deduplication barrier;
*ordered, streamable <code>UNION ALL</code>: merge through <code>scan_order_multi</code>;
*non-streamable ordered unions: materialize only the narrow relation required by the ordering barrier.
A window expression does not automatically force materialization. If one scan order satisfies the base query and every window partition/order, MemCP can fuse the window computation into that ordered scan. ORC or a window stage is used when orders conflict, results are shared, or an order-dependent value must be cached as a computed column.
==Planner cache and guarded specialization==
The query-plan cache and storage-engine group caches are separate:
*The query-plan cache stores compiled Scheme plan formulas for normalized SQL shapes.
*A group cache is a reusable storage-engine relation, normally represented by a group keytable with computed aggregate columns.
For a SELECT whose optimal physical plan depends on parameters or changing statistics, one cache entry may contain several guarded physical variants. A guard records the condition under which its plan remains valid. On a guard miss, MemCP compiles one new variant for the current parameter/statistics regime and places it before older variants.

Guards repeat cost comparisons, not query planning. They do not scan tables, build indexes or caches, or materialize data. Queries without a cost-model tipping decision keep the smaller exact-cache path.
==Inspecting plans with EXPLAIN==
Both MySQL and PostgreSQL syntax modes expose planner diagnostics:

<code>EXPLAIN SELECT ...; EXPLAIN IR SELECT ...; EXPLAIN REORDER SELECT ...; EXPLAIN COMPILE SELECT ...;</code>
*<code>EXPLAIN</code> shows the optimized executable Scheme plan.
*<code>EXPLAIN IR</code> shows the logical representation before physical emission.
*<code>EXPLAIN REORDER</code> exposes join-reordering information and selectivity facts.
*<code>EXPLAIN COMPILE</code> reports compile-phase accounting, including parsing, logical work, reordering, physical preparation, emission, and optimization.
*<code>EXPLAIN PHYSICAL</code> reports plan operator decisions along with query runtime predictions.
Important physical choices should be visible in EXPLAIN output so plan-shape tests can protect them against regressions. Pretty-print width is controlled by:

<code>(settings "ExplainWidth" 80)</code>

For scan and adaptive-index diagnostics, development environments can also use:

<code>(settings "ScanDebugging" true)</code>
==Adaptive indexes and physical plans==
Adaptive indexing belongs to storage access, not logical planning. Scan boundaries describe the desired equality, range, pattern, and ordering prefix. The storage engine can reuse a compatible longer index, avoid creating indexes for tiny shards, accumulate expected savings, and build an index when its cost is amortized.

Main-storage and delta rows participate in ordered index scans. Delta rows are kept in an index-local ordered structure and merged with the compressed main permutation during iteration, so an ordered physical plan does not append delta rows out of order.
==Failure behavior and correctness boundaries==
The planner favors explicit failure over hidden fallback behavior. An unsupported correlated or physical shape should report that limitation. It must not silently:
*execute a correlated subquery once per outer row;
*discard residual predicates;
*turn an equijoin into a cross product;
*treat <code>NOT IN</code> as two-valued logic;
*reduce scalar cardinality checks to <code>LIMIT 1</code>;
*materialize an unbounded relation in a Scheme list;
*choose a second join order during physical lowering.
This rule keeps performance decisions separate from SQL correctness and makes missing optimizer cases visible in tests and issue reports.
==Terminology==
{| class="wikitable"
|-
!Term
!Meaning
|-
|Logical stage
|Semantic query work independent of storage representation.
|-
|Scan source
|A storage-engine representation consumable by a physical scan.
|-
|Intermediate relation
|A relational result materialized as a scan source between plan stages.
|-
|Group cache
|A reusable intermediate relation for group keys and aggregates.
|-
|Group keytable
|The normal physical representation of a group cache.
|-
|ORC
|An order-dependent reusable computed column.
|-
|RecSet
|A query-local set or candidate set of record IDs for one base relation.
|-
|Physical lowering
|Selection of concrete scan sources/operators after logical planning.
|-
|Guarded specialization
|Multiple cost-guarded physical plans for one normalized SELECT shape.
|}
==Further reading==
*[[Data_Auto_Sharding_and_Auto_Indexing|Data Auto Sharding and Auto Indexing]]
*[[Temporary_Computed_Columns|Temporary Computed Columns]]
*[[Columnar_Storage|Columnar Storage]]
*[[Parallel_Computing|Parallel Computing]]
*MemCP source: <code>INVARIANTS.md</code>, <code>lib/queryplan.scm</code>, <code>lib/sql-parser.scm</code>, <code>lib/psql-parser.scm</code>, <code>storage/scan.go</code>, <code>storage/scan_order.go</code>, and <code>storage/recset.go</code>

Persistency and Performance Guarantees

2026-03-09T12:20:19Z

Carli: /* Persistency Guarantees */

MemCP gives the user several Guarantees for persistency and performance. These are the guarantees:

== Performance Guarantees and Read Speed ==
MemCP guarantees that:

* All reading operations are from RAM and are fast
* Insert/Update without any unique key check or foreign key check will scale over shards
* Aggregate functions like SUM, AVG will scale perfectly with the amount of CPUs

With these guarantees, you can add more CPU cores whenever you have more data to ensure the same query time even with growing data.

Please consider that memory bandwith still might be a bottleneck. Even modern 128 core CPUs do not scale better than 8 cores for uncompressed data (usually 40 GiB/s bandwith on tested machines). '''MemCPs [[Columnar Storage]] scales beyond that limit.'''
== Persistency Guarantees and Write Speed ==
There are five persistency modes per table which are:

* ENGINE = cache
* ENGINE = memory
* ENGINE = sloppy
* ENGINE = logged
* ENGINE = safe
In comparison to other RDBMs, MemCP can do this setting per table. MySQL/MariaDB for instance only allows a global setting which is sometimes not what you want.

In short, these engines guarantee different performance:

* cache+memory -> RAM speed, no persistency
* sloppy+logged -> HDD speed, persistency guarantees under circumstances
* safe -> Sync speed (slow) and persistency guarantees even in case of power outage during queries

=== ENGINE = cache ===

* all data is held in memory and only in memory
* in case of a crash, all data is gone
* in case the server is running out of RAM, the data can be deleted, too
* the schema is saved on disk
* after a recovery, the table starts empty
* fastest way to store data
* use it for session data, observer handles, caches and other data that can be recreated by the software

=== ENGINE = memory ===

* all data is held in memory and only in memory
* in case of a crash, all data is gone
* while the server is running, the data will be kept alive
* the schema is saved on disk
* after a recovery, the table starts empty
* fastest way to store data
* use it for session data, observer handles, caches and other data that can be recreated by the software

=== ENGINE = sloppy ===

* all data is held in memory
* the main storage is mirrored on disk
* the delta storage is RAM-only
* a main storage rebuild is triggered every 15 minutes, so data older than 15 minutes are guaranteed to be persistent
* in case of a crash, the delta storage is gone, the main storage is recovered
* after a recovery, some datasets or deletions that happend in the last 15 minutes before the crash may be gone
* extremely fast way to store data without the fear of losing them in normal operation
* use it for frequently updated tables with unimportant data like usage statistics or sensor data

=== ENGINE = logged ===

* all data is held in memory
* the main storage is mirrored on disk
* changes to the delta storage are logged on disk files
* an operation succeeds even if data is not synced to disk permanently yet
* in case of a crash, data might be recovered
** in case of a crash of the process, all data will be recovered
** in case of a power outage or kernel crash, data might be lost
* allows buffering of the log file in return for the risk of data loss
* use it for data where you need a high update performance but cannot afford losing the last few seconds of your work

=== ENGINE = safe ===

* all data is held in memory
* the main storage is mirrored on disk
* changes to the delta storage are logged on disk files
* an operation only succeeds after the data is synced to disk permanently
* in case of a crash, all data will be recovered
* introduces delays for each transaction since the system has to wait for a write fence
* IO bound (limited to ~1,700 write operations per second)
* use it for accounting data or any data that must not be lost

Persistency and Performance Guarantees

2026-03-09T12:16:10Z

Carli: /* Persistency Guarantees */

MemCP gives the user several Guarantees for persistency and performance. These are the guarantees:

== Performance Guarantees ==
MemCP guarantees that:

* Insert/Update without any unique key check or foreign key check will scale over shards
* Aggregate functions like SUM, AVG will scale perfectly with the amount of CPUs

With these guarantees, you can add more CPU cores whenever you have more data to ensure the same query time even with growing data.

Please consider that memory bandwith still might be a bottleneck. Even modern 128 core CPUs do not scale better than 8 cores for uncompressed data (usually 40 GiB/s bandwith on tested machines). '''MemCPs [[Columnar Storage]] scales beyond that limit.'''
== Persistency Guarantees ==
There are five persistency modes per table which are:

* ENGINE = cache
* ENGINE = memory
* ENGINE = sloppy
* ENGINE = logged
* ENGINE = safe
In comparison to other RDBMs, MemCP can do this setting per table. MySQL/MariaDB for instance only allows a global setting which is sometimes not what you want.

In short, these engines guarantee different performance:

* cache+memory -> RAM speed, no persistency
* sloppy+logged -> HDD speed, persistency guarantees under circumstances
* safe -> Sync speed (slow) and persistency guarantees even in case of power outage during queries

=== ENGINE = cache ===

* all data is held in memory and only in memory
* in case of a crash, all data is gone
* in case the server is running out of RAM, the data can be deleted, too
* the schema is saved on disk
* after a recovery, the table starts empty
* fastest way to store data
* use it for session data, observer handles, caches and other data that can be recreated by the software

=== ENGINE = memory ===

* all data is held in memory and only in memory
* in case of a crash, all data is gone
* while the server is running, the data will be kept alive
* the schema is saved on disk
* after a recovery, the table starts empty
* fastest way to store data
* use it for session data, observer handles, caches and other data that can be recreated by the software

=== ENGINE = sloppy ===

* all data is held in memory
* the main storage is mirrored on disk
* the delta storage is RAM-only
* a main storage rebuild is triggered every 15 minutes, so data older than 15 minutes are guaranteed to be persistent
* in case of a crash, the delta storage is gone, the main storage is recovered
* after a recovery, some datasets or deletions that happend in the last 15 minutes before the crash may be gone
* extremely fast way to store data without the fear of losing them in normal operation
* use it for frequently updated tables with unimportant data like usage statistics or sensor data

=== ENGINE = logged ===

* all data is held in memory
* the main storage is mirrored on disk
* changes to the delta storage are logged on disk files
* an operation succeeds even if data is not synced to disk permanently yet
* in case of a crash, data might be recovered
** in case of a crash of the process, all data will be recovered
** in case of a power outage or kernel crash, data might be lost
* allows buffering of the log file in return for the risk of data loss
* use it for data where you need a high update performance but cannot afford losing the last few seconds of your work

=== ENGINE = safe ===

* all data is held in memory
* the main storage is mirrored on disk
* changes to the delta storage are logged on disk files
* an operation only succeeds after the data is synced to disk permanently
* in case of a crash, all data will be recovered
* introduces delays for each transaction since the system has to wait for a write fence
* IO bound (limited to ~1,700 write operations per second)
* use it for accounting data or any data that must not be lost

Persistency and Performance Guarantees

2026-03-09T12:13:49Z

Carli: /* Persistency Guarantees */

MemCP gives the user several Guarantees for persistency and performance. These are the guarantees:

== Performance Guarantees ==
MemCP guarantees that:

* Insert/Update without any unique key check or foreign key check will scale over shards
* Aggregate functions like SUM, AVG will scale perfectly with the amount of CPUs

With these guarantees, you can add more CPU cores whenever you have more data to ensure the same query time even with growing data.

Please consider that memory bandwith still might be a bottleneck. Even modern 128 core CPUs do not scale better than 8 cores for uncompressed data (usually 40 GiB/s bandwith on tested machines). '''MemCPs [[Columnar Storage]] scales beyond that limit.'''
== Persistency Guarantees ==
There are five persistency modes per table which are:

* ENGINE = cache
* ENGINE = memory
* ENGINE = sloppy
* ENGINE = logged
* ENGINE = safe
In comparison to other RDBMs, MemCP can do this setting per table. MySQL/MariaDB for instance only allows a global setting which is sometimes not what you want.

=== ENGINE = cache ===

* all data is held in memory and only in memory
* in case of a crash, all data is gone
* in case the server is running out of RAM, the data can be deleted, too
* the schema is saved on disk
* after a recovery, the table starts empty
* fastest way to store data
* use it for session data, observer handles, caches and other data that can be recreated by the software

=== ENGINE = memory ===

* all data is held in memory and only in memory
* in case of a crash, all data is gone
* while the server is running, the data will be kept alive
* the schema is saved on disk
* after a recovery, the table starts empty
* fastest way to store data
* use it for session data, observer handles, caches and other data that can be recreated by the software

=== ENGINE = sloppy ===

* all data is held in memory
* the main storage is mirrored on disk
* the delta storage is RAM-only
* a main storage rebuild is triggered every 15 minutes, so data older than 15 minutes are guaranteed to be persistent
* in case of a crash, the delta storage is gone, the main storage is recovered
* after a recovery, some datasets or deletions that happend in the last 15 minutes before the crash may be gone
* extremely fast way to store data without the fear of losing them in normal operation
* use it for frequently updated tables with unimportant data like usage statistics or sensor data

=== ENGINE = logged ===

* all data is held in memory
* the main storage is mirrored on disk
* changes to the delta storage are logged on disk files
* an operation succeeds even if data is not synced to disk permanently yet
* in case of a crash, data might be recovered
** in case of a crash of the process, all data will be recovered
** in case of a power outage or kernel crash, data might be lost
* allows buffering of the log file in return for the risk of data loss
* use it for data where you need a high update performance but cannot afford losing the last few seconds of your work

=== ENGINE = safe ===

* all data is held in memory
* the main storage is mirrored on disk
* changes to the delta storage are logged on disk files
* an operation only succeeds after the data is synced to disk permanently
* in case of a crash, all data will be recovered
* introduces delays for each transaction since the system has to wait for a write fence
* IO bound (limited to ~1,700 write operations per second)
* use it for accounting data or any data that must not be lost