Current Status and Open Issues

Status: Beta · Verified against commit e90d6ce7fd on 21 August 2026. MemCP is usable for tested workloads, but SQL compatibility and complex planner cases continue to evolve. Validate every application-critical query, durability requirement and client before a production migration.

This page describes the current implementation and the remaining publicly tracked work. It replaces the old static TODO list from November 2024. Active tasks and priorities belong in the GitHub issue tracker; completed work belongs in the changelog and repository history.

Current status

Area	Status	Notes
SQL frontend	Broad support, Beta compatibility	DDL, DML, joins, transactions, views, triggers, window functions, UNION and extensively tested subqueries are available. MemCP does not claim complete MySQL or PostgreSQL compatibility. See Supported SQL.
Storage and durability	Operational	Per-table safe, logged, sloppy, memory and cache engines are implemented. See Persistency and Performance Guarantees before choosing or changing an engine.
Query planner	New architecture implemented, actively optimized	Logical decorrelation and join ordering are separated from cost-based physical lowering. RecSets, indexes, direct scans, group caches, ORC columns and pipeline variants are physical alternatives. See Query Planner and Physical Lowering.
Transactions and constraints	Implemented for tested semantics	Cursor-stability transactions, an explicit ACID/OCC mode, rollback, unique and NOT NULL constraints, and enforced CREATE TABLE foreign keys are covered by tests. XA and complete MySQL isolation-level compatibility are not claimed.
Operations	Implemented, still maturing	Dashboard, metrics, processlist, query/connection cancellation, graceful shutdown, memory budgets and eviction are available.
Storage backends	Implemented	Local files, S3-compatible object storage and optional Ceph/RADOS are supported. Backend-specific outage, backup and latency behavior must be validated operationally.
RDF/SPARQL	Supported subset	Query, FILTER, OPTIONAL, update forms and Turtle loading are tested. This is not a claim of complete SPARQL standard coverage.
Scheme runtime	Operational	The functional runtime, concurrency primitives and native x86-64 JIT are integrated. Unsupported JIT shapes fall back to interpreted execution.

What changed since the old roadmap

The previous page was a development checklist. The table below preserves every old item and records what happened to it.

Storage engine

Original roadmap item	Current result
ALTER TABLE … ENGINE	Implemented. Tables can transition between the five MemCP engines. Some transitions change durability; persisted → memory permanently removes on-disk data.
LRU garbage collection for temporary columns	Implemented and expanded. Global memory budgets cover persisted columns, indexes, temporary/computed columns, keytables, cache tables and dictionaries. Reloadable data can be evicted; memory-engine rows cannot.
Triggers and change hooks on computed columns	Implemented. Persistent SQL triggers and hidden dependency/invalidation triggers maintain computed columns, group caches, prejoins and order-dependent caches.
Respect foreign keys	Implemented for foreign keys declared with CREATE TABLE. RESTRICT, CASCADE and SET NULL paths are tested for inserts, deletes and key updates. Some ALTER TABLE foreign-key forms remain compatibility-only syntax.
Serialize into memory-mapped database-sized key/value files	Not adopted as the persistence architecture. MemCP retains versioned per-column storage and WAL files and now also supports object backends. Selected storage/JIT implementations use mmap internally, but there is no single 100-GiB database blob contract.
Sort index delta storage and merge it with the main index	Implemented. Ordered iteration merges the compressed main permutation with an ordered index-local delta structure.
Multi-table scan_star/merge-join operator	Superseded by the current planner. Logical join trees are reordered and lowered into costed nested, parallel, indexed, ordered, RecSet or cached physical paths instead of one mandatory combined operator.
Process IDs, request context and kill switch	Implemented. SHOW [FULL] PROCESSLIST, dashboard process controls, KILL QUERY, KILL CONNECTION and cancellation-aware table waits are available.
Transaction insertion/deletion overlays and conflict handling	Implemented and expanded. The storage engine provides cursor-stability undo handling plus snapshot/OCC transaction infrastructure, shard visibility, commit conflict checks, rollback and WAL synchronization at commit.
Indexes for LIKE queries	Implemented. LIKE-prefix and adaptive match-set boundaries participate in scan and physical cost decisions. Residual predicates remain when a boundary is not an exact proof.

RDF frontend

The three former RDF items are implemented and covered by integration tests:

• SPARQL update with DELETE { … } INSERT { … } WHERE { … };
• update templates driven by SELECT/WHERE bindings;
• OPTIONAL { … } with left-join behavior, including unmatched and multiple optional blocks.

RDF support remains a tested subset. Unsupported SPARQL syntax should not be inferred from the implementation of these constructs.

SQL frontend

Original roadmap item	Current result
AUTO_INCREMENT	Implemented and persisted, including restart and ALTER TABLE behavior.
Convert subqueries into joins	Implemented far beyond the original item. The logical planner decorrelates scalar, IN/NOT IN and EXISTS/NOT EXISTS shapes using Neumann-style domains before join reordering and physical lowering.
Shard group tables like their group keys	Implemented as a physical option. Group keytables can be partitioned from group-key information when the cost model selects reusable materialization.
Prejoin complex plans	Implemented as one of several alternatives. Prejoins, FK/PK group reuse, group caches, direct scans and RecSets compete according to semantics and cost.
Restrict users to databases	Implemented. Users, grants, revokes and database access policies are represented in system metadata and enforced by the SQL/API frontends.
Test DBeaver, phpMyAdmin and metadata compatibility	Ongoing compatibility work. MySQL protocol, prepared statements, SHOW metadata and selected INFORMATION_SCHEMA relations are implemented, but individual client releases still require dated compatibility tests.

Scheme language and infrastructure

• Native JIT: implemented. Supported hot Scheme procedures can be compiled to x86-64 machine code, including an expanding set of calls, control flow and planner-generated lambdas. Unsupported procedures continue in the interpreter.
• HTTP/IPFS stream filenames: not generally implemented. Local files and supported virtual archive paths are available; arbitrary http:// and ipfs:// loading should not be documented as supported.
• General native plugin system: not implemented as a stable public API. MemCP has extension hooks such as storage boundary matchers and embedded Scheme modules, but no promised ABI for arbitrary C++/GPU plugins.

Current roadmap

There is currently no published release milestone with a fixed feature schedule. As of the verification date, the public issue tracker contains one open feature issue:

• #9 – Vector extension: helper functions

Next SQL roadmap: JSON compatibility

JSON support is planned as a compatibility and integration feature, not as a replacement for relational modelling. Frequently queried fields should normally remain typed columns with constraints, statistics and indexes. JSON is useful for sparse external attributes, application payloads and migration compatibility; the SQL interface must still make extraction into relational rows and typed values straightforward.

MemCP already has JSONL import, JSON serialization for HTTP and Scheme, and type mapping for MySQL json and PostgreSQL json/jsonb imports. These facilities do not yet constitute MySQL, PostgreSQL or SQL/JSON query compatibility.

Phase	Planned scope	Required semantic decisions
1. JSON value contract	A documented SQL JSON type/storage contract; validation and canonical serialization; JSON_VALID, JSON_TYPE, JSON_QUOTE and JSON_UNQUOTE; casts between text and JSON.	Distinguish SQL NULL, JSON null and a missing path; preserve numeric precision; define duplicate-object-key behavior, invalid-input errors and equality/comparison rules before adding optimizer shortcuts.
2. Paths and read access	MySQL-compatible JSON_EXTRACT, JSON_VALUE, JSON_QUERY, JSON_EXISTS, -> and ->>; JSON_LENGTH, JSON_DEPTH, JSON_KEYS, JSON_CONTAINS, JSON_CONTAINS_PATH and JSON_SEARCH where compatible semantics are defined.	Define one parsed path representation shared by functions and operators. Constant paths should be parsed once, not once per row. Dynamic paths must have explicit cost and error behavior.
3. Construction and aggregation	JSON_OBJECT, JSON_ARRAY, JSON_OBJECTAGG and JSON_ARRAYAGG with deterministic documented handling of NULLs, duplicate keys and aggregate ordering.	Constructors must not silently conflate Scheme association lists, SQL rows and JSON objects. Aggregate results need stable semantics across shards and parallel execution.
4. Mutation	JSON_SET, JSON_INSERT, JSON_REPLACE, JSON_REMOVE, array append/insert functions and merge-patch behavior.	Functions remain pure expressions; an SQL UPDATE writes the resulting JSON value. Partial-update storage optimization may follow later but must not change transactional, trigger or durability semantics.
5. Relational projection	SQL/JSON JSON_TABLE or an equivalent table-producing operator, with typed columns, nested paths, defaults and explicit error/empty behavior.	JSON expansion belongs in the logical relational plan and must participate correctly in joins, correlation, NULL extension, cardinality estimation and cancellation. It must not be hidden as an uncosted per-row fallback.
6. Indexing and physical optimization	Reusable computed columns and adaptive indexes over stable JSON path expressions; statistics for extracted scalar values; batch extraction for scans.	Optimize declared path expressions rather than pretending an opaque document is relationally indexed. Invalidating or updating a JSON document must invalidate every dependent extracted value safely.
7. Dialect compatibility	Map PostgreSQL json/jsonb extraction and containment operators such as ->, ->>, #>, #>> and @> onto the same semantic core where behavior is genuinely compatible.	MySQL, PostgreSQL and SQL/JSON differ in path grammar, scalar return types, ordering, containment and errors. Dialect spellings may share internals, but must not be advertised as aliases when their observable semantics differ.

Definition of done for every JSON feature:

• successful and must-fail SQL tests for SQL NULL, JSON null, missing paths, malformed JSON and malformed paths;
• MySQL and PostgreSQL dialect tests where the feature is advertised for both;
• persistence/restart, trigger, transaction and prepared-statement coverage;
• batch and allocation benchmarks before enabling path extraction in scan hot loops;
• EXPLAIN visibility for table expansion, computed extraction and index use;
• explicit documentation of deviations from MySQL, PostgreSQL or the SQL/JSON standard.

No phase above has a promised release date. Before implementation starts, the agreed compatibility slice should be represented by one or more GitHub issues and tests, rather than treating every vendor-specific JSON function as automatically in scope.

Recent development has concentrated on correctness and cost calibration of the planner, RecSet representations and bulk scans, JIT coverage, shard rebuild concurrency and query cancellation. Git history shows completed changes; it does not constitute a delivery promise for future work.

Areas that continue to need testing and incremental improvement include:

• MySQL/PostgreSQL syntax and metadata compatibility required by real applications;
• rare combinations of correlated subqueries, outer joins, UNION, grouping, windows and SQL three-valued logic;
• transaction contention, long-running rebuilds and crash recovery across engine/backends;
• ORM, administration-tool and connector compatibility by concrete version;
• non-x86-64 platform behavior where native JIT acceleration is unavailable;
• reproducible performance measurements across representative workloads.

These are quality areas, not commitments to a particular release date. Concrete work should be represented by a GitHub issue before this page calls it scheduled.

Reporting an issue

Please open a GitHub issue with:

• MemCP version or commit;
• frontend used: MySQL protocol, /sql, /psql, RDF or embedded Scheme;
• minimal schema, setup data and query;
• expected and actual result;
• table ENGINE and storage backend;
• whether the problem reproduces after a clean restart;
• for performance reports: row counts, EXPLAIN output, hardware, concurrency, cache state, successful response validation and raw samples.

See Contributing for the development and test workflow.