Main Page

MemCP Database – Fast, Compressed SQL for OLTP and OLAP

MemCP

A fast, compressed, MySQL-protocol-compatible columnar database for modern OLTP and OLAP workloads

MemCP is an open-source SQL database written in Go. It combines persistent in-memory operation, adaptive column compression, parallel query execution and direct application APIs so operational data can be queried and analyzed without maintaining a separate analytical copy.

Beta Open Source MySQL Protocol + HTTP APIs

Get started with Docker Explore SQL support View on GitHub

Development status: Beta. MemCP is under active development. Check Supported SQL, Current Status and Open Issues and the durability requirements of your workload before migrating production data.

Your MySQL queries are too slow? Run the same representative workload on MemCP instead of assuming another index or a larger MySQL server is the only answer. Compare results, durability and authenticated end-to-end latency; if the migration gates pass, move the performance-critical workload to MemCP. Start the MemCP performance evaluation →

Observed performance profile: MemCP has achieved up to 10× the performance of MariaDB/PostgreSQL in OLAP and search-oriented workflows, where RecSets and compressed column scans avoid wide row materialization. Isolated OLTP paths currently take about 1.3–2.0× as long, but in complete WordPress- and wiki-style page builds this has made no significant difference to overall page-loading time in the measured application workflows. Filtered-list queries over roughly one million documents, by contrast, have taken around 30 seconds on PostgreSQL. These are workload observations, not universal guarantees; reproduce them on your data.

What is MemCP?

MemCP Database is a persistent, compressed, column-oriented SQL database for mixed OLTP and OLAP workloads. It speaks the MySQL client protocol, also exposes SQL over HTTP, and executes supported queries through a functional compiler and a parallel column-storage engine.

The name on this site refers to the database project. MemCP Database is not a Model Context Protocol (MCP) memory server and is unrelated to the C/C++ memcpy() memory-copy function. When writing about or linking to the project, the unambiguous name is MemCP Database.

Problems MemCP solves

Slow filtered lists on millions of rows

Complex WHERE, joins, membership tests, ORDER BY and LIMIT can make a row store inspect or materialize far more data than the page returns. MemCP uses RecSets, late materialization, adaptive indexes and ordered braking to keep the working domain compact.
Evaluate the query · How RecSets work

Slow GROUP BY, COUNT and dashboards

Column scans read only referenced compressed values. Parallel shard-local aggregation, group caches and computed structures target repeated analytical queries over fresh operational data.
Columnar Storage · Temporary Computed Columns

Slow database writes caused by fsync

Durability is selectable per table. safe protects commits through power loss; logged has measured about 10× its write throughput when process-crash recovery is sufficient.
Choose a write strategy

Database writes wear out an SD card

For reconstructible edge data, sloppy avoids a continuous WAL and normally publishes one compressed generation every 15 minutes, reducing constant flash writes while making the loss window explicit.
Hardware Requirements · Persistency and Performance Guarantees

Choose your path

Evaluate MemCP

Understand the workload model, current status, hardware needs and differences from MySQL.

OLTP and OLAP · Compare with MySQL · Hardware requirements

Build an application

Use the MySQL protocol, SQL over HTTP, RDF or application-specific endpoints inside MemCP.

SQL tutorial · SQL over HTTP · Client tooling

Operate MemCP

Plan deployment, migration, persistence, storage backends, settings and performance measurement.

Deployment · Durability · Migration

Understand and contribute

Explore the storage engine, optimizer, embedded Scheme runtime and project internals.

Query planner · RecSets · Columnar Storage · Contributing

Quickstart

Run with Docker

<syntaxhighlight lang="bash"> docker run --name memcp \

 -e ROOT_PASSWORD='choose-a-password' \
 -p 4321:4321 -p 3307:3307 \
 -v memcp-data:/data \
 carli2/memcp:latest

</syntaxhighlight>

Then open http://localhost:4321 and connect a MySQL application to 127.0.0.1 on port 3307. The volume mounted at /data retains persistent table data across container replacement. See the complete Docker guide before deploying it as a service.

Build from source

<syntaxhighlight lang="bash"> git clone https://github.com/launix-de/memcp cd memcp go mod download make ./memcp --api-port=4321 --mysql-port=3307 lib/main.scm </syntaxhighlight>

Connect with MySQL tooling:

<syntaxhighlight lang="bash"> mysql -h 127.0.0.1 -u root -p -P 3307

1. Enter the password selected for this data directory.

</syntaxhighlight>

The development default for a fresh data directory is admin. Change it before exposing MemCP to another machine. MemCP can also be supervised with PM2:

<syntaxhighlight lang="bash"> pm2 start ./memcp --name memcp -- \

 --no-repl -data ./data --api-port=4321 --mysql-port=3307 lib/main.scm

</syntaxhighlight>

Background deployments must use --no-repl so that closing standard input does not terminate the interactive console and stop the server. See Build from Source, Deployment and MemCP Console.

Key features

High performance
Parallel, batch-oriented query execution is designed for multicore CPUs, compact working sets and fast persistent storage, serving both OLTP and OLAP workloads.

Columnar storage
Data is stored by column for improved compression, a smaller memory footprint and fast analytical access.

Persistent in-memory operation
MemCP is designed to keep active data in memory while offering configurable per-table durability and persistence backends.

Fast configurable writes
Choose durability per table: safe protects committed data through power loss, while measured logged paths have reached about 10× its write throughput; sloppy batches reconstructible data into the normal 15-minute rebuild cycle.

Built-in APIs
SQL over HTTP and in-database services can remove an extra middleware hop for suitable applications.

Adaptive compression
Bit-packing, dictionary encoding and sequence compression reduce the bytes read for suitable data. Historical imports have reached reductions around 80% versus their MySQL/MariaDB representation; measure your own schema.

Simple deployment
Start with Docker, PM2 or the native binary. The compact application has historically had an installation footprint of approximately 10 MB.

Extensible frontends
The Go storage engine and embedded Scheme environment support SQL, RDF, REST and custom application interfaces.

Cost-based query execution
Logical decorrelation and join optimization are separated from cost-based physical selection of scans, indexes, RecSets, reusable caches and execution pipelines.
How MemCP plans and lowers queries → · How compact record sets process large domains →

Why MemCP?

Traditional relational databases were designed around spinning disks and comparatively small numbers of CPU cores. MemCP rethinks storage and query execution for modern multicore systems, large caches, fast storage and mixed workloads.

Typical use cases include:

• real-time dashboards and analytics;
• data-heavy SaaS platforms;
• embedded systems with limited resources;
• high-throughput OLTP/OLAP hybrids.

Whether MemCP is a good fit depends on the required SQL compatibility, durability, query mix, data size and operational environment. Review Supported SQL, Persistency and Performance Guarantees and Performance Measurement rather than treating benchmark figures as universal guarantees.

MemCP vs. MySQL

Feature	MySQL	MemCP
Storage model	Primarily row-based	Column-based and compressed
Performance focus	General-purpose relational workloads	Parallel, in-memory-oriented execution for mixed operational and analytical workloads
In-memory capability	Available through selected engines and caching	Central design goal and default operating model
REST API integration	Normally external	Built in
Installation footprint	Common server installations are approximately 150 MB or larger	The native application has historically been approximately 10 MB
Open source	✅	✅

MemCP provides MySQL protocol compatibility but does not claim to implement every MySQL feature. See Comparison: MemCP vs. MySQL for a detailed comparison and Database Tools compatibility with MemCP for client compatibility.

Architecture overview

• Tables, schemas and columns: Familiar SQL structures use a compressed columnar physical layout. See Databases, Tables and Columns and Columnar Storage.
• Transaction model: Delta and main storage support mixed OLTP and OLAP semantics. See Transactions and Isolation and Shards, RecordIDs, Main Storage, Delta Storage.
• Query planning: Logical optimization is separated from physical execution decisions. See Query Planner and Physical Lowering.
• Persistence: Per-table durability can use filesystem, S3 or Ceph/RADOS storage. See Persistency and Performance Guarantees and Storage Backends.
• Operations: The dashboard, metrics, process inspection and memory controls support day-to-day operation. See Dashboard and Operations and Memory Management and Eviction.
• Scale-out roadmap: MemCP is single-node today. A leaderless CRUSH/RADOS and MOESI-inspired cluster design is planned, but must not be treated as an available production feature. See Cluster Monitor.
• Frontends: MemCP provides multiple query and application interfaces:
  • SQL through the MySQL wire protocol and SQL over HTTP;
  • an RDF/graph query frontend;
  • custom APIs through in-database web applications.

Documentation

The following navigation remains on the start page so users and search engines can reach every major documentation area directly.

Introduction and evaluation

• What is OLTP and OLAP
• History of the MemCP project
• Hardware Requirements
• Persistency and Performance Guarantees
• Current Status and Open Issues
• SQL performance problems MemCP solves
• Comparison: MemCP vs. MySQL

Getting started

• Install with Docker
• Install with Singularity/Apptainer
• Build from Source
• Contributing
• Introduction to Scheme
• Full SCM API documentation

Administration

• Deployment
• Migration from MySQL and PostgreSQL
• Settings
• Security and Authentication
• Dashboard and Operations
• Memory Management and Eviction
• Process Hibernation
• Performance Measurement
• MemCP Console

Frontends

SQL frontend

• Supported SQL
• Advanced SQL Tutorial
• JSON and SQL/JSON
• Triggers
• SQL over REST
• Supported Tooling
• How SQL operators are implemented
• Add custom SQL operators to MemCP

RDF frontend

• Introduction to RDF
• Advanced Graph Querying
• RDF browser and templating example

Custom frontends

• In-Database WebApps and REST Services
• MemCP for Microservices
• Websockets in MemCP

Persistence backends and storage

• File System
• S3-compatible and Ceph/RADOS storage
• Cluster Monitor

Internals

How MemCP works

• Databases, Tables and Columns
• Shards, RecordIDs, Main Storage, Delta Storage
• Columnar Storage
• Transactions and Isolation
• Query Planner and Physical Lowering
• Full SCM API documentation

Scheme documentation

• SCM Builtins
• Arithmetic / Logic
• Strings
• Streams
• Lists
• Associative Lists / Dictionaries
• Date
• Vectors
• Parsers
• Sync
• IO
• Storage

Optimizations

• Query Planner and Physical Lowering
• In-Memory Compression, Columnar Compression Techniques
• Temporary Computed Columns
• Data Auto Sharding and Auto Indexing
• Parallel Computing

Further reading

• MemCP on GitHub
• VLDB Research Paper
• LNI Proceedings Paper
• Large Graph Algorithms

Additional blog posts on design decisions, compression techniques and performance optimization are available on the Launix blog.

Community

MemCP is an open-source project maintained by developers for developers. Contributions are welcome in the form of bug reports, feature requests and pull requests.

See Contributing and the GitHub repository.