Main Page

MemCP – A Modern In-Memory Columnar Database

MemCP

A fast, compressed, MySQL-compatible columnar database for modern OLTP and OLAP workloads

MemCP combines persistent in-memory operation, adaptive column compression, parallel query execution and direct application APIs in a lightweight database written in Go.

Beta Open Source MySQL Protocol + HTTP APIs

Get started with Docker Explore SQL support View on GitHub

Development status: Beta. MemCP is under active development. Check Supported SQL, Current Status and Open Issues and the durability requirements of your workload before migrating production data.

Choose your path

Evaluate MemCP

Understand the workload model, current status, hardware needs and differences from MySQL.

OLTP and OLAP · Compare with MySQL · Hardware requirements

Build an application

Use the MySQL protocol, SQL over HTTP, RDF or application-specific endpoints inside MemCP.

SQL tutorial · SQL over HTTP · Client tooling

Operate MemCP

Plan deployment, migration, persistence, storage backends, settings and performance measurement.

Deployment · Durability · Migration

Understand and contribute

Explore the storage engine, optimizer, embedded Scheme runtime and project internals.

Query planner · Columnar Storage · Contributing

Quickstart

Run with Docker

<syntaxhighlight lang="bash"> docker run --name memcp \

 -e ROOT_PASSWORD='choose-a-password' \
 -p 4321:4321 -p 3307:3307 \
 carli2/memcp:latest

</syntaxhighlight>

Then open http://localhost:4321 and connect a MySQL application to 127.0.0.1 on port 3307. Add a volume mounted at /data when the container must retain persistent data. See the complete Docker guide before deploying it as a service.

Build from source

<syntaxhighlight lang="bash"> git clone https://github.com/launix-de/memcp cd memcp go mod download make ./memcp --api-port=4321 --mysql-port=3307 lib/main.scm </syntaxhighlight>

Connect with MySQL tooling:

<syntaxhighlight lang="bash"> mysql -h 127.0.0.1 -u root -p -P 3307

1. Enter the password selected for this data directory.

</syntaxhighlight>

The development default for a fresh data directory is admin. Change it before exposing MemCP to another machine. MemCP can also be supervised with PM2:

<syntaxhighlight lang="bash"> pm2 start ./memcp --name memcp -- \

 --no-repl -data ./data --api-port=4321 --mysql-port=3307 lib/main.scm

</syntaxhighlight>

Background deployments must use --no-repl so that closing standard input does not terminate the interactive console and stop the server. See Build from Source, Deployment and MemCP Console.

Key features

High performance
NUMA-aware, parallelized query execution is optimized for multicore CPUs, large caches and NVMe SSDs, serving both OLTP and OLAP workloads.

Columnar storage
Data is stored by column for improved compression, a smaller memory footprint and fast analytical access.

Persistent in-memory operation
MemCP is designed to keep active data in memory while offering configurable per-table durability and persistence backends.

Built-in APIs
SQL over HTTP and in-database services can remove an extra middleware hop for suitable applications.

Adaptive compression
Bit-packing, dictionary encoding and sequence compression can reduce suitable datasets by up to 80% compared with their MySQL/MariaDB representation.

Simple deployment
Start with Docker, PM2 or the native binary. The compact application has historically had an installation footprint of approximately 10 MB.

Extensible frontends
The Go storage engine and embedded Scheme environment support SQL, RDF, REST and custom application interfaces.

Cost-based query execution
Logical decorrelation and join optimization are separated from cost-based physical selection of scans, indexes, RecSets, reusable caches and execution pipelines.
How MemCP plans and lowers queries →

Why MemCP?

Traditional relational databases were designed around spinning disks and comparatively small numbers of CPU cores. MemCP rethinks storage and query execution for modern multicore systems, large caches, fast storage and mixed workloads.

Typical use cases include:

• real-time dashboards and analytics;
• data-heavy SaaS platforms;
• embedded systems with limited resources;
• high-throughput OLTP/OLAP hybrids.

Whether MemCP is a good fit depends on the required SQL compatibility, durability, query mix, data size and operational environment. Review Supported SQL, Persistency and Performance Guarantees and Performance Measurement rather than treating benchmark figures as universal guarantees.

MemCP vs. MySQL

Feature	MySQL	MemCP
Storage model	Primarily row-based	Column-based and compressed
Performance focus	General-purpose relational workloads	NUMA-aware, in-memory execution for mixed operational and analytical workloads
In-memory capability	Available through selected engines and caching	Central design goal and default operating model
REST API integration	Normally external	Built in
Installation footprint	Common server installations are approximately 150 MB or larger	The native application has historically been approximately 10 MB
Open source	✅	✅

MemCP provides MySQL protocol compatibility but does not claim to implement every MySQL feature. See Comparison: MemCP vs. MySQL for a detailed comparison and Database Tools compatibility with MemCP for client compatibility.

Architecture overview

• Tables, schemas and columns: Familiar SQL structures use a compressed columnar physical layout. See Databases, Tables and Columns and Columnar Storage.
• Transaction model: Delta and main storage support mixed OLTP and OLAP semantics. See Transactions and Isolation and Shards, RecordIDs, Main Storage, Delta Storage.
• Query planning: Logical optimization is separated from physical execution decisions. See Query Planner and Physical Lowering.
• Persistence: Per-table durability can use filesystem, S3 or Ceph/RADOS storage. See Persistency and Performance Guarantees.
• Frontends: MemCP provides multiple query and application interfaces:
  • SQL through the MySQL wire protocol and SQL over HTTP;
  • an RDF/graph query frontend;
  • custom APIs through in-database web applications.

Documentation

The following navigation remains on the start page so users and search engines can reach every major documentation area directly.

Introduction and evaluation

• What is OLTP and OLAP
• History of the MemCP project
• Hardware Requirements
• Persistency and Performance Guarantees
• Current Status and Open Issues
• Comparison: MemCP vs. MySQL

Getting started

• Install with Docker
• Install with Singularity/Apptainer
• Build from Source
• Contributing
• Introduction to Scheme
• Full SCM API documentation

Administration

• Deployment
• Migration from MySQL and PostgreSQL
• Settings
• Process Hibernation
• Performance Measurement
• MemCP Console

Frontends

SQL frontend

• Supported SQL
• Advanced SQL Tutorial
• SQL over REST
• Supported Tooling
• How SQL Operators are implemented on MemCP
• Add custom SQL operators to MemCP

RDF frontend

• Introduction to RDF
• Advanced Graph Querying
• RDF templating and model driven development

Custom frontends

• In-Database WebApps and REST Services
• MemCP for Microservices
• Websockets in MemCP

Persistence backends and storage

• File System
• S3 Buckets
• Ceph/Rados
• Cluster Monitor

Internals

How MemCP works

• Databases, Tables and Columns
• Shards, RecordIDs, Main Storage, Delta Storage
• Columnar Storage
• Transactions and Isolation
• Query Planner and Physical Lowering
• Full SCM API documentation

Scheme documentation

• SCM Builtins
• Arithmetic / Logic
• Strings
• Streams
• Lists
• Associative Lists / Dictionaries
• Date
• Vectors
• Parsers
• Sync
• IO
• Storage

Optimizations

• Query Planner and Physical Lowering
• In-Memory Compression, Columnar Compression Techniques
• Temporary Columns
• Data Auto Sharding and Auto Indexing
• Parallel Computing

Further reading

• MemCP on GitHub
• VLDB Research Paper
• LNI Proceedings Paper
• Large Graph Algorithms

Additional blog posts on design decisions, compression techniques and performance optimization are available on the Launix blog.

Community

MemCP is an open-source project maintained by developers for developers. Contributions are welcome in the form of bug reports, feature requests and pull requests.

See Contributing and the GitHub repository.