Main Page

From MemCP
Jump to navigation Jump to search

MemCP – A Modern In-Memory Columnar Database

MemCP
A fast, compressed, MySQL-compatible columnar database for modern OLTP and OLAP workloads
MemCP combines persistent in-memory operation, adaptive column compression, parallel query execution and direct application APIs in a lightweight database written in Go.
Beta Open Source MySQL Protocol + HTTP APIs
Get started with Docker Explore SQL support View on GitHub
MemCP database dashboard and workload overview

Development status: Beta. MemCP is under active development. Check Supported SQL, Current Status and Open Issues and the durability requirements of your workload before migrating production data.

Choose your path

Evaluate MemCP

Understand the workload model, current status, hardware needs and differences from MySQL.

OLTP and OLAP · Compare with MySQL · Hardware requirements

Build an application

Use the MySQL protocol, SQL over HTTP, RDF or application-specific endpoints inside MemCP.

SQL tutorial · SQL over HTTP · Client tooling

Operate MemCP

Plan deployment, migration, persistence, storage backends, settings and performance measurement.

Deployment · Durability · Migration

Understand and contribute

Explore the storage engine, optimizer, embedded Scheme runtime and project internals.

Query planner · Columnar Storage · Contributing

Quickstart

Run with Docker

<syntaxhighlight lang="bash"> docker run --name memcp \ 

 -e ROOT_PASSWORD='choose-a-password' \
 -p 4321:4321 -p 3307:3307 \
 carli2/memcp:latest

</syntaxhighlight>

Then open http://localhost:4321 and connect a MySQL application to 127.0.0.1 on port 3307. Add a volume mounted at /data when the container must retain persistent data. See the complete Docker guide before deploying it as a service.

Build from source

<syntaxhighlight lang="bash"> git clone https://github.com/launix-de/memcp cd memcp go mod download make ./memcp --api-port=4321 --mysql-port=3307 lib/main.scm </syntaxhighlight>

Connect with MySQL tooling:

<syntaxhighlight lang="bash"> mysql -h 127.0.0.1 -u root -p -P 3307

  1. Enter the password selected for this data directory.

</syntaxhighlight>

The development default for a fresh data directory is admin. Change it before exposing MemCP to another machine. MemCP can also be supervised with PM2:

<syntaxhighlight lang="bash"> pm2 start ./memcp --name memcp -- \ 

 --no-repl -data ./data --api-port=4321 --mysql-port=3307 lib/main.scm

</syntaxhighlight>

Background deployments must use --no-repl so that closing standard input does not terminate the interactive console and stop the server. See Build from Source, Deployment and MemCP Console.

Key features

High performance
NUMA-aware, parallelized query execution is optimized for multicore CPUs, large caches and NVMe SSDs, serving both OLTP and OLAP workloads.
Columnar storage
Data is stored by column for improved compression, a smaller memory footprint and fast analytical access.
Persistent in-memory operation
MemCP is designed to keep active data in memory while offering configurable per-table durability and persistence backends.
Built-in APIs
SQL over HTTP and in-database services can remove an extra middleware hop for suitable applications.
Adaptive compression
Bit-packing, dictionary encoding and sequence compression can reduce suitable datasets by up to 80% compared with their MySQL/MariaDB representation.
Simple deployment
Start with Docker, PM2 or the native binary. The compact application has historically had an installation footprint of approximately 10 MB.
Extensible frontends
The Go storage engine and embedded Scheme environment support SQL, RDF, REST and custom application interfaces.
Cost-based query execution
Logical decorrelation and join optimization are separated from cost-based physical selection of scans, indexes, RecSets, reusable caches and execution pipelines.
How MemCP plans and lowers queries →

Why MemCP?

Traditional relational databases were designed around spinning disks and comparatively small numbers of CPU cores. MemCP rethinks storage and query execution for modern multicore systems, large caches, fast storage and mixed workloads.

Typical use cases include:

  • real-time dashboards and analytics;
  • data-heavy SaaS platforms;
  • embedded systems with limited resources;
  • high-throughput OLTP/OLAP hybrids.

Whether MemCP is a good fit depends on the required SQL compatibility, durability, query mix, data size and operational environment. Review Supported SQL, Persistency and Performance Guarantees and Performance Measurement rather than treating benchmark figures as universal guarantees.

MemCP vs. MySQL

Feature MySQL MemCP
Storage model Primarily row-based Column-based and compressed
Performance focus General-purpose relational workloads NUMA-aware, in-memory execution for mixed operational and analytical workloads
In-memory capability Available through selected engines and caching Central design goal and default operating model
REST API integration Normally external Built in
Installation footprint Common server installations are approximately 150 MB or larger The native application has historically been approximately 10 MB
Open source ✅ ✅

MemCP provides MySQL protocol compatibility but does not claim to implement every MySQL feature. See Comparison: MemCP vs. MySQL for a detailed comparison and Database Tools compatibility with MemCP for client compatibility.

Architecture overview

Documentation

The following navigation remains on the start page so users and search engines can reach every major documentation area directly.

Introduction and evaluation

Getting started

Administration

Frontends

SQL frontend

RDF frontend

Custom frontends

Persistence backends and storage

Internals

How MemCP works

Scheme documentation

Optimizations

Further reading

Additional blog posts on design decisions, compression techniques and performance optimization are available on the Launix blog.

Community

MemCP is an open-source project maintained by developers for developers. Contributions are welcome in the form of bug reports, feature requests and pull requests.

See Contributing and the GitHub repository.

Retrieved from "https://www.memcp.org/index.php?title=Main_Page&oldid=271"