MemCP memcp_wiki https://www.memcp.org/wiki/Main_Page MediaWiki 1.39.1 first-letter Media Special Talk User User talk MemCP MemCP talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk 0 2 259 256 2026-02-27T02:21:08Z Carli 3 259 wikitext text/x-wiki = MemCP – A Modern In-Memory Columnar Database = '''MemCP is a high-performance, in-memory, column-oriented database designed for modern workloads.''' It provides a lightweight, developer-friendly alternative to traditional relational databases such as MySQL, with a focus on speed, compression, and direct API integration. [[File:Memcp-Load.png|center|frameless|1600x1600px]] ---- == Key Features == * '''High Performance''': NUMA-aware, parallelized query execution optimized for multicore CPUs, large caches, and NVMe SSDs. Handles both OLTP and OLAP workloads efficiently. * '''Columnar Storage''': Data is stored by column for improved compression, reduced memory footprint, and faster analytical queries. * '''In-Memory Operation''': Designed to keep data in memory, with configurable persistence backends for durability. * '''Built-in APIs''': Exposes RESTful endpoints directly from the database, reducing middleware overhead. * '''Compression''': Multiple strategies (bit-packing, dictionary encoding, sequence compression) reduce storage by up to 80% compared to MySQL/MariaDB. * '''Simple Deployment''': Start with a single <code>docker run</code> or <code>pm2 start</code> command. Lightweight footprint (~10MB). * '''Extensible''': Written in Go, with pluggable storage backends and custom frontend support (SQL, RDF, REST). ---- == Why MemCP? == Traditional relational databases were designed decades ago, optimized for spinning disks and single-core CPUs. MemCP rethinks the core design for today’s hardware and workloads: * Real-time dashboards and analytics * Data-heavy SaaS platforms * Embedded systems with limited resources * High-throughput OLTP/OLAP hybrids ---- == Quick Start == Clone and build MemCP from source: <pre> git clone https://github.com/launix-de/memcp cd memcp go get make pm2 start ./memcp ./data/ </pre> Connect with MySQL tooling: <pre> mysql -u root -p -P 3307 Enter password: admin </pre> ---- == MemCP vs. MySQL == {| class="wikitable" ! Feature ! MySQL ! MemCP |- | Storage Model | Row-based | Column-based (compressed) |- | Performance | Good | NUMA-optimized, in-memory |- | In-Memory Capable | Limited | Yes (default) |- | REST API Integration | External | Built-in |- | Installation Footprint | ~150MB+ | ~10MB |- | Open Source | ✅ | ✅ |} ---- == Architecture Overview == * '''Tables, Schemas, Columns''': Familiar SQL-style structures with a columnar physical layout. * '''Transaction Model''': Supports both OLTP and OLAP semantics with delta + main storage. * '''Persistence''': Configurable storage backends (filesystem, S3, Ceph). * '''Frontends''': Multiple query interfaces: - SQL frontend (MySQL wire protocol + SQL over REST) - RDF/graph query engine - Custom APIs via in-database web apps ---- == Documentation == * [[What is OLTP and OLAP]] * [[History of the MemCP project]] * [[Hardware Requirements]] * [[Persistency and Performance Guarantees]] * [[Comparison: MemCP vs. MySQL]] * [[Install MemCP with Docker|Install with Docker]] * [[Compile MemCP from Source|Build from Source]] * [[Contributing]] * [[SQL over REST]] * [[In-Database WebApps|REST & Microservices]] ---- ===Navigation=== ====Introduction==== *[[What is OLTP and OLAP]] *[[History of the MemCP project]] *[[Hardware Requirements]] *[[Persistency and Performance Guarantees]] *[[Current Status and Open Issues]] *[[Comparison: MemCP vs. MySQL]] ====Getting Started==== *[[Install MemCP with Docker|With Docker]] *[[With Singularity]] *[[Compile MemCP from Source|Build from Source]] *[[Contributing]] *[[Introduction to Scheme]] *[[Full SCM API documentation]] ====Administration==== * [[Deployment]] * [[Migration from MySQL and PostgreSQL]] * [[Settings]] *[[Process Hibernation]] *[[Performance Measurement]] *[[MemCP Console]] *[[Performance Measurement]] ====Frontends==== =====SQL Frontend===== *[[Supported SQL]] *[[Advanced SQL Tutorial]] *[[SQL over REST]] *[[Database Tools compatibility with MemCP|Supported Tooling]] *[[How SQL Operators are implemented on MemCP]] *[[Add custom SQL operators to MemCP]] =====RDF Frontend===== *[[Introduction to RDF]] *[[Advanced Graph Querying]] *[[RDF templating and model driven development]] =====Custom Frontends===== *[[In-Database WebApps|In-Database WebApps and REST Services]] *[[MemCP for Microservices]] *[[Websockets in MemCP]] ==== Persistency Backends (= Storage) ==== * [[File System]] * [[S3 Buckets]] * [[Ceph/Rados]] * [[Cluster Monitor]] ====Internals==== =====How things work in MemCP===== *[[Databases, Tables and Columns]] *[[Shards, RecordIDs, Main Storage, Delta Storage]] *[[Columnar Storage]] *[[Transactions]] *[[Full SCM API documentation]] ===== SCM Documentation ===== * [[SCM Builtins]] * [[Arithmetic / Logic]] * [[Strings]] * [[Streams]] * [[Lists]] * [[Associative Lists / Dictionaries]] * [[Date]] * [[Vectors]] * [[Parsers]] * [[Sync]] * [[IO]] * [[Storage]] =====Optimizations===== *[[In-Memory Compression, Columnar Compression Techniques]] *[[Temporary Columns]] *[[Data Auto Sharding and Auto Indexing]] * [[Parallel Computing]] ---- == Further Reading == * [https://github.org/launix-de/memcp MemCP on GitHub] * [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper] * [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper] * [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms] Additional blog posts on design decisions, compression techniques, and performance optimization are available on the [https://launix.de/launix/ Launix blog]. ---- == Community == MemCP is an open-source project maintained by developers for developers. Contributions are welcome — whether in the form of bug reports, feature requests, or pull requests. See: [[Contributing]] 765drhjl97ynk40u51037uptkrtx7ns