Main Page: Difference between revisions

From MemCP
Jump to navigation Jump to search
No edit summary
 
(4 intermediate revisions by the same user not shown)
Line 1: Line 1:
   
 
=== What is memcp? ===
  
=== What is memcp? ===
  +
[[File:Webapps.svg|left|frameless]]
−
[[https://www.memcp.org/images/0/03/Webapps_with_MemCP.svg]]
  
 
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.
  
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.
   
Line 7: Line 7:
   
   
−
===Features[[File:MemCP Port.png|frameless]]===
 +
===Features===
   
 
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.
  
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.
 
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB
  
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB
 
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs
  
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs
−
*'''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime
 +
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime
−
*Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.
 +
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.
−
*In-memory database: Stores all data in memory, which allows for extremely fast query execution.
 +
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.
 
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)
  
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)
 
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.
  
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.
 
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding
  
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding
−
*Scalability: Designed to scale on a single node with huge NUMA memory
 +
* Scalability: Designed to scale on a single node with huge NUMA memory
 
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time
  
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time
  +
[[File:MemCP Port.png|frameless]]
  +
 
<youtube>g29FR4Jwius</youtube>
  
<youtube>g29FR4Jwius</youtube>
   
Line 30: Line 32:
 
*[[History of the MemCP project]]
  
*[[History of the MemCP project]]
 
*[[Hardware Requirements]]
  
*[[Hardware Requirements]]
−
*[[Persistency and Performance Guarantees]]
 +
*[[Persistency and Performance Guarantees]]
 
*[[Current Status and Open Issues]]
  
*[[Current Status and Open Issues]]
   
−
==== Getting Started====
 +
====Getting Started====
 
*[[Install MemCP with Docker|With Docker]]
  
*[[Install MemCP with Docker|With Docker]]
 
*[[With Singularity]]
  
*[[With Singularity]]
 
*[[Compile MemCP from Source|Build from Source]]
  
*[[Compile MemCP from Source|Build from Source]]
−
*[[Contributing]]
 +
*[[Contributing]]
 
*[[Introduction to Scheme]]
  
*[[Introduction to Scheme]]
   
−
==== Frontends====
 +
====Frontends====
   
 
=====SQL Frontend=====
 
=====SQL Frontend=====
Line 50: Line 52:
 
*[[How SQL Operators are implemented on MemCP]]
  
*[[How SQL Operators are implemented on MemCP]]
   
−
===== RDF Frontend=====
+
=====RDF Frontend=====
 
*[[Introduction to RDF]]
  
*[[Introduction to RDF]]
 
*[[Advanced Graph Querying]]
  
*[[Advanced Graph Querying]]
Line 62: Line 64:
 
====Administration====
  
====Administration====
   
−
*[[Settings]]
 +
* [[Settings]]
 
*[[Process Hibernation]]
  
*[[Process Hibernation]]
 
*[[Performance Measurement]]
  
*[[Performance Measurement]]
Line 73: Line 75:
 
*[[Shards, RecordIDs, Main Storage, Delta Storage]]
  
*[[Shards, RecordIDs, Main Storage, Delta Storage]]
 
*[[Columnar Storage]]
  
*[[Columnar Storage]]
−
*[[Transactions]]
 +
*[[Transactions]]
   
 
=====Optimizations=====
  
=====Optimizations=====
Line 79: Line 81:
 
*[[Temporary Columns]]
  
*[[Temporary Columns]]
 
*[[Data Auto Sharding and Auto Indexing]]
  
*[[Data Auto Sharding and Auto Indexing]]
−
*[[Parallel Computing]]
 +
* [[Parallel Computing]]
   
   
Line 94: Line 96:
 
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]
  
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]
 
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]
  
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]
−
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/
 +
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/
   
 
====How MemCP was built====
  
====How MemCP was built====

Latest revision as of 11:37, 31 October 2024

What is memcp?

Webapps.svg

memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.

memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.


Features

  • fast: MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.
  • efficient: The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB
  • modern: MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs
  • versatile: Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime
  • Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.
  • In-memory database: Stores all data in memory, which allows for extremely fast query execution.
  • Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)
  • OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.
  • Compression: Lots of compression formats are supported like bit-packing and dictionary encoding
  • Scalability: Designed to scale on a single node with huge NUMA memory
  • Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time

MemCP Port.png

https://www.youtube.com/watch?v=g29FR4Jwius

Navigation

Introduction

Getting Started

Frontends

SQL Frontend
RDF Frontend
Custom Frontends

Administration

Internals

How things work in MemCP
Optimizations


Screenshot from htop.png


Further Reading

MemCP on Github

Scientific

How MemCP was built

Retrieved from "https://www.memcp.org/index.php?title=Main_Page&oldid=177"