User contributions for Carli
Jump to navigation
Jump to search
17 May 2024
- 22:3222:32, 17 May 2024 diff hist 0 In-Memory Compression, Columnar Compression Techniques No edit summary Tag: Visual edit
- 22:3122:31, 17 May 2024 diff hist +332 In-Memory Compression, Columnar Compression Techniques No edit summary Tag: Visual edit
- 22:2822:28, 17 May 2024 diff hist +7,746 In-Memory Compression, Columnar Compression Techniques No edit summary Tag: Visual edit
- 22:2622:26, 17 May 2024 diff hist +164 Databases, Tables and Columns No edit summary current Tag: Visual edit
- 22:2422:24, 17 May 2024 diff hist +2,922 N Current Status and Open Issues Created page with "There are several TODOs in MemCP still. These are categorized as follows: === Storage Engine === * Allow ALTER TABLE ENGINE = ... * Garbage Collection with an LRU policy on temporary columns * Triggers and change hooks on computed columns * Respect Foreign Keys * Serialize and Deserialize into MMapped big files (these bigfiles must be organized as key-value stores) * iterateIndex: sort delta storage and merge it with main, so that correct order is always guaranteed * m..." Tag: Visual edit
- 19:0819:08, 17 May 2024 diff hist +37 Main Page →Introduction Tag: Visual edit
- 19:0719:07, 17 May 2024 diff hist +7,345 N Index Compression Created page with "== Memory-Efficient Indices for In-Memory Storages == Most databases implement indices as a kind of tree. I will show you that columnar storages can do even better. The most widely used kind of tree structure in databases is the B-Tree. A B-Tree is a n-ary tree whose nodes fit exactly into one „page“ – may it be a cache line of 64 bytes or a HDD page of 512 bytes or a memory page of 4K. The advantage of btrees is that it balances the memory fetch time with the ti..." current Tag: Visual edit
- 19:0419:04, 17 May 2024 diff hist +74 In-Memory Compression, Columnar Compression Techniques No edit summary Tag: Visual edit
- 17:1817:18, 17 May 2024 diff hist +23 Main Page No edit summary Tag: Visual edit
- 17:1517:15, 17 May 2024 diff hist +212 N Compile MemCP from Source Created page with "Make sure, <code>go</code> is installed on your computer. (at least 1.22.0) Compile the project with: git clone <nowiki>https://github.com/launix-de/memcp</nowiki> cd memcp make Run the engine with: ./memcp" current Tag: Visual edit
- 17:1317:13, 17 May 2024 diff hist +441 N With Singularity Created page with "Singularity is a rootless container format for restricted environments like HPC. Here are the installation instructions: https://docs.sylabs.io/guides/latest/user-guide/quick_start.html#quick-installation-steps Build with: git clone <nowiki>https://github.com/launix-de/memcp</nowiki> cd memcp sudo singularity build memcp.sif memcp.singularity.recipe Now you can run memcp.sif: mkdir data singularity run --bind data:/data memcp.sif" current Tag: Visual edit
- 17:1017:10, 17 May 2024 diff hist +660 N Install MemCP with Docker Created page with "To build the image the first time: git clone https://github.com/launix-de/memcp cd memcp docker build . -t memcp For app development and to enter the Scheme console: mkdir data docker run -v data:/data -it -p 4321:4321 -p 3307:3307 memcp To run your custom Scheme apps (like the one from RDF templating and model driven development): make data docker run -e PARAMS="lib/main.scm apps/minigame.scm" -v data:/data -it -p 4321:4321 -p 3307:3307 memcp During productio..." current Tag: Visual edit
- 17:0617:06, 17 May 2024 diff hist +2,427 N Persistency and Performance Guarantees Created page with "MemCP gives the user several Guarantees for persistency and performance. These are the guarantees: == Persistency Guarantees == There are three persistency modes per table which are: * ENGINE = memory * ENGINE = sloppy * ENGINE = logged * ENGINE = safe === ENGINE = memory === * all data is held in memory and only in memory * in case of a crash, all data is gone * the schema is saved on disk * after a recovery, the table starts empty * fastest way to store data * use..." Tag: Visual edit
- 16:3216:32, 17 May 2024 diff hist +451 Lists and Objects No edit summary current Tag: Visual edit
- 16:2716:27, 17 May 2024 diff hist +1,805 N Lists and Objects Created page with "Lists are a powerful feature in Scheme. Lists can express both: Data and Programs. ==List handling== The following list functions can be used: > '(1 2 3) "list literal" = (1 2 3) > (cons 0 '(1 2 3)) "prepend item to list" = (0 1 2 3) > (append '(1 2 3) 4 5) "append items to list" = (1 2 3 4 5) > (has? '(1 2 3) 2) "check if item is in list" = true > (filter '(1 2 3) (lambda (x) (< x 2))) "filter items that are smaller than 2" = (1) > (map '(1 2 3) (l..." Tag: Visual edit
- 16:2616:26, 17 May 2024 diff hist +12 Introduction to Scheme →Deeper Topics into Scheme current Tag: Visual edit
- 16:2516:25, 17 May 2024 diff hist −1,600 Introduction to Scheme No edit summary Tag: Visual edit
- 16:2416:24, 17 May 2024 diff hist +1,740 Introduction to Scheme No edit summary Tag: Visual edit
- 15:5115:51, 17 May 2024 diff hist +46 Main Page →Navigation Tag: Visual edit
- 15:3615:36, 17 May 2024 diff hist +10,258 N Introduction to Scheme Created page with "When you run <code>./memcp</code>, you will be dropped at a scheme shell like this: memcp Copyright (C) 2023, 2024 Carl-Philip Hänsch This program comes with ABSOLUTELY NO WARRANTY; This is free software, and you are welcome to redistribute it under certain conditions; loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-id of type 10 loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-username of type 20 loading stora..." Tag: Visual edit
- 15:3515:35, 17 May 2024 diff hist +50 Main Page No edit summary Tag: Visual edit
- 15:0515:05, 17 May 2024 diff hist +30 Main Page →Navigation Tag: Visual edit
- 15:0415:04, 17 May 2024 diff hist +3,441 N Sequence Compression Created page with "One of the most interesting compression techniques on columnar storages is '''sequence compression'''. Sequence Compression in In-Memory Database yields 99% memory savings and a total of 13% A sequence is a column of numbers where each distance between two neighbouring numbers is equal. Example: * <code>1 2 3 4 5 6 7 8 9</code> * <code>3 3 3 3 3 3 3 3 3 3</code> * <code>10 20 30 40</code> * <code>8 7 6 5</code> A sequence can be described by its starting point, its st..." current Tag: Visual edit
- 15:0115:01, 17 May 2024 diff hist +398 N In-Memory Compression, Columnar Compression Techniques Created page with "The following compression techniques are implemented: * Sparse Storage for columns with lots of NULL values * Float Storage for scientific data * Integer Compression with a bit-size approach * Sequence Compression for sequences of integers like IDs (1, 2, 3, 4, 5 ...) * Dictionary Compression for short strings * BLOB Zipping and Deduplication for strings longer than 1KiB" Tag: Visual edit
- 14:5714:57, 17 May 2024 diff hist +4,408 N Columnar Storage Created page with "The advantages for columnar storages over row based storages are the ability for good in-memory compression '''(low memory usage)''' and cache locality when accessing only few columns '''(performance)'''. == How we designed the Interface == When designing an interface for a storage engine for an in-memory database, a lot of considerations have to be made. Here are the design goals: * The interface must be simple so that using it does not require implementing all kinds..." current Tag: Visual edit
- 14:5314:53, 17 May 2024 diff hist +4 Main Page No edit summary Tag: Visual edit
- 14:5314:53, 17 May 2024 diff hist +20 Main Page No edit summary Tag: Visual edit
- 14:5214:52, 17 May 2024 diff hist +432 N Hardware Requirements Created page with "MemCP can run on any hardware that is supported by the <code>go</code> compiler. To find the right hardware dimensions, you need to consider the following: * The process itself needs at least 8 MiB of RAM * The size of a 5 GiB MySQL database fits into ~1 GiB of RAM * To have a decent working set, take the RAM requirements x3, so a 5 GiB MySQL database should run on 3 GiB RAM * MemCP needs at least 1 CPU core, more is better ;)" Tag: Visual edit
- 13:5113:51, 17 May 2024 diff hist +3,769 N Parallel Computing Created page with "Almost 99% of all newly invented are imperative programming languages. But imperative languages have one drawback: their parallelization is hard. == Drawbacks of Imperative Programming Languages == Imperative programming languages do have one mayor drawback: state. The concept of an imperative language is that commands are executed which change the content of variables or complex objects in the memory. When trying to create an optimizing compiler that from itself finds..." current Tag: Visual edit
- 13:3613:36, 17 May 2024 diff hist +16 History of the MemCP project No edit summary Tag: Visual edit
- 09:2409:24, 17 May 2024 diff hist +439 N History of the MemCP project Created page with "* On the beforehand, a lot of research was done at the HAEC project * MemCPs predecessor is the [https://wwwdb.inf.tu-dresden.de/research-projects/eris/ Eris] project at TU Dresden * The first commit on MemCP was done on 24th of December 2022 at 12:13:37 * MemCP was originally written in go * The first SQL query on MemCP was successfuly executed on 12.02.2023 source: https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/" Tag: Visual edit
- 09:2009:20, 17 May 2024 diff hist +222 N Databases, Tables and Columns Created page with "* MemCP has a data folder which defaults to "./data". Every subdirectory of that folder is a database. * Databases consist of tables * Tables consist of columns and shards" Tag: Visual edit
- 09:1909:19, 17 May 2024 diff hist +57 Main Page →Navigation Tag: Visual edit
- 09:1809:18, 17 May 2024 diff hist +340 N Shards, RecordIDs, Main Storage, Delta Storage Created page with "* Datasets of a table are organized in shards of roughly 60,000 items * Inside a shard with n items, the datasets are numbered from 0..n-1 * The first few datasets of each shard are organized in a column store * The rest of the datasets are organized in a row store * The delta store consists of the insert list as well as the deletion mask" current Tag: Visual edit
- 09:1409:14, 17 May 2024 diff hist +2,475 N What is OLTP and OLAP Created page with "There are basically two types database workloads leading to different database designs: {| class="wikitable" |+ ! !OLTP !OLAP |- |abbr. |Online Transaction Processing |Online ''Analytical'' Processing |- |Example applications |ERP, Ticket System |BI suite, Statistical programs |- |Typical SQL Queries |large amounts of small UPDATE, INSERT |few SELECT with big computation demand |- |Best Storage Format |Row Store |Columnar store |- |Database Engines |MySQL, Postgres |Mone..." Tag: Visual edit
- 09:0209:02, 17 May 2024 diff hist +77 Main Page No edit summary Tag: Visual edit
- 08:5008:50, 17 May 2024 diff hist +476 Main Page No edit summary Tag: Visual edit
15 May 2024
- 13:4613:46, 15 May 2024 diff hist +76 MediaWiki:Common.css No edit summary current
- 12:5012:50, 15 May 2024 diff hist +12 Main Page No edit summary Tag: Visual edit
- 12:4912:49, 15 May 2024 diff hist 0 N File:Screenshot from htop.png No edit summary current
14 May 2024
- 22:3822:38, 14 May 2024 diff hist −24 Main Page No edit summary Tag: Visual edit
- 19:3019:30, 14 May 2024 diff hist +57 MediaWiki:Common.css No edit summary
- 18:4818:48, 14 May 2024 diff hist +91 N MediaWiki:Common.css Created page with "→CSS placed here will be applied to all skins: #p-logo a { background-size: contain; }"
- 18:4218:42, 14 May 2024 diff hist −7 Main Page No edit summary Tag: Visual edit: Switched
- 18:4118:41, 14 May 2024 diff hist +83 Main Page No edit summary Tag: Visual edit: Switched
- 18:3118:31, 14 May 2024 diff hist +53 Main Page No edit summary Tag: Visual edit
- 18:2718:27, 14 May 2024 diff hist +142 Main Page No edit summary Tag: Visual edit
- 18:2418:24, 14 May 2024 diff hist −10 Main Page No edit summary Tag: Visual edit: Switched
- 18:2418:24, 14 May 2024 diff hist +2,182 Main Page No edit summary Tag: Visual edit: Switched
- 18:1718:17, 14 May 2024 diff hist +1,833 N Main Page Created page with " === What is memcp? === memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world. memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalab..." Tag: Visual edit