User contributions for Carli
Jump to navigation
Jump to search
19 May 2024
- 09:1609:16, 19 May 2024 diff hist +20 Main Page →SQL Frontend Tag: Visual edit
- 09:0609:06, 19 May 2024 diff hist +102 Current Status and Open Issues No edit summary Tag: Visual edit
- 09:0409:04, 19 May 2024 diff hist +44 N Current Status and open issues Carli moved page Current Status and open issues to Current Status and Open Issues current Tag: New redirect
- 09:0409:04, 19 May 2024 diff hist 0 m Current Status and Open Issues Carli moved page Current Status and open issues to Current Status and Open Issues
- 09:0409:04, 19 May 2024 diff hist −31 Main Page →Introduction Tag: Visual edit
- 09:0209:02, 19 May 2024 diff hist +82 Main Page →Navigation Tag: Visual edit
18 May 2024
- 21:4121:41, 18 May 2024 diff hist +1,950 Hardware Requirements →Micro-Benchmark Tag: Visual edit
- 20:5320:53, 18 May 2024 diff hist +31 Main Page →Introduction Tag: Visual edit
- 17:4117:41, 18 May 2024 diff hist +4,012 Hardware Requirements →Running on Raspberry Pi Tag: Visual edit
- 17:3117:31, 18 May 2024 diff hist +841 Hardware Requirements No edit summary Tag: Visual edit
- 12:5812:58, 18 May 2024 diff hist +88 Integer Compression →See also Tag: Visual edit
17 May 2024
- 22:0722:07, 17 May 2024 diff hist +419 N Process Hibernation Created page with "MemCP is a In-Memory database. In case you use the <code>memory</code> engine, there is no data stored on disk. <code>criu</code> is an OpenSource project to hibernate processes on linux on disk and restore them later (see https://criu.org/Main_Page). You can hibernate the memcp process with sudo criu dump -t [PID] -j -D [DUMP_DIRECTORY] And reactivate the process with: sudo criu restore -j -D [DUMP_DIRECTORY]" Tag: Visual edit
- 22:0322:03, 17 May 2024 diff hist +31 Settings No edit summary Tag: Visual edit
- 22:0222:02, 17 May 2024 diff hist +734 N Settings Created page with "Settings are changed by: (settings "key" "value") The following settings are available: * <code>"Backtrace"</code>: <code>true</code> or <code>false</code> whether to print backtraces from Scheme. Using true decreases the performance by ~10% but gives you better debugging capabilities. * <code>"PartitionMaxDimensions"</code>: number from 0 to 10 how many partitioning dimensions are allowed at max. 0 means, don't partition shards for values (use it to workaround bugs),..." Tag: Visual edit
- 21:5821:58, 17 May 2024 diff hist +97 Main Page →Navigation Tag: Visual edit
- 21:5121:51, 17 May 2024 diff hist +3,585 Integer Compression No edit summary Tag: Visual edit
- 21:4921:49, 17 May 2024 diff hist +40 N Temporary Columns Carli moved page Temporary Columns to Temporary Computed Columns current Tag: New redirect
- 21:4921:49, 17 May 2024 diff hist 0 m Temporary Computed Columns Carli moved page Temporary Columns to Temporary Computed Columns
- 21:4821:48, 17 May 2024 diff hist +2,999 N Temporary Computed Columns Created page with "Most (or better: all) RDBMS know the concept of so-called „temporary tables“. These tables act as a buffer when complex computations or joins are made and the result has to be sorted afterwards. Temporary tables are also a big loss of performance in daily workloads: Let’s say a table consists of 1,000,000 items and you display them with ORDER BY … LIMIT 100. Because auf some complex join, the 1,000,000 items spanning sorted temptable has to be rendered first and..." Tag: Visual edit
- 21:4621:46, 17 May 2024 diff hist +89 Main Page →Internals Tag: Visual edit
- 21:4421:44, 17 May 2024 diff hist −12 In-Memory Compression, Columnar Compression Techniques No edit summary Tag: Visual edit
- 21:4321:43, 17 May 2024 diff hist +1,232 N Integer Compression Created page with "Traditional Database Management Systems use fixed width integers of 32 or 64 bit that are administered by the database operator. MemCP however uses a different approach: according to a data analysis pass, we detect the maximum bit width of an integer to be encoded into our database. This is done by: * finding the smallest integer in a shard * finding the highest integer * <code>offset = smallest</code> * <code>bitwidth = ⌈ld (highest - smallest)⌉</code> * allocate..." Tag: Visual edit
- 21:3221:32, 17 May 2024 diff hist 0 In-Memory Compression, Columnar Compression Techniques No edit summary Tag: Visual edit
- 21:3121:31, 17 May 2024 diff hist +332 In-Memory Compression, Columnar Compression Techniques No edit summary Tag: Visual edit
- 21:2821:28, 17 May 2024 diff hist +7,746 In-Memory Compression, Columnar Compression Techniques No edit summary Tag: Visual edit
- 21:2621:26, 17 May 2024 diff hist +164 Databases, Tables and Columns No edit summary Tag: Visual edit
- 21:2421:24, 17 May 2024 diff hist +2,922 N Current Status and Open Issues Created page with "There are several TODOs in MemCP still. These are categorized as follows: === Storage Engine === * Allow ALTER TABLE ENGINE = ... * Garbage Collection with an LRU policy on temporary columns * Triggers and change hooks on computed columns * Respect Foreign Keys * Serialize and Deserialize into MMapped big files (these bigfiles must be organized as key-value stores) * iterateIndex: sort delta storage and merge it with main, so that correct order is always guaranteed * m..." Tag: Visual edit
- 18:0818:08, 17 May 2024 diff hist +37 Main Page →Introduction Tag: Visual edit
- 18:0718:07, 17 May 2024 diff hist +7,345 N Index Compression Created page with "== Memory-Efficient Indices for In-Memory Storages == Most databases implement indices as a kind of tree. I will show you that columnar storages can do even better. The most widely used kind of tree structure in databases is the B-Tree. A B-Tree is a n-ary tree whose nodes fit exactly into one „page“ – may it be a cache line of 64 bytes or a HDD page of 512 bytes or a memory page of 4K. The advantage of btrees is that it balances the memory fetch time with the ti..." Tag: Visual edit
- 18:0418:04, 17 May 2024 diff hist +74 In-Memory Compression, Columnar Compression Techniques No edit summary Tag: Visual edit
- 16:1816:18, 17 May 2024 diff hist +23 Main Page No edit summary Tag: Visual edit
- 16:1516:15, 17 May 2024 diff hist +212 N Compile MemCP from Source Created page with "Make sure, <code>go</code> is installed on your computer. (at least 1.22.0) Compile the project with: git clone <nowiki>https://github.com/launix-de/memcp</nowiki> cd memcp make Run the engine with: ./memcp" Tag: Visual edit
- 16:1316:13, 17 May 2024 diff hist +441 N With Singularity Created page with "Singularity is a rootless container format for restricted environments like HPC. Here are the installation instructions: https://docs.sylabs.io/guides/latest/user-guide/quick_start.html#quick-installation-steps Build with: git clone <nowiki>https://github.com/launix-de/memcp</nowiki> cd memcp sudo singularity build memcp.sif memcp.singularity.recipe Now you can run memcp.sif: mkdir data singularity run --bind data:/data memcp.sif" Tag: Visual edit
- 16:1016:10, 17 May 2024 diff hist +660 N Install MemCP with Docker Created page with "To build the image the first time: git clone https://github.com/launix-de/memcp cd memcp docker build . -t memcp For app development and to enter the Scheme console: mkdir data docker run -v data:/data -it -p 4321:4321 -p 3307:3307 memcp To run your custom Scheme apps (like the one from RDF templating and model driven development): make data docker run -e PARAMS="lib/main.scm apps/minigame.scm" -v data:/data -it -p 4321:4321 -p 3307:3307 memcp During productio..." Tag: Visual edit
- 16:0616:06, 17 May 2024 diff hist +2,427 N Persistency and Performance Guarantees Created page with "MemCP gives the user several Guarantees for persistency and performance. These are the guarantees: == Persistency Guarantees == There are three persistency modes per table which are: * ENGINE = memory * ENGINE = sloppy * ENGINE = logged * ENGINE = safe === ENGINE = memory === * all data is held in memory and only in memory * in case of a crash, all data is gone * the schema is saved on disk * after a recovery, the table starts empty * fastest way to store data * use..." Tag: Visual edit
- 15:3215:32, 17 May 2024 diff hist +451 Lists and Objects No edit summary Tag: Visual edit
- 15:2715:27, 17 May 2024 diff hist +1,805 N Lists and Objects Created page with "Lists are a powerful feature in Scheme. Lists can express both: Data and Programs. ==List handling== The following list functions can be used: > '(1 2 3) "list literal" = (1 2 3) > (cons 0 '(1 2 3)) "prepend item to list" = (0 1 2 3) > (append '(1 2 3) 4 5) "append items to list" = (1 2 3 4 5) > (has? '(1 2 3) 2) "check if item is in list" = true > (filter '(1 2 3) (lambda (x) (< x 2))) "filter items that are smaller than 2" = (1) > (map '(1 2 3) (l..." Tag: Visual edit
- 15:2615:26, 17 May 2024 diff hist +12 Introduction to Scheme →Deeper Topics into Scheme Tag: Visual edit
- 15:2515:25, 17 May 2024 diff hist −1,600 Introduction to Scheme No edit summary Tag: Visual edit
- 15:2415:24, 17 May 2024 diff hist +1,740 Introduction to Scheme No edit summary Tag: Visual edit
- 14:5114:51, 17 May 2024 diff hist +46 Main Page →Navigation Tag: Visual edit
- 14:3614:36, 17 May 2024 diff hist +10,258 N Introduction to Scheme Created page with "When you run <code>./memcp</code>, you will be dropped at a scheme shell like this: memcp Copyright (C) 2023, 2024 Carl-Philip Hänsch This program comes with ABSOLUTELY NO WARRANTY; This is free software, and you are welcome to redistribute it under certain conditions; loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-id of type 10 loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-username of type 20 loading stora..." Tag: Visual edit
- 14:3514:35, 17 May 2024 diff hist +50 Main Page No edit summary Tag: Visual edit
- 14:0514:05, 17 May 2024 diff hist +30 Main Page →Navigation Tag: Visual edit
- 14:0414:04, 17 May 2024 diff hist +3,441 N Sequence Compression Created page with "One of the most interesting compression techniques on columnar storages is '''sequence compression'''. Sequence Compression in In-Memory Database yields 99% memory savings and a total of 13% A sequence is a column of numbers where each distance between two neighbouring numbers is equal. Example: * <code>1 2 3 4 5 6 7 8 9</code> * <code>3 3 3 3 3 3 3 3 3 3</code> * <code>10 20 30 40</code> * <code>8 7 6 5</code> A sequence can be described by its starting point, its st..." Tag: Visual edit
- 14:0114:01, 17 May 2024 diff hist +398 N In-Memory Compression, Columnar Compression Techniques Created page with "The following compression techniques are implemented: * Sparse Storage for columns with lots of NULL values * Float Storage for scientific data * Integer Compression with a bit-size approach * Sequence Compression for sequences of integers like IDs (1, 2, 3, 4, 5 ...) * Dictionary Compression for short strings * BLOB Zipping and Deduplication for strings longer than 1KiB" Tag: Visual edit
- 13:5713:57, 17 May 2024 diff hist +4,408 N Columnar Storage Created page with "The advantages for columnar storages over row based storages are the ability for good in-memory compression '''(low memory usage)''' and cache locality when accessing only few columns '''(performance)'''. == How we designed the Interface == When designing an interface for a storage engine for an in-memory database, a lot of considerations have to be made. Here are the design goals: * The interface must be simple so that using it does not require implementing all kinds..." Tag: Visual edit
- 13:5313:53, 17 May 2024 diff hist +4 Main Page No edit summary Tag: Visual edit
- 13:5313:53, 17 May 2024 diff hist +20 Main Page No edit summary Tag: Visual edit
- 13:5213:52, 17 May 2024 diff hist +432 N Hardware Requirements Created page with "MemCP can run on any hardware that is supported by the <code>go</code> compiler. To find the right hardware dimensions, you need to consider the following: * The process itself needs at least 8 MiB of RAM * The size of a 5 GiB MySQL database fits into ~1 GiB of RAM * To have a decent working set, take the RAM requirements x3, so a 5 GiB MySQL database should run on 3 GiB RAM * MemCP needs at least 1 CPU core, more is better ;)" Tag: Visual edit