All public logs

Combined display of all available logs of MemCP. You can narrow down the view by selecting a log type, the username (case-sensitive), or the affected page (also case-sensitive).

• 11:27, 31 October 2024 Carli talk contribs uploaded File:Webapps.svg
• 11:23, 31 October 2024 Carli talk contribs uploaded File:Webapps with MemCP.svg
• 11:23, 31 October 2024 Carli talk contribs created page File:Webapps with MemCP.svg
• 11:21, 31 October 2024 Carli talk contribs deleted page File:Webapps with MemCP.svg
• 11:21, 31 October 2024 Carli talk contribs deleted page File:Ports.svg
• 11:19, 31 October 2024 Carli talk contribs created page File:Webapps with MemCP.svg
• 11:19, 31 October 2024 Carli talk contribs uploaded File:Webapps with MemCP.svg
• 11:18, 31 October 2024 Carli talk contribs deleted page File:MemCP Webapps.png
• 11:04, 31 October 2024 Carli talk contribs uploaded File:Ports.svg
• 11:04, 31 October 2024 Carli talk contribs created page File:Ports.svg
• 11:00, 31 October 2024 Carli talk contribs created page Replace MySQL with MemCP (Created page with "thumb === Connection to MemCP === MemCP can be connected with any compatible MySQL connector: $db = new \PDO<code>"mysql:host=localhost;port=3307;dbname=system", 'root', 'admin');</code>") Tag: Visual edit
• 10:55, 31 October 2024 Carli talk contribs uploaded File:MemCP Port.png
• 10:55, 31 October 2024 Carli talk contribs created page File:MemCP Port.png
• 10:50, 31 October 2024 Carli talk contribs created page File:MemCP Webapps.png
• 10:50, 31 October 2024 Carli talk contribs uploaded File:MemCP Webapps.png
• 10:48, 31 October 2024 Carli talk contribs uploaded File:Webapps.png
• 10:48, 31 October 2024 Carli talk contribs created page File:Webapps.png
• 21:54, 8 October 2024 Carli talk contribs created page Websockets in MemCP (Created page with "In MemCP, you can call <code>(set send ((res "websocket") onReceiveFunction onCloseFunction))</code>to upgrade a http session into a websocket (<code>res</code> is the response object from your http handler, see serve). The <code>onReceiveFunction</code> will take message as it's parameter. <code>onCloseFunction</code> will be called when the socket is closed. You can call <code>(send 1 "my message")</code> to send a string to th...") Tag: Visual edit
• 06:38, 3 June 2024 Carli talk contribs uploaded File:Explainer.png
• 06:38, 3 June 2024 Carli talk contribs created page File:Explainer.png
• 13:20, 26 May 2024 Carli talk contribs created page Advanced Graph Querying (Created page with "== Property Paths == see also: https://en.wikibooks.org/wiki/SPARQL/Property_paths TODO for MemCP: * property/propertypath * propertypath*, propertypath+ (mark&sweep, then iterate over the results) * property? * (propertypath) * ^property (inverse path)") Tag: Visual edit
• 13:12, 20 May 2024 Carli talk contribs created page Contributing (Created page with "Here's how you can contribute: * Go to https://github.com/launix-de/memcp * Build From Source * Add unit tests * Implement more SQL Syntax * Find and report Bugs * Add Passes to the Optimizer") Tag: Visual edit
• 13:02, 20 May 2024 Carli talk contribs created page Database Tools compatibility with MemCP (Created page with "== PHPMyAdmin == no support yet. Read more under: Contributing == DBeaver == Support for: * List Databases * List Tables") Tag: Visual edit
• 11:54, 20 May 2024 Carli talk contribs created page Scan (Created page with "Scan is the most important function in whole MemCP. It implements an optimized, indexed and parallelized <code>for</code> loop over items in a table. There are two variants of scan: <code>scan</code> and <code>scan_order</code>. == (scan schema table filterColumns filter mapColumns map reduce neutral reduce2 isOuter) == Help for: scan === does an unordered parallel filter-map-reduce pass on a single table and returns the reduced result Allowed nø of parameters...") Tag: Visual edit
• 11:41, 20 May 2024 Carli talk contribs created page Data Auto Sharding and Auto Indexing (Created page with "== Auto-Indexing == == Auto-Sharding ==") Tag: Visual edit
• 10:55, 20 May 2024 Carli talk contribs created page Advanced SQL Tutorial (Created page with "== Atomic Counters through Unique Key Checks == Imagine the following table: CREATE TABLE u(id INT PRIMARY KEY, cnt INT); INSERT INTO u VALUES (1, 2), (2, 4); SELECT * FROM u; +------+------+ | id | cnt | +------+------+ | 1 | 2 | | 2 | 4 | +------+------+ '''3 rows in set (0,001 sec)''' If you want to add to <code>cnt</code> whenever an event occurs, add a unique key over <code>id</code> and do the following: INSERT INTO u VALUES (2, 1) O...") Tag: Visual edit
• 14:46, 19 May 2024 Carli talk contribs moved page In-Database WebApps to In-Database WebApps and REST Services
• 10:08, 19 May 2024 Carli talk contribs created page In-Database WebApps (Created page with "In-Database WebApps are a huge game changer in REST API performance since the web app runs in the same context as the == REST Endpoint setup == You basically set up a REST endpoint by overwriting the <code>http_handler</code> variable in the global scope. Best practice is here to overload http_handler, store the old handler in <code>old_handler</code> and cascade the router with a prefix: (define http_handler (begin (set old_handler (coalesce http_handler han...") Tag: Visual edit
• 09:47, 19 May 2024 Carli talk contribs created page SQL over REST (Created page with "There are two methods to do SQL over REST: a) GET method: <code>curl <nowiki>http://root:admin@localhost:4321/sql/</nowiki>[SCHEMA]/[SQL_QUERY]</code> b) POST method: <code>curl -X POST -d '[SQL_QUERY]' <nowiki>http://root:admin@localhost:4321/sql/</nowiki>[SCHEMA]</code> == JSONL results == The result is a <code>jsonl</code> document containing the result lines: curl -X POST -d 'select a, sum(b) as sum_b from a group by a' <nowiki>http://root:admin@localhost:4321/sq...") Tag: Visual edit
• 09:42, 19 May 2024 Carli talk contribs created page Introduction to RDF (Created page with "RDF is the '''Resource Description Format'''. It describes all data in ''Triple Format'': {| class="wikitable" |+ !Subject !Predicate !Object |- |item:1 |a |foaf:Person |- |item:1 |foaf:name |Klaus |- |item:1 |foaf:mbox |klaus@example.com |- |item:1 |foaf:knows |item:2 |- |item:2 |a |foaf:Person |- |item:2 |foaf:name |Dieter |} See also: https://github.com/launix-de/rdfop") Tag: Visual edit
• 09:23, 19 May 2024 Carli talk contribs created page Supported SQL (Created page with "The following SQL statements are supported: * SELECT FROM GROUP BY HAVING ORDER BY LIMIT OFFSET * UPDATE SET WHERE * DELETE FROM WHERE * INSERT INTO VALUES * CREATE TABLE * ALTER TABLE * CREATE DATABASE * CREATE USER * ALTER USER * SHOW DATABASES * SHOW TABLES * SHOW TABLE STATUS * SHOW VARIABLES * SET NAMES (no function) * DROP DATABASE * DROP TABLE * SET SESSION * LOCK TABLES (no function) * UNLOCK TABLES (no function) The following functions are supported: * CASE...") Tag: Visual edit
• 09:04, 19 May 2024 Carli talk contribs moved page Current Status and open issues to Current Status and Open Issues
• 22:07, 17 May 2024 Carli talk contribs created page Process Hibernation (Created page with "MemCP is a In-Memory database. In case you use the <code>memory</code> engine, there is no data stored on disk. <code>criu</code> is an OpenSource project to hibernate processes on linux on disk and restore them later (see https://criu.org/Main_Page). You can hibernate the memcp process with sudo criu dump -t [PID] -j -D [DUMP_DIRECTORY] And reactivate the process with: sudo criu restore -j -D [DUMP_DIRECTORY]") Tag: Visual edit
• 22:02, 17 May 2024 Carli talk contribs created page Settings (Created page with "Settings are changed by: (settings "key" "value") The following settings are available: * <code>"Backtrace"</code>: <code>true</code> or <code>false</code> whether to print backtraces from Scheme. Using true decreases the performance by ~10% but gives you better debugging capabilities. * <code>"PartitionMaxDimensions"</code>: number from 0 to 10 how many partitioning dimensions are allowed at max. 0 means, don't partition shards for values (use it to workaround bugs),...") Tag: Visual edit
• 21:49, 17 May 2024 Carli talk contribs moved page Temporary Columns to Temporary Computed Columns
• 21:48, 17 May 2024 Carli talk contribs created page Temporary Columns (Created page with "Most (or better: all) RDBMS know the concept of so-called „temporary tables“. These tables act as a buffer when complex computations or joins are made and the result has to be sorted afterwards. Temporary tables are also a big loss of performance in daily workloads: Let’s say a table consists of 1,000,000 items and you display them with ORDER BY … LIMIT 100. Because auf some complex join, the 1,000,000 items spanning sorted temptable has to be rendered first and...") Tag: Visual edit
• 21:43, 17 May 2024 Carli talk contribs created page Integer Compression (Created page with "Traditional Database Management Systems use fixed width integers of 32 or 64 bit that are administered by the database operator. MemCP however uses a different approach: according to a data analysis pass, we detect the maximum bit width of an integer to be encoded into our database. This is done by: * finding the smallest integer in a shard * finding the highest integer * <code>offset = smallest</code> * <code>bitwidth = ⌈ld (highest - smallest)⌉</code> * allocate...") Tag: Visual edit
• 21:24, 17 May 2024 Carli talk contribs created page Current Status and open issues (Created page with "There are several TODOs in MemCP still. These are categorized as follows: === Storage Engine === * Allow ALTER TABLE ENGINE = ... * Garbage Collection with an LRU policy on temporary columns * Triggers and change hooks on computed columns * Respect Foreign Keys * Serialize and Deserialize into MMapped big files (these bigfiles must be organized as key-value stores) * iterateIndex: sort delta storage and merge it with main, so that correct order is always guaranteed * m...") Tag: Visual edit
• 18:07, 17 May 2024 Carli talk contribs created page Index Compression (Created page with "== Memory-Efficient Indices for In-Memory Storages == Most databases implement indices as a kind of tree. I will show you that columnar storages can do even better. The most widely used kind of tree structure in databases is the B-Tree. A B-Tree is a n-ary tree whose nodes fit exactly into one „page“ – may it be a cache line of 64 bytes or a HDD page of 512 bytes or a memory page of 4K. The advantage of btrees is that it balances the memory fetch time with the ti...") Tag: Visual edit
• 16:15, 17 May 2024 Carli talk contribs created page Compile MemCP from Source (Created page with "Make sure, <code>go</code> is installed on your computer. (at least 1.22.0) Compile the project with: git clone <nowiki>https://github.com/launix-de/memcp</nowiki> cd memcp make Run the engine with: ./memcp") Tag: Visual edit
• 16:13, 17 May 2024 Carli talk contribs created page With Singularity (Created page with "Singularity is a rootless container format for restricted environments like HPC. Here are the installation instructions: https://docs.sylabs.io/guides/latest/user-guide/quick_start.html#quick-installation-steps Build with: git clone <nowiki>https://github.com/launix-de/memcp</nowiki> cd memcp sudo singularity build memcp.sif memcp.singularity.recipe Now you can run memcp.sif: mkdir data singularity run --bind data:/data memcp.sif") Tag: Visual edit
• 16:10, 17 May 2024 Carli talk contribs created page Install MemCP with Docker (Created page with "To build the image the first time: git clone https://github.com/launix-de/memcp cd memcp docker build . -t memcp For app development and to enter the Scheme console: mkdir data docker run -v data:/data -it -p 4321:4321 -p 3307:3307 memcp To run your custom Scheme apps (like the one from RDF templating and model driven development): make data docker run -e PARAMS="lib/main.scm apps/minigame.scm" -v data:/data -it -p 4321:4321 -p 3307:3307 memcp During productio...") Tag: Visual edit
• 16:06, 17 May 2024 Carli talk contribs created page Persistency and Performance Guarantees (Created page with "MemCP gives the user several Guarantees for persistency and performance. These are the guarantees: == Persistency Guarantees == There are three persistency modes per table which are: * ENGINE = memory * ENGINE = sloppy * ENGINE = logged * ENGINE = safe === ENGINE = memory === * all data is held in memory and only in memory * in case of a crash, all data is gone * the schema is saved on disk * after a recovery, the table starts empty * fastest way to store data * use...") Tag: Visual edit
• 15:27, 17 May 2024 Carli talk contribs created page Lists and Objects (Created page with "Lists are a powerful feature in Scheme. Lists can express both: Data and Programs. ==List handling== The following list functions can be used: > '(1 2 3) "list literal" = (1 2 3) > (cons 0 '(1 2 3)) "prepend item to list" = (0 1 2 3) > (append '(1 2 3) 4 5) "append items to list" = (1 2 3 4 5) > (has? '(1 2 3) 2) "check if item is in list" = true > (filter '(1 2 3) (lambda (x) (< x 2))) "filter items that are smaller than 2" = (1) > (map '(1 2 3) (l...") Tag: Visual edit
• 14:36, 17 May 2024 Carli talk contribs created page Introduction to Scheme (Created page with "When you run <code>./memcp</code>, you will be dropped at a scheme shell like this: memcp Copyright (C) 2023, 2024 Carl-Philip Hänsch This program comes with ABSOLUTELY NO WARRANTY; This is free software, and you are welcome to redistribute it under certain conditions; loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-id of type 10 loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-username of type 20 loading stora...") Tag: Visual edit
• 14:04, 17 May 2024 Carli talk contribs created page Sequence Compression (Created page with "One of the most interesting compression techniques on columnar storages is '''sequence compression'''. Sequence Compression in In-Memory Database yields 99% memory savings and a total of 13% A sequence is a column of numbers where each distance between two neighbouring numbers is equal. Example: * <code>1 2 3 4 5 6 7 8 9</code> * <code>3 3 3 3 3 3 3 3 3 3</code> * <code>10 20 30 40</code> * <code>8 7 6 5</code> A sequence can be described by its starting point, its st...") Tag: Visual edit
• 14:01, 17 May 2024 Carli talk contribs created page In-Memory Compression, Columnar Compression Techniques (Created page with "The following compression techniques are implemented: * Sparse Storage for columns with lots of NULL values * Float Storage for scientific data * Integer Compression with a bit-size approach * Sequence Compression for sequences of integers like IDs (1, 2, 3, 4, 5 ...) * Dictionary Compression for short strings * BLOB Zipping and Deduplication for strings longer than 1KiB") Tag: Visual edit
• 13:57, 17 May 2024 Carli talk contribs created page Columnar Storage (Created page with "The advantages for columnar storages over row based storages are the ability for good in-memory compression '''(low memory usage)''' and cache locality when accessing only few columns '''(performance)'''. == How we designed the Interface == When designing an interface for a storage engine for an in-memory database, a lot of considerations have to be made. Here are the design goals: * The interface must be simple so that using it does not require implementing all kinds...") Tag: Visual edit
• 13:52, 17 May 2024 Carli talk contribs created page Hardware Requirements (Created page with "MemCP can run on any hardware that is supported by the <code>go</code> compiler. To find the right hardware dimensions, you need to consider the following: * The process itself needs at least 8 MiB of RAM * The size of a 5 GiB MySQL database fits into ~1 GiB of RAM * To have a decent working set, take the RAM requirements x3, so a 5 GiB MySQL database should run on 3 GiB RAM * MemCP needs at least 1 CPU core, more is better ;)") Tag: Visual edit
• 12:51, 17 May 2024 Carli talk contribs created page Parallel Computing (Created page with "Almost 99% of all newly invented are imperative programming languages. But imperative languages have one drawback: their parallelization is hard. == Drawbacks of Imperative Programming Languages == Imperative programming languages do have one mayor drawback: state. The concept of an imperative language is that commands are executed which change the content of variables or complex objects in the memory. When trying to create an optimizing compiler that from itself finds...") Tag: Visual edit