Current Status and Open Issues - Revision history

Carli: /* Scheme language */

2024-11-20T18:38:41Z

Scheme language

← Older revision		Revision as of 20:38, 20 November 2024
Line 24:		Line 24:
	** <code>type MMapReader interface { Reader MMap(int size) []byte }</code>		** <code>type MMapReader interface { Reader MMap(int size) []byte }</code>
	** mutex: only one WriteCloser is able to append to that file		** mutex: only one WriteCloser is able to append to that file
			* Indexes for <code>LIKE</code> queries

	=== RDF Frontend ===		=== RDF Frontend ===
Line 42:		Line 43:
	=== Scheme language ===		=== Scheme language ===

	* Logging and time measurement (controlled via contexts in <code>sync.go</code>)		* support for http:// links in filenames for <code>load</code> and stream
	* support for http:// links in filenames for <code>load</code> and ~~<code>watch</code>~~		* support for ipfs:// links in filenames for <code>load</code> and stream
	* support for ipfs:// links in filenames for <code>load</code> and ~~<code>watch</code>~~
	* JIT engine: specialize code either on bitcode level or on machine code level according to https://cs.emis.de/LNI/Proceedings/Proceedings241/363.pdf		* JIT engine: specialize code either on bitcode level or on machine code level according to https://cs.emis.de/LNI/Proceedings/Proceedings241/363.pdf

	=== Infrastructure ===		=== Infrastructure ===

	* MySQL importer
	* Plugin concept e.g. for AI or external C++ libraries (GPU BLAS or something like that)		* Plugin concept e.g. for AI or external C++ libraries (GPU BLAS or something like that)

Carli at 08:15, 30 May 2024

2024-05-30T08:15:25Z

← Older revision		Revision as of 10:15, 30 May 2024
Line 43:		Line 43:

	* Logging and time measurement (controlled via contexts in <code>sync.go</code>)		* Logging and time measurement (controlled via contexts in <code>sync.go</code>)
	* <code>(watch "filename" (lambda (newcontent) ...))</code> for file-based observers (RDFfop dynamic file updating)
	* support for http:// links in filenames for <code>load</code> and <code>watch</code>		* support for http:// links in filenames for <code>load</code> and <code>watch</code>
	* support for ipfs:// links in filenames for <code>load</code> and <code>watch</code>		* support for ipfs:// links in filenames for <code>load</code> and <code>watch</code>

Carli: /* Scheme language */

2024-05-27T07:19:53Z

Scheme language

← Older revision		Revision as of 09:19, 27 May 2024
Line 10:		Line 10:
	* iterateIndex: sort delta storage and merge it with main, so that correct order is always guaranteed		* iterateIndex: sort delta storage and merge it with main, so that correct order is always guaranteed
	* merge join: <code>(scan_star schema tbls[] joincols[] filtercols[][] filterfn mapcols[][] mapfn reduce neutral)</code>		* merge join: <code>(scan_star schema tbls[] joincols[] filtercols[][] filterfn mapcols[][] mapfn reduce neutral)</code>
	* processes: ~~implementation with https://pkg~~.go~~.dev~~/context ~~(killable, timeoutable), see also: https://pkg~~.go~~.dev/github.com~~/~~jtolds/gls and support KILL pid (processlist stored in in sync.Map pid-~~>~~{user~~,~~commandstring,cancelfunc})~~		* processes: implement kill switch in <code>sync.go</code> and set a correct context in <code>mysql.go</code>, also a process id concept is missing in the dependent mysql connection library
	* transactions: map[shard]{deletionOverlay, insertionOverlay NonBlockingBitmap}		* transactions: map[shard]{deletionOverlay, insertionOverlay NonBlockingBitmap}
	** inserts are inserted as deleted for the main view but the insertionOverlay will tell that the deletion is reversed after commit		** inserts are inserted as deleted for the main view but the insertionOverlay will tell that the deletion is reversed after commit
Line 42:		Line 42:
	=== Scheme language ===		=== Scheme language ===

	* Logging and time measurement (controlled via contexts ~~and https://pkg~~.go~~.dev/github.com/jtolds~~/~~gls~~)		* Logging and time measurement (controlled via contexts in <code>sync.go</code>)
	* <code>(watch "filename" (lambda (newcontent) ...))</code> for file-based observers (RDFfop dynamic file updating)		* <code>(watch "filename" (lambda (newcontent) ...))</code> for file-based observers (RDFfop dynamic file updating)
	* support for http:// links in filenames for <code>load</code> and <code>watch</code>		* support for http:// links in filenames for <code>load</code> and <code>watch</code>

Carli at 08:06, 19 May 2024

2024-05-19T08:06:50Z

← Older revision		Revision as of 10:06, 19 May 2024
Line 42:		Line 42:
	=== Scheme language ===		=== Scheme language ===

			* Logging and time measurement (controlled via contexts and https://pkg.go.dev/github.com/jtolds/gls)
	* <code>(watch "filename" (lambda (newcontent) ...))</code> for file-based observers (RDFfop dynamic file updating)		* <code>(watch "filename" (lambda (newcontent) ...))</code> for file-based observers (RDFfop dynamic file updating)
	* support for http:// links in filenames for <code>load</code> and <code>watch</code>		* support for http:// links in filenames for <code>load</code> and <code>watch</code>

Carli: Carli moved page Current Status and open issues to Current Status and Open Issues

2024-05-19T08:04:30Z

Carli moved page Current Status and open issues to Current Status and Open Issues

← Older revision	Revision as of 10:04, 19 May 2024
(No difference)

Carli: Created page with "There are several TODOs in MemCP still. These are categorized as follows: === Storage Engine === * Allow ALTER TABLE ENGINE = ... * Garbage Collection with an LRU policy on temporary columns * Triggers and change hooks on computed columns * Respect Foreign Keys * Serialize and Deserialize into MMapped big files (these bigfiles must be organized as key-value stores) * iterateIndex: sort delta storage and merge it with main, so that correct order is always guaranteed * m..."

2024-05-17T20:24:25Z

Created page with "There are several TODOs in MemCP still. These are categorized as follows: === Storage Engine === * Allow ALTER TABLE ENGINE = ... * Garbage Collection with an LRU policy on temporary columns * Triggers and change hooks on computed columns * Respect Foreign Keys * Serialize and Deserialize into MMapped big files (these bigfiles must be organized as key-value stores) * iterateIndex: sort delta storage and merge it with main, so that correct order is always guaranteed * m..."

New page

There are several TODOs in MemCP still. These are categorized as follows:

=== Storage Engine ===

* Allow ALTER TABLE ENGINE = ...
* Garbage Collection with an LRU policy on temporary columns
* Triggers and change hooks on computed columns
* Respect Foreign Keys
* Serialize and Deserialize into MMapped big files (these bigfiles must be organized as key-value stores)
* iterateIndex: sort delta storage and merge it with main, so that correct order is always guaranteed
* merge join: <code>(scan_star schema tbls[] joincols[] filtercols[][] filterfn mapcols[][] mapfn reduce neutral)</code>
* processes: implementation with https://pkg.go.dev/context (killable, timeoutable), see also: https://pkg.go.dev/github.com/jtolds/gls and support KILL pid (processlist stored in in sync.Map pid->{user,commandstring,cancelfunc})
* transactions: map[shard]{deletionOverlay, insertionOverlay NonBlockingBitmap}
** inserts are inserted as deleted for the main view but the insertionOverlay will tell that the deletion is reversed after commit
** during commit, all shards are locked at the same time
** if deletions & deletionOverlay != 0, abort transaction (write after write conflict)
** deletions = (deletions | deletionOverlay) & (^insertionOverlay) to apply the changes
** during scans, the overlays must be respected
* Memory-Mapped Serialize & Deserialize
** <code>import "<nowiki>https://github.com/edsrzf/mmap-go/blob/main/mmap.go</nowiki>"</code>
** <code>mmap.Map(file, RDWR, 0)</code>
** map blocks of 100GiB chunks per database
** encode a map[string]blob into these blocks
** <code>type MMapReader interface { Reader MMap(int size) []byte }</code>
** mutex: only one WriteCloser is able to append to that file

=== RDF Frontend ===

* INSERT {triples} after a SELECT
* DELETE {triples} after a SELECT
* WHERE { OPTIONAL {} } syntax which is translated to an outer join

=== SQL Frontend ===

* AUTO_INCREMENT
* Convert Subqueries into LEFT JOIN (https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf)
* GROUP: force sharding of the <code>grouptbl</code> to be the same sharding schema as the group keys
* Prejoin complex query plans (group on inter-table conditions)
* system.grant table to restrict users to databases
* Test tools like DBeaver and phpmyadmin and extend the parsed syntax and supported metadata tables

=== Scheme language ===

* <code>(watch "filename" (lambda (newcontent) ...))</code> for file-based observers (RDFfop dynamic file updating)
* support for http:// links in filenames for <code>load</code> and <code>watch</code>
* support for ipfs:// links in filenames for <code>load</code> and <code>watch</code>
* JIT engine: specialize code either on bitcode level or on machine code level according to https://cs.emis.de/LNI/Proceedings/Proceedings241/363.pdf

=== Infrastructure ===

* MySQL importer
* Plugin concept e.g. for AI or external C++ libraries (GPU BLAS or something like that)