Current Status and Open Issues
Jump to navigation
Jump to search
There are several TODOs in MemCP still. These are categorized as follows:
Storage Engine
- Allow ALTER TABLE ENGINE = ...
- Garbage Collection with an LRU policy on temporary columns
- Triggers and change hooks on computed columns
- Respect Foreign Keys
- Serialize and Deserialize into MMapped big files (these bigfiles must be organized as key-value stores)
- iterateIndex: sort delta storage and merge it with main, so that correct order is always guaranteed
- merge join:
(scan_star schema tbls[] joincols[] filtercols[][] filterfn mapcols[][] mapfn reduce neutral)
- processes: implementation with https://pkg.go.dev/context (killable, timeoutable), see also: https://pkg.go.dev/github.com/jtolds/gls and support KILL pid (processlist stored in in sync.Map pid->{user,commandstring,cancelfunc})
- transactions: map[shard]{deletionOverlay, insertionOverlay NonBlockingBitmap}
- inserts are inserted as deleted for the main view but the insertionOverlay will tell that the deletion is reversed after commit
- during commit, all shards are locked at the same time
- if deletions & deletionOverlay != 0, abort transaction (write after write conflict)
- deletions = (deletions | deletionOverlay) & (^insertionOverlay) to apply the changes
- during scans, the overlays must be respected
- Memory-Mapped Serialize & Deserialize
import "https://github.com/edsrzf/mmap-go/blob/main/mmap.go"
mmap.Map(file, RDWR, 0)
- map blocks of 100GiB chunks per database
- encode a map[string]blob into these blocks
type MMapReader interface { Reader MMap(int size) []byte }
- mutex: only one WriteCloser is able to append to that file
RDF Frontend
- INSERT {triples} after a SELECT
- DELETE {triples} after a SELECT
- WHERE { OPTIONAL {} } syntax which is translated to an outer join
SQL Frontend
- AUTO_INCREMENT
- Convert Subqueries into LEFT JOIN (https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf)
- GROUP: force sharding of the
grouptbl
to be the same sharding schema as the group keys - Prejoin complex query plans (group on inter-table conditions)
- system.grant table to restrict users to databases
- Test tools like DBeaver and phpmyadmin and extend the parsed syntax and supported metadata tables
Scheme language
(watch "filename" (lambda (newcontent) ...))
for file-based observers (RDFfop dynamic file updating)- support for http:// links in filenames for
load
andwatch
- support for ipfs:// links in filenames for
load
andwatch
- JIT engine: specialize code either on bitcode level or on machine code level according to https://cs.emis.de/LNI/Proceedings/Proceedings241/363.pdf
Infrastructure
- MySQL importer
- Plugin concept e.g. for AI or external C++ libraries (GPU BLAS or something like that)