Current Status and Open Issues

There are several TODOs in MemCP still. These are categorized as follows:

Storage Engine

• Allow ALTER TABLE ENGINE = ...
• Garbage Collection with an LRU policy on temporary columns
• Triggers and change hooks on computed columns
• Respect Foreign Keys
• Serialize and Deserialize into MMapped big files (these bigfiles must be organized as key-value stores)
• iterateIndex: sort delta storage and merge it with main, so that correct order is always guaranteed
• merge join: (scan_star schema tbls[] joincols[] filtercols[][] filterfn mapcols[][] mapfn reduce neutral)
• processes: implement kill switch in sync.go and set a correct context in mysql.go, also a process id concept is missing in the dependent mysql connection library
• transactions: map[shard]{deletionOverlay, insertionOverlay NonBlockingBitmap}
  • inserts are inserted as deleted for the main view but the insertionOverlay will tell that the deletion is reversed after commit
  • during commit, all shards are locked at the same time
  • if deletions & deletionOverlay != 0, abort transaction (write after write conflict)
  • deletions = (deletions | deletionOverlay) & (^insertionOverlay) to apply the changes
  • during scans, the overlays must be respected
• Memory-Mapped Serialize & Deserialize
  • import "https://github.com/edsrzf/mmap-go/blob/main/mmap.go"
  • mmap.Map(file, RDWR, 0)
  • map blocks of 100GiB chunks per database
  • encode a map[string]blob into these blocks
  • type MMapReader interface { Reader MMap(int size) []byte }
  • mutex: only one WriteCloser is able to append to that file

RDF Frontend

• INSERT {triples} after a SELECT
• DELETE {triples} after a SELECT
• WHERE { OPTIONAL {} } syntax which is translated to an outer join

SQL Frontend

• AUTO_INCREMENT
• Convert Subqueries into LEFT JOIN (https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf)
• GROUP: force sharding of the grouptbl to be the same sharding schema as the group keys
• Prejoin complex query plans (group on inter-table conditions)
• system.grant table to restrict users to databases
• Test tools like DBeaver and phpmyadmin and extend the parsed syntax and supported metadata tables

Scheme language

• Logging and time measurement (controlled via contexts in sync.go)
• support for http:// links in filenames for load and watch
• support for ipfs:// links in filenames for load and watch
• JIT engine: specialize code either on bitcode level or on machine code level according to https://cs.emis.de/LNI/Proceedings/Proceedings241/363.pdf

Infrastructure

• MySQL importer
• Plugin concept e.g. for AI or external C++ libraries (GPU BLAS or something like that)