Current Status and Open Issues

From MemCP
Jump to navigation Jump to search

There are several TODOs in MemCP still. These are categorized as follows:

Storage Engine

  • Allow ALTER TABLE ENGINE = ...
  • Garbage Collection with an LRU policy on temporary columns
  • Triggers and change hooks on computed columns
  • Respect Foreign Keys
  • Serialize and Deserialize into MMapped big files (these bigfiles must be organized as key-value stores)
  • iterateIndex: sort delta storage and merge it with main, so that correct order is always guaranteed
  • merge join: (scan_star schema tbls[] joincols[] filtercols[][] filterfn mapcols[][] mapfn reduce neutral)
  • processes: implement kill switch in sync.go and set a correct context in mysql.go, also a process id concept is missing in the dependent mysql connection library
  • transactions: map[shard]{deletionOverlay, insertionOverlay NonBlockingBitmap}
    • inserts are inserted as deleted for the main view but the insertionOverlay will tell that the deletion is reversed after commit
    • during commit, all shards are locked at the same time
    • if deletions & deletionOverlay != 0, abort transaction (write after write conflict)
    • deletions = (deletions | deletionOverlay) & (^insertionOverlay) to apply the changes
    • during scans, the overlays must be respected
  • Memory-Mapped Serialize & Deserialize
    • import "https://github.com/edsrzf/mmap-go/blob/main/mmap.go"
    • mmap.Map(file, RDWR, 0)
    • map blocks of 100GiB chunks per database
    • encode a map[string]blob into these blocks
    • type MMapReader interface { Reader MMap(int size) []byte }
    • mutex: only one WriteCloser is able to append to that file

RDF Frontend

  • INSERT {triples} after a SELECT
  • DELETE {triples} after a SELECT
  • WHERE { OPTIONAL {} } syntax which is translated to an outer join

SQL Frontend

  • AUTO_INCREMENT
  • Convert Subqueries into LEFT JOIN (https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf)
  • GROUP: force sharding of the grouptbl to be the same sharding schema as the group keys
  • Prejoin complex query plans (group on inter-table conditions)
  • system.grant table to restrict users to databases
  • Test tools like DBeaver and phpmyadmin and extend the parsed syntax and supported metadata tables

Scheme language

  • Logging and time measurement (controlled via contexts in sync.go)
  • support for http:// links in filenames for load and watch
  • support for ipfs:// links in filenames for load and watch
  • JIT engine: specialize code either on bitcode level or on machine code level according to https://cs.emis.de/LNI/Proceedings/Proceedings241/363.pdf

Infrastructure

  • MySQL importer
  • Plugin concept e.g. for AI or external C++ libraries (GPU BLAS or something like that)
Retrieved from "https://www.memcp.org/index.php?title=Current_Status_and_Open_Issues&oldid=127"