Websockets in MemCP: Difference between revisions

From MemCP
Jump to navigation Jump to search
(Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference)
Line 1: Line 1:
Websockets are an extension to the HTTP protocol where an open connection can be used as a TCP socket to communicate between client and server two-directional and in realtime.
<!-- Copyright (C) 2026 Carl-Philip Haensch -->
<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
= Websockets in MemCP =


The advantage of MemCP is that you can define your websocket services inside the database itself (loaded as a module via <code>import</code>). This is faster than traditional Node.js applications since there is no need for a socket or TCP connection between your database and the application. The application is hosted by the database itself. Prepared statements will be precompiled and optimized so that there is no compile / parse phase for every single query. Everything is baked into one piece of software without the loss of data protection a RDBMS provies.
An embedded HTTP handler can upgrade a request through the response object's WebSocket facility and receive messages through a callback. The <code>apps/minigame.scm</code> example in the repository demonstrates the current API and routing pattern.


To further enhance the performance for realtime critical scenarios with a lot of write load, you can use the <code>sloppy</code> storage strategy to take away write load from your hard disk. To more on this, read [[Persistency and Performance Guarantees]].
WebSockets keep one bidirectional connection open for live dashboards, notifications, collaborative state, games, or change feeds. They are a delivery channel, not a durable message queue: reconnect, replay, acknowledgement, ordering, and missed-message semantics belong to the application.


=== How to use websockets ===
== Server-side upgrade ==
In MemCP, you can call <code>(set send ((res "websocket") onReceiveFunction onCloseFunction))</code>to upgrade a http session into a websocket (<code>res</code> is the response object from your http handler, see [[In-Database WebApps and REST Services|serve]]). The <code>onReceiveFunction</code> will take message as it's parameter. <code>onCloseFunction</code> will be called when the socket is closed. You can call <code>(send 1 "my message")</code> to send a string to the client. You can of course use <code>json_encode</code>, <code>json_encode_assoc</code> and <code>json_decode</code> to transfer data between JavaScript and your database.


=== Example Code ===
<syntaxhighlight lang="scheme">
(define http_handler (lambda (req res) (begin
(set send ((res "websocket")
        (set send ((res "websocket") (lambda (msg) (begin
(lambda (message) (print "message: " message))
                (print "I received: " msg)
(lambda () (print "connection closed"))))
                (send 1 (concat "I received: " msg))
(send 1 "Hello from MemCP")
        ) (lambda () (begin
</syntaxhighlight>
                (print "user has left")
        ))
        (send 1 "Welcome to our server")
)))
(serve 8000 http_handler)


=== Complex Example ===
The numeric first argument is the WebSocket frame opcode used by the current response API. Keep the sending function only for the connection lifetime.
Here's an example application of a publish subscribe network for a game where player positions are streamed to other players (used in [https://hardlife.io hardlife.io]). It basically creates 9 prepared SQL statements in advance and calls them during the lifetime of a websocket. The parameters are passed via the <code>session</code> object. Every user is authenticated by a MAC (message authentication code) so the client proves that he is really the owner of that player handle.
 
(print "")
A browser client can connect to the routed endpoint with the standard API:
(print "Welcome to our Worldserver")
 
(print "")
<syntaxhighlight lang="javascript">
(print "set MEMCP and PORT in your environment")
const socket = new WebSocket("wss://example.test/minigame/ws");
socket.addEventListener("open", () => socket.send(JSON.stringify({type: "hello"})));
/* this can be overhooked */
socket.addEventListener("message", event => console.log(JSON.parse(event.data)));
(define http_handler (lambda (req res) (begin
</syntaxhighlight>
        (print "request " req)
 
        ((res "header") "Content-Type" "text/plain")
== Complete routed example ==
        ((res "status") 404)
 
        ((res "println") "404 not found")
The repository's <code>apps/minigame.scm</code> shows the complete routing pattern: preserve the previous handler, serve an HTML/JavaScript client below one prefix, and upgrade only the WebSocket path.
)))
 
<syntaxhighlight lang="scheme">
(set MEMCP (env "MEMCP" "../../memcp"))
(define http_handler (begin
(import (concat MEMCP "/lib/sql.scm"))
(set old_handler http_handler)
(lambda (req res) (begin
/* load config */
(match (req "path")
(set CONF (env "CONF" "../out/conf.json"))
"/minigame/ws" (begin
(set conf (json_decode (load CONF)))
(set send ((res "websocket")
(set macKey (((conf "Types") "Password") "encryptKey"))
(lambda (message) (begin
(print "message: " message)
/* init database */
(send 1 (concat "echo: " message))))))
(settings "DefaultEngine" "sloppy")
(send 1 "Hello from MemCP"))
(createdatabase "hardlife" true)
(regex "^/minigame/(.*)$" path asset) (begin
(define resultrow print)
((res "header") "Content-Type" "text/plain")
(define session (newsession))
((res "status") 200)
(eval (parse_sql "hardlife" "CREATE TABLE IF NOT EXISTS avatar(ID int, x double, y double, z double, r double, location int, UNIQUE KEY PRIMARY(ID))"))
((res "print") "WebSocket client assets belong here"))
(eval (parse_sql "hardlife" "CREATE TABLE IF NOT EXISTS avatarListen(ID int, container int, UNIQUE KEY PRIMARY(ID, container))"))
(old_handler req res))))))
(eval (parse_sql "hardlife" "CREATE TABLE IF NOT EXISTS avatarDirty(ID int, other int, UNIQUE KEY PRIMARY(ID, other))"))
</syntaxhighlight>
(set update_location_prepared (parse_sql "hardlife" "INSERT INTO avatar(ID, x, y, z, r, location) VALUES (@avatar, @x, @y, @z, @r, @l) ON DUPLICATE KEY UPDATE x = @x, y = @y, z = @z, r = @r, location = @l"))
 
(set update_dirtylist_prepared (parse_sql "hardlife" "INSERT IGNORE INTO avatarDirty(ID, other) SELECT ID, @avatar FROM avatarListen WHERE ID != @avatar AND container = @l"))
When this module is imported by <code>lib/main.scm</code>, the existing server uses the wrapped handler. A standalone application can call <code>serve</code> explicitly. Keep the receive callback and its connection-local <code>send</code> closure together; never store that closure after close.
(set clear_listen_prepared (parse_sql "hardlife" "DELETE FROM avatarListen WHERE ID = @avatar"))
 
(set insert_listen_prepared (parse_sql "hardlife" "INSERT IGNORE INTO avatarListen(ID, container) VALUES (@avatar, @l2)"))
Prefer JSON messages with an explicit type and version. Validate the decoded shape before reading fields, cap message size, and define how the client resubscribes or resumes after reconnect.
(set get_listen_prepared (parse_sql "hardlife" "SELECT avatar.ID, avatar.x, avatar.y, avatar.z, avatar.r, avatar.location AS l FROM avatarDirty, avatar WHERE avatar.ID = avatarDirty.other AND avatarDirty.ID = @avatar"))
 
(set clear_dirty_prepared (parse_sql "hardlife" "DELETE FROM avatarDirty WHERE ID = @avatar"))
== Database-backed publish/subscribe ==
 
(define http_handler (begin
Prepare fixed queries during module startup, create a session per connection, and bind authenticated identity and subscription state there. A receive callback can update state atomically; query results can be emitted through <code>send</code>. On close, release subscriptions and other references so they do not keep sessions or sending functions alive.
(set old_handler http_handler)
 
(lambda (req res) (begin
For fan-out, use bounded per-client queues and a policy for slow consumers (drop/coalesce messages or disconnect). A database table can persist events for replay, but a <code>memory</code> or <code>cache</code> table cannot provide durable delivery after restart.
/* hooked our additional paths to it */
 
(match (req "path")
The earlier hardlife.io example used this structure at larger scale: prepare position-update, listener-membership and dirty-list queries once; authenticate a player handle with a message authentication code; keep the connection's player/location state in a session; mark affected listeners dirty after a position change; send only changed avatars; and remove listener state on disconnect. The complete historical listing coupled credentials, global <code>sloppy</code> defaults and application-specific tables, so reproducing it as a copy-and-paste server would be unsafe. The pipeline remains a useful design example.
"/" (begin
 
(set session (newsession))
Hosting the handler beside the database removes an application/database socket hop and repeated parsing for prepared shapes. That is a performance opportunity, not proof that every embedded WebSocket service is faster than a separate Node.js or Go service; benchmark the authenticated end-to-end path and account for the shared failure domain.
(set send ((res "websocket") (lambda (msg) (begin
 
(set msg (json_decode msg))
== Production considerations ==
(if
 
(and (has_assoc? msg "l") (session "avatar")) (begin
Authenticate the upgrade, validate browser Origin, limit message and queue sizes, apply backpressure, and remove subscriptions when the connection closes. A slow client must not create an unbounded in-memory backlog. Long database operations should observe cancellation. Use TLS at a trusted proxy or transport layer and do not expose default credentials.
/* position report */
 
(if (and (session "l") (not (equal? (msg "l") (session "l"))))
Storage ENGINE choice is independent of WebSocket delivery: <code>sloppy</code> can lose recent writes, and <code>memory</code>/<code>cache</code> are not durable. See [[In-Database WebApps and REST Services]] and [[Persistency and Performance Guarantees]].
(eval update_dirtylist_prepared)) /* update dirtylist if we leave a room */
(session "x" (msg "x"))
(session "y" (msg "y"))
(session "z" (msg "z"))
(session "r" (msg "r"))
(session "l" (msg "l"))
(eval update_location_prepared)
(eval update_dirtylist_prepared)
/* send back dirty items */
(set resultrow (lambda (item) (begin
(send 1 (json_encode_assoc item))
)))
(eval get_listen_prepared)
/* flush all dirtyflags of this avatar
(eval clear_dirty_prepared)
)
(and (has_assoc? msg "listen") (session "avatar")) (begin
/* update listener list */
(eval clear_listen_prepared)
(map (msg "listen") (lambda (container) (begin
(session "l2" container)
(eval insert_listen_prepared)
)))
)
(has_assoc? msg "authenticate") (begin
(if (equal? (bin2hex (password (concat "a:" (msg "authenticate") ":" macKey))) (msg "mac")) (begin
(print "authenicated avatar " (msg "authenticate"))
(session "avatar" (msg "authenticate"))
) (send 1 (json_encode_assoc '("error" "could not authenticate to avatar"))))
)
(print "unhandled message: " (json_encode msg))
)
)) (lambda () (print "player has left"))))
(send 1 "\"Hello World from server\"")
)
/* default */
(old_handler req res))
))
))
/* '''read'''  http_handler fresh from the '''environment''' */
(set port (env "PORT" "8001"))
(serve port (lambda (req res) (http_handler req res)))
(print "listening on <nowiki>http://localhost</nowiki>:" port)
You can execute this code via <code>memcp my_script.scm</code>

Revision as of 11:59, 28 August 2026

Websockets in MemCP

An embedded HTTP handler can upgrade a request through the response object's WebSocket facility and receive messages through a callback. The apps/minigame.scm example in the repository demonstrates the current API and routing pattern.

WebSockets keep one bidirectional connection open for live dashboards, notifications, collaborative state, games, or change feeds. They are a delivery channel, not a durable message queue: reconnect, replay, acknowledgement, ordering, and missed-message semantics belong to the application.

Server-side upgrade

<syntaxhighlight lang="scheme"> (set send ((res "websocket") (lambda (message) (print "message: " message)) (lambda () (print "connection closed")))) (send 1 "Hello from MemCP") </syntaxhighlight>

The numeric first argument is the WebSocket frame opcode used by the current response API. Keep the sending function only for the connection lifetime.

A browser client can connect to the routed endpoint with the standard API:

<syntaxhighlight lang="javascript"> const socket = new WebSocket("wss://example.test/minigame/ws"); socket.addEventListener("open", () => socket.send(JSON.stringify({type: "hello"}))); socket.addEventListener("message", event => console.log(JSON.parse(event.data))); </syntaxhighlight>

Complete routed example

The repository's apps/minigame.scm shows the complete routing pattern: preserve the previous handler, serve an HTML/JavaScript client below one prefix, and upgrade only the WebSocket path.

<syntaxhighlight lang="scheme"> (define http_handler (begin (set old_handler http_handler) (lambda (req res) (begin (match (req "path") "/minigame/ws" (begin (set send ((res "websocket") (lambda (message) (begin (print "message: " message) (send 1 (concat "echo: " message)))))) (send 1 "Hello from MemCP")) (regex "^/minigame/(.*)$" path asset) (begin ((res "header") "Content-Type" "text/plain") ((res "status") 200) ((res "print") "WebSocket client assets belong here")) (old_handler req res)))))) </syntaxhighlight>

When this module is imported by lib/main.scm, the existing server uses the wrapped handler. A standalone application can call serve explicitly. Keep the receive callback and its connection-local send closure together; never store that closure after close.

Prefer JSON messages with an explicit type and version. Validate the decoded shape before reading fields, cap message size, and define how the client resubscribes or resumes after reconnect.

Database-backed publish/subscribe

Prepare fixed queries during module startup, create a session per connection, and bind authenticated identity and subscription state there. A receive callback can update state atomically; query results can be emitted through send. On close, release subscriptions and other references so they do not keep sessions or sending functions alive.

For fan-out, use bounded per-client queues and a policy for slow consumers (drop/coalesce messages or disconnect). A database table can persist events for replay, but a memory or cache table cannot provide durable delivery after restart.

The earlier hardlife.io example used this structure at larger scale: prepare position-update, listener-membership and dirty-list queries once; authenticate a player handle with a message authentication code; keep the connection's player/location state in a session; mark affected listeners dirty after a position change; send only changed avatars; and remove listener state on disconnect. The complete historical listing coupled credentials, global sloppy defaults and application-specific tables, so reproducing it as a copy-and-paste server would be unsafe. The pipeline remains a useful design example.

Hosting the handler beside the database removes an application/database socket hop and repeated parsing for prepared shapes. That is a performance opportunity, not proof that every embedded WebSocket service is faster than a separate Node.js or Go service; benchmark the authenticated end-to-end path and account for the shared failure domain.

Production considerations

Authenticate the upgrade, validate browser Origin, limit message and queue sizes, apply backpressure, and remove subscriptions when the connection closes. A slow client must not create an unbounded in-memory backlog. Long database operations should observe cancellation. Use TLS at a trusted proxy or transport layer and do not expose default credentials.

Storage ENGINE choice is independent of WebSocket delivery: sloppy can lose recent writes, and memory/cache are not durable. See In-Database WebApps and REST Services and Persistency and Performance Guarantees.

Retrieved from "https://www.memcp.org/index.php?title=Websockets_in_MemCP&oldid=366"