Hardware Requirements

From MemCP
Revision as of 21:41, 18 May 2024 by Carli (talk | contribs) (→‎Micro-Benchmark)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

MemCP can run on any hardware that is supported by the go compiler.

To find the right hardware dimensions, you need to consider the following:

  • The process itself needs at least 8 MiB of RAM
  • The size of a 5 GiB MySQL database fits into ~1 GiB of RAM
  • To have a decent working set, take the RAM requirements x3, so a 5 GiB MySQL database should run on 3 GiB RAM
  • MemCP needs at least 1 CPU core, more is better ;)

Running on Big SMP Machines

MemCP runs best on machiens that have a lot of RAM and CPU cores.

We tested on:

Micro-Benchmark

This is measured on AMD Ryzen 9 7900X3D 12-Core Processor over a Gigabit connection: 

~$ ab -n 1000000 -c 1000 'http://192.168.178.108:4321/sql/test/select%20*%20from%20a'
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.178.108 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests


Server Software:        
Server Hostname:        192.168.178.108
Server Port:            4321

Document Path:          /sql/test/select%20*%20from%20a
Document Length:        12 bytes

Concurrency Level:      1000
Time taken for tests:   48.842 seconds
Complete requests:      1000000
Failed requests:        0
Non-2xx responses:      1000000
Total transferred:      180000000 bytes
HTML transferred:       12000000 bytes
Requests per second:    20474.09 [#/sec] (mean)
Time per request:       48.842 [ms] (mean)
Time per request:       0.049 [ms] (mean, across all concurrent requests)
Transfer rate:          3598.96 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   11   6.0     11      32
Processing:     0   38  21.3     33     279
Waiting:        0   33  21.9     27     278
Total:          0   49  21.0     46     287

Percentage of the requests served within a certain time (ms)
  50%     46
  66%     52
  75%     57
  80%     61
  90%     73
  95%     87
  98%    107
  99%    122
 100%    287 (longest request)

Running on Raspberry Pi

These are our recommendations to run MemCP on Raspberry Pi:

  • Avoid ENGINE=safe. To have a long life for your flash drive, you should choose ENGINE=sloppy. (ENGINE=logging would be a compromise between safe and sloppy) - for more information see Persistency and Performance Guarantees
  • Raspberry Pis have very limited RAM. In most cases, the RAM is sufficient though since most databases only take a few megabytes of RAM. For bigger data, you should wait for MMaped shards, see also: Current Status and open issues

Micro-Benchmark

Raspberry Pi 3

Here's a micro benchmark for putting HTTP load on memcp on Raspberry Pi 3: behind a 100 MBit/s ethernet. The request is a rather simple one. 

$ ab -n 1000000 -c 1000 'http://192.168.178.73:4321/sql/test/select%20*%20from%20a'
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.178.73 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests


Server Software:        
Server Hostname:        192.168.178.73
Server Port:            4321

Document Path:          /sql/test/select%20*%20from%20a
Document Length:        12 bytes

Concurrency Level:      1000
Time taken for tests:   1079.876 seconds
Complete requests:      1000000
Failed requests:        0
Non-2xx responses:      1000000
Total transferred:      180000000 bytes
HTML transferred:       12000000 bytes
Requests per second:    926.03 [#/sec] (mean)
Time per request:       1079.876 [ms] (mean)
Time per request:       1.080 [ms] (mean, across all concurrent requests)
Transfer rate:          162.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1  346 404.9    239    7464
Processing:     3  733 574.8    523    8723
Waiting:        3  728 573.5    518    8723
Total:          4 1079 685.0    818    9758

Percentage of the requests served within a certain time (ms)
  50%    818
  66%   1195
  75%   1489
  80%   1612
  90%   1956
  95%   2425
  98%   2920
  99%   3348
 100%   9758 (longest request)

Raspberry Pi manages to do about 1,000 requests per second while modern x86_64 computers are able to serve 12,000 requests and more.

Raspberry Pi 4

Disclaimer: Those two benchmarks are not comparable. The Raspi 3 benchmark requested a 401 page while this benchmark logs into a database and does a minimal query: 

$ ab -n 10000 -c 600 -A root:admin 'http://192.168.178.189:4321/sql/oppelbi/select%201'
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.178.189 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        
Server Hostname:        192.168.178.189
Server Port:            4321

Document Path:          /sql/oppelbi/select%201
Document Length:        9 bytes

Concurrency Level:      600
Time taken for tests:   23.173 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      1100000 bytes
HTML transferred:       90000 bytes
Requests per second:    431.54 [#/sec] (mean)
Time per request:       1390.375 [ms] (mean)
Time per request:       2.317 [ms] (mean, across all concurrent requests)
Transfer rate:          46.36 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3  32.8      1    1024
Processing:     7 1350 1101.9   1109    8906
Waiting:        7 1348 1101.4   1106    8906
Total:          8 1353 1102.9   1110    8908

Percentage of the requests served within a certain time (ms)
  50%   1110
  66%   1572
  75%   1902
  80%   2143
  90%   2847
  95%   3511
  98%   4329
  99%   4866
 100%   8908 (longest request)
Retrieved from "https://www.memcp.org/index.php?title=Hardware_Requirements&oldid=79"