Hardware Requirements
Jump to navigation
Jump to search
MemCP can run on any hardware that is supported by the go compiler.
To find the right hardware dimensions, you need to consider the following:
- The process itself needs at least 8 MiB of RAM
- The size of a 5 GiB MySQL database fits into ~1 GiB of RAM
- To have a decent working set, take the RAM requirements x3, so a 5 GiB MySQL database should run on 3 GiB RAM
- MemCP needs at least 1 CPU core, more is better ;)
Running on Big SMP Machines
MemCP runs best on machiens that have a lot of RAM and CPU cores.
We tested on:
- AMD Ryzen (in various forms)
- HPE Superfome Flex (https://www.hpe.com/us/en/servers/superdome.html)
Micro-Benchmark
This is measured on AMD Ryzen 9 7900X3D 12-Core Processor over a Gigabit connection:
~$ ab -n 1000000 -c 1000 'http://192.168.178.108:4321/sql/test/select%20*%20from%20a'
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.178.108 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests
Server Software:
Server Hostname: 192.168.178.108
Server Port: 4321
Document Path: /sql/test/select%20*%20from%20a
Document Length: 12 bytes
Concurrency Level: 1000
Time taken for tests: 48.842 seconds
Complete requests: 1000000
Failed requests: 0
Non-2xx responses: 1000000
Total transferred: 180000000 bytes
HTML transferred: 12000000 bytes
Requests per second: 20474.09 [#/sec] (mean)
Time per request: 48.842 [ms] (mean)
Time per request: 0.049 [ms] (mean, across all concurrent requests)
Transfer rate: 3598.96 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 11 6.0 11 32
Processing: 0 38 21.3 33 279
Waiting: 0 33 21.9 27 278
Total: 0 49 21.0 46 287
Percentage of the requests served within a certain time (ms)
50% 46
66% 52
75% 57
80% 61
90% 73
95% 87
98% 107
99% 122
100% 287 (longest request)
Running on Raspberry Pi
These are our recommendations to run MemCP on Raspberry Pi:
- Avoid
ENGINE=safe. To have a long life for your flash drive, you should chooseENGINE=sloppy. (ENGINE=loggingwould be a compromise betweensafeandsloppy) - for more information see Persistency and Performance Guarantees - Raspberry Pis have very limited RAM. In most cases, the RAM is sufficient though since most databases only take a few megabytes of RAM. For bigger data, you should wait for MMaped shards, see also: Current Status and open issues
Micro-Benchmark
Raspberry Pi 3
Here's a micro benchmark for putting HTTP load on memcp on Raspberry Pi 3: behind a 100 MBit/s ethernet. The request is a rather simple one.
$ ab -n 1000000 -c 1000 'http://192.168.178.73:4321/sql/test/select%20*%20from%20a'
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.178.73 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests
Server Software:
Server Hostname: 192.168.178.73
Server Port: 4321
Document Path: /sql/test/select%20*%20from%20a
Document Length: 12 bytes
Concurrency Level: 1000
Time taken for tests: 1079.876 seconds
Complete requests: 1000000
Failed requests: 0
Non-2xx responses: 1000000
Total transferred: 180000000 bytes
HTML transferred: 12000000 bytes
Requests per second: 926.03 [#/sec] (mean)
Time per request: 1079.876 [ms] (mean)
Time per request: 1.080 [ms] (mean, across all concurrent requests)
Transfer rate: 162.78 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 346 404.9 239 7464
Processing: 3 733 574.8 523 8723
Waiting: 3 728 573.5 518 8723
Total: 4 1079 685.0 818 9758
Percentage of the requests served within a certain time (ms)
50% 818
66% 1195
75% 1489
80% 1612
90% 1956
95% 2425
98% 2920
99% 3348
100% 9758 (longest request)
Raspberry Pi manages to do about 1,000 requests per second while modern x86_64 computers are able to serve 12,000 requests and more.
Raspberry Pi 4
Disclaimer: Those two benchmarks are not comparable. The Raspi 3 benchmark requested a 401 page while this benchmark logs into a database and does a minimal query:
$ ab -n 10000 -c 600 -A root:admin 'http://192.168.178.189:4321/sql/oppelbi/select%201'
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.178.189 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software:
Server Hostname: 192.168.178.189
Server Port: 4321
Document Path: /sql/oppelbi/select%201
Document Length: 9 bytes
Concurrency Level: 600
Time taken for tests: 23.173 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 1100000 bytes
HTML transferred: 90000 bytes
Requests per second: 431.54 [#/sec] (mean)
Time per request: 1390.375 [ms] (mean)
Time per request: 2.317 [ms] (mean, across all concurrent requests)
Transfer rate: 46.36 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 3 32.8 1 1024
Processing: 7 1350 1101.9 1109 8906
Waiting: 7 1348 1101.4 1106 8906
Total: 8 1353 1102.9 1110 8908
Percentage of the requests served within a certain time (ms)
50% 1110
66% 1572
75% 1902
80% 2143
90% 2847
95% 3511
98% 4329
99% 4866
100% 8908 (longest request)