Hardware Requirements: Difference between revisions

From MemCP
Jump to navigation Jump to search
No edit summary
Line 15: Line 15:
 
* AMD Ryzen (in various forms)
  
* AMD Ryzen (in various forms)
 
* HPE Superfome Flex (https://www.hpe.com/us/en/servers/superdome.html)
  
* HPE Superfome Flex (https://www.hpe.com/us/en/servers/superdome.html)
  +
  +
=== Micro-Benchmark ===
  +
This is measured on AMD Ryzen 9 7900X3D 12-Core Processor over a Gigabit connection:
  +
'''~'''$ ab -n 1000000 -c 1000 '<nowiki>http://192.168.178.108:4321/sql/test/select%20*%20from%20a'</nowiki>
  +
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
  +
Copyright 1996 Adam Twiss, Zeus Technology Ltd, <nowiki>http://www.zeustech.net/</nowiki>
  +
Licensed to The Apache Software Foundation, <nowiki>http://www.apache.org/</nowiki>
  +
  +
Benchmarking 192.168.178.108 (be patient)
  +
Completed 100000 requests
  +
Completed 200000 requests
  +
Completed 300000 requests
  +
Completed 400000 requests
  +
Completed 500000 requests
  +
Completed 600000 requests
  +
Completed 700000 requests
  +
Completed 800000 requests
  +
Completed 900000 requests
  +
Completed 1000000 requests
  +
Finished 1000000 requests
  +
  +
  +
Server Software:
  +
Server Hostname: 192.168.178.108
  +
Server Port: 4321
  +
  +
Document Path: /sql/test/select%20*%20from%20a
  +
Document Length: 12 bytes
  +
  +
Concurrency Level: 1000
  +
Time taken for tests: 48.842 seconds
  +
Complete requests: 1000000
  +
Failed requests: 0
  +
Non-2xx responses: 1000000
  +
Total transferred: 180000000 bytes
  +
HTML transferred: 12000000 bytes
  +
Requests per second: 20474.09 [#/sec] (mean)
  +
Time per request: 48.842 [ms] (mean)
  +
Time per request: 0.049 [ms] (mean, across all concurrent requests)
  +
Transfer rate: 3598.96 [Kbytes/sec] received
  +
  +
Connection Times (ms)
  +
min mean[+/-sd] median max
  +
Connect: 0 11 6.0 11 32
  +
Processing: 0 38 21.3 33 279
  +
Waiting: 0 33 21.9 27 278
  +
Total: 0 49 21.0 46 287
  +
  +
Percentage of the requests served within a certain time (ms)
  +
50% 46
  +
66% 52
  +
75% 57
  +
80% 61
  +
90% 73
  +
95% 87
  +
98% 107
  +
99% 122
  +
100% 287 (longest request)
   
 
== Running on Raspberry Pi ==
  
== Running on Raspberry Pi ==
Line 21: Line 79:
 
* Avoid <code>ENGINE=safe</code>. To have a long life for your flash drive, you should choose <code>ENGINE=sloppy</code>. (<code>ENGINE=logging</code> would be a compromise between <code>safe</code> and <code>sloppy</code>) - for more information see [[Persistency and Performance Guarantees]]
  
* Avoid <code>ENGINE=safe</code>. To have a long life for your flash drive, you should choose <code>ENGINE=sloppy</code>. (<code>ENGINE=logging</code> would be a compromise between <code>safe</code> and <code>sloppy</code>) - for more information see [[Persistency and Performance Guarantees]]
 
* Raspberry Pis have very limited RAM. In most cases, the RAM is sufficient though since most databases only take a few megabytes of RAM. For bigger data, you should wait for MMaped shards, see also: [[Current Status and open issues]]
  
* Raspberry Pis have very limited RAM. In most cases, the RAM is sufficient though since most databases only take a few megabytes of RAM. For bigger data, you should wait for MMaped shards, see also: [[Current Status and open issues]]
  +
  +
=== Micro-Benchmark ===
  +
Here's a micro benchmark for putting HTTP load on memcp on Raspberry Pi: behind a 100 MBit/s ethernet. The request is a rather simple one.
  +
$ ab -n 1000000 -c 1000 '<nowiki>http://192.168.178.73:4321/sql/test/select%20*%20from%20a'</nowiki>
  +
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
  +
Copyright 1996 Adam Twiss, Zeus Technology Ltd, <nowiki>http://www.zeustech.net/</nowiki>
  +
Licensed to The Apache Software Foundation, <nowiki>http://www.apache.org/</nowiki>
  +
  +
Benchmarking 192.168.178.73 (be patient)
  +
Completed 100000 requests
  +
Completed 200000 requests
  +
Completed 300000 requests
  +
Completed 400000 requests
  +
Completed 500000 requests
  +
Completed 600000 requests
  +
Completed 700000 requests
  +
Completed 800000 requests
  +
Completed 900000 requests
  +
Completed 1000000 requests
  +
Finished 1000000 requests
  +
  +
  +
Server Software:
  +
Server Hostname: 192.168.178.73
  +
Server Port: 4321
  +
  +
Document Path: /sql/test/select%20*%20from%20a
  +
Document Length: 12 bytes
  +
  +
Concurrency Level: 1000
  +
Time taken for tests: 1079.876 seconds
  +
Complete requests: 1000000
  +
Failed requests: 0
  +
Non-2xx responses: 1000000
  +
Total transferred: 180000000 bytes
  +
HTML transferred: 12000000 bytes
  +
Requests per second: 926.03 [#/sec] (mean)
  +
Time per request: 1079.876 [ms] (mean)
  +
Time per request: 1.080 [ms] (mean, across all concurrent requests)
  +
Transfer rate: 162.78 [Kbytes/sec] received
  +
  +
Connection Times (ms)
  +
min mean[+/-sd] median max
  +
Connect: 1 346 404.9 239 7464
  +
Processing: 3 733 574.8 523 8723
  +
Waiting: 3 728 573.5 518 8723
  +
Total: 4 1079 685.0 818 9758
  +
  +
Percentage of the requests served within a certain time (ms)
  +
50% 818
  +
66% 1195
  +
75% 1489
  +
80% 1612
  +
90% 1956
  +
95% 2425
  +
98% 2920
  +
99% 3348
  +
100% 9758 (longest request)
  +
Raspberry Pi manages to do about 1,000 requests per second while modern x86_64 computers are able to serve 12,000 requests and more.

Revision as of 17:41, 18 May 2024

MemCP can run on any hardware that is supported by the go compiler.

To find the right hardware dimensions, you need to consider the following:

  • The process itself needs at least 8 MiB of RAM
  • The size of a 5 GiB MySQL database fits into ~1 GiB of RAM
  • To have a decent working set, take the RAM requirements x3, so a 5 GiB MySQL database should run on 3 GiB RAM
  • MemCP needs at least 1 CPU core, more is better ;)

Running on Big SMP Machines

MemCP runs best on machiens that have a lot of RAM and CPU cores.

We tested on:

Micro-Benchmark

This is measured on AMD Ryzen 9 7900X3D 12-Core Processor over a Gigabit connection: 

~$ ab -n 1000000 -c 1000 'http://192.168.178.108:4321/sql/test/select%20*%20from%20a'
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.178.108 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests


Server Software:        
Server Hostname:        192.168.178.108
Server Port:            4321

Document Path:          /sql/test/select%20*%20from%20a
Document Length:        12 bytes

Concurrency Level:      1000
Time taken for tests:   48.842 seconds
Complete requests:      1000000
Failed requests:        0
Non-2xx responses:      1000000
Total transferred:      180000000 bytes
HTML transferred:       12000000 bytes
Requests per second:    20474.09 [#/sec] (mean)
Time per request:       48.842 [ms] (mean)
Time per request:       0.049 [ms] (mean, across all concurrent requests)
Transfer rate:          3598.96 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   11   6.0     11      32
Processing:     0   38  21.3     33     279
Waiting:        0   33  21.9     27     278
Total:          0   49  21.0     46     287

Percentage of the requests served within a certain time (ms)
  50%     46
  66%     52
  75%     57
  80%     61
  90%     73
  95%     87
  98%    107
  99%    122
 100%    287 (longest request)

Running on Raspberry Pi

These are our recommendations to run MemCP on Raspberry Pi:

  • Avoid ENGINE=safe. To have a long life for your flash drive, you should choose ENGINE=sloppy. (ENGINE=logging would be a compromise between safe and sloppy) - for more information see Persistency and Performance Guarantees
  • Raspberry Pis have very limited RAM. In most cases, the RAM is sufficient though since most databases only take a few megabytes of RAM. For bigger data, you should wait for MMaped shards, see also: Current Status and open issues

Micro-Benchmark

Here's a micro benchmark for putting HTTP load on memcp on Raspberry Pi: behind a 100 MBit/s ethernet. The request is a rather simple one. 

$ ab -n 1000000 -c 1000 'http://192.168.178.73:4321/sql/test/select%20*%20from%20a'
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.178.73 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests


Server Software:        
Server Hostname:        192.168.178.73
Server Port:            4321

Document Path:          /sql/test/select%20*%20from%20a
Document Length:        12 bytes

Concurrency Level:      1000
Time taken for tests:   1079.876 seconds
Complete requests:      1000000
Failed requests:        0
Non-2xx responses:      1000000
Total transferred:      180000000 bytes
HTML transferred:       12000000 bytes
Requests per second:    926.03 [#/sec] (mean)
Time per request:       1079.876 [ms] (mean)
Time per request:       1.080 [ms] (mean, across all concurrent requests)
Transfer rate:          162.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1  346 404.9    239    7464
Processing:     3  733 574.8    523    8723
Waiting:        3  728 573.5    518    8723
Total:          4 1079 685.0    818    9758

Percentage of the requests served within a certain time (ms)
  50%    818
  66%   1195
  75%   1489
  80%   1612
  90%   1956
  95%   2425
  98%   2920
  99%   3348
 100%   9758 (longest request)

Raspberry Pi manages to do about 1,000 requests per second while modern x86_64 computers are able to serve 12,000 requests and more.

Retrieved from "https://www.memcp.org/index.php?title=Hardware_Requirements&oldid=77"