В одной из предыдущих статей я привел результаты тестирования дешевых виртуальных серверов от различных хостеров рунета. Спасибо всем комментаторам и людям, писавшим в личные сообщения, за обратную связь.

В этот раз я хочу представить результаты тестирования виртуальных серверов от известных и крупных компаний: DigitalOcean, Vultr, Linode и Hetzner. Сделано 38 тестов для всех доступных локаций.



Хотя характеристики виртуальных серверов на этот раз очень близки, но все же стоит напомнить, что формат тестирования не слишком объективен, и результаты стоит рассматривать с определенной долей иронии, так как измеренные величины весьма относительны.

Как и в прошлый раз тестирование проводилось при помощи unixbench, dd, fio, speedtest, работающих на Debian 9, параметры не менялись (два теста dd: блоками по 64K и 1M; четыре теста fio: в режимах random read, random direct read, random write и random direct write).

Описание тарифов и цены:

Хостер Объем диска / Ресурсы процессора / Оперативная память Цена
DigitalOcean 25 GB SSD
CA, DE, IN, NL, US (CA) — 1x 2.2 GHz (Intel Xeon E5-2650v4);
SG, UK, US (NY) — 1x 2.3 GHz (Intel Xeon Gold 6140)
1 GB (MemTotal: 1020380 kB)
$5 / месяц (~313 рублей)
Vultr 25 GB SSD
AU, DE, FR, JP, NL, SG, UK, US (GA, IL, TX, CA, FL, NJ, WA) — 1x 2.4 GHz (vCPU);
CA — 1x 2.6 GHz (vCPU)
1 GB (MemTotal: 1020268 kB)
$5 / месяц (~313 рублей)
Linode 25 GB SSD
AU, US (NJ) — 1x 2.2 GHz (AMD EPYC 7601);
CA, DE, IN, US (GA) — 1x 2 GHz (AMD EPYC 7501);
JP, SG, US (CA, TX) — 1x 2.3 GHz (Intel Xeon E5-2697v4);
UK — 1x 2.5 GHz (Intel Xeon E5-2680v3);
1 GB (MemTotal: 1020264 kB)
$5 / месяц (~313 рублей)
Hetzner 20 GB NVMe SSD
DE (Фалькенштейн) — 1x 2.3 GHz (Intel Xeon Skylake);
DE (Нюрнберг), FI — 1x 2.1 GHz (Intel Xeon Skylake)
2 GB (MemTotal: 2003300 kB)
2.99€ / месяц (~208 рублей)


Результаты тестов находятся под спойлерами, а графики — в конце статьи.

Виртуальный сервер от DigitalOcean:




  • тариф Standard Droplet (25 GB SSD / 1x vCPU / 1 GB RAM / 1000 Mbit/s за $5 в месяц);
  • дата-центры в Великобритании, Германии, Канаде, Индии, Нидерландах, Сингапуре, США.

Тесты:


Unixbench (Канада, Торонто)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: debian-s-1vcpu-1gb-tor1-01: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (4400.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   15:52:08 up 6 min,  1 user,  load average: 0.86, 0.44, 0.19; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Sun Dec 15 2019 15:52:08 - 16:19:36
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       17067275.6 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3581.7 MWIPS (6.4 s, 7 samples)
Execl Throughput                               2166.6 lps   (29.7 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        244701.5 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks           71178.2 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks        663117.5 KBps  (30.0 s, 2 samples)
Pipe Throughput                              334042.9 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  58850.2 lps   (10.0 s, 7 samples)
Process Creation                               4662.5 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   2937.6 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    393.6 lpm   (60.1 s, 2 samples)
System Call Overhead                         230331.4 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   17067275.6   1462.5
Double-Precision Whetstone                       55.0       3581.7    651.2
Execl Throughput                                 43.0       2166.6    503.9
File Copy 1024 bufsize 2000 maxblocks          3960.0     244701.5    617.9
File Copy 256 bufsize 500 maxblocks            1655.0      71178.2    430.1
File Copy 4096 bufsize 8000 maxblocks          5800.0     663117.5   1143.3
Pipe Throughput                               12440.0     334042.9    268.5
Pipe-based Context Switching                   4000.0      58850.2    147.1
Process Creation                                126.0       4662.5    370.0
Shell Scripts (1 concurrent)                     42.4       2937.6    692.8
Shell Scripts (8 concurrent)                      6.0        393.6    656.0
System Call Overhead                          15000.0     230331.4    153.6
                                                                   ========
System Benchmarks Index Score                                         479.8


Unixbench (Германия, Франкфурт)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: debian-s-1vcpu-1gb-fra1-01: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (4400.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   23:37:04 up 3 min,  1 user,  load average: 0.66, 0.24, 0.09; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Fri Dec 13 2019 23:37:04 - 00:05:10
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       26912014.0 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4483.5 MWIPS (9.9 s, 7 samples)
Execl Throughput                               3672.5 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        463711.6 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          126002.9 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1201991.6 KBps  (30.0 s, 2 samples)
Pipe Throughput                              630949.6 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  99295.5 lps   (10.0 s, 7 samples)
Process Creation                               8599.9 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   6028.7 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    780.4 lpm   (60.0 s, 2 samples)
System Call Overhead                         355362.1 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   26912014.0   2306.1
Double-Precision Whetstone                       55.0       4483.5    815.2
Execl Throughput                                 43.0       3672.5    854.1
File Copy 1024 bufsize 2000 maxblocks          3960.0     463711.6   1171.0
File Copy 256 bufsize 500 maxblocks            1655.0     126002.9    761.3
File Copy 4096 bufsize 8000 maxblocks          5800.0    1201991.6   2072.4
Pipe Throughput                               12440.0     630949.6    507.2
Pipe-based Context Switching                   4000.0      99295.5    248.2
Process Creation                                126.0       8599.9    682.5
Shell Scripts (1 concurrent)                     42.4       6028.7   1421.9
Shell Scripts (8 concurrent)                      6.0        780.4   1300.6
System Call Overhead                          15000.0     355362.1    236.9
                                                                   ========
System Benchmarks Index Score                                         833.0


Unixbench (Индия, Бангалор)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: debian-s-1vcpu-1gb-blr1-01: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (4400.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   18:52:38 up 3 min,  1 user,  load average: 0.66, 0.24, 0.09; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Sun Dec 15 2019 18:52:38 - 19:20:46
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       23572479.8 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4128.6 MWIPS (10.0 s, 7 samples)
Execl Throughput                               3005.2 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        350138.9 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks           95265.1 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks        982227.6 KBps  (30.0 s, 2 samples)
Pipe Throughput                              532053.3 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  85820.9 lps   (10.0 s, 7 samples)
Process Creation                               7340.8 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   4636.3 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    597.8 lpm   (60.1 s, 2 samples)
System Call Overhead                         308446.2 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   23572479.8   2019.9
Double-Precision Whetstone                       55.0       4128.6    750.6
Execl Throughput                                 43.0       3005.2    698.9
File Copy 1024 bufsize 2000 maxblocks          3960.0     350138.9    884.2
File Copy 256 bufsize 500 maxblocks            1655.0      95265.1    575.6
File Copy 4096 bufsize 8000 maxblocks          5800.0     982227.6   1693.5
Pipe Throughput                               12440.0     532053.3    427.7
Pipe-based Context Switching                   4000.0      85820.9    214.6
Process Creation                                126.0       7340.8    582.6
Shell Scripts (1 concurrent)                     42.4       4636.3   1093.5
Shell Scripts (8 concurrent)                      6.0        597.8    996.3
System Call Overhead                          15000.0     308446.2    205.6
                                                                   ========
System Benchmarks Index Score                                         686.4


Unixbench (Нидерланды, Амстердам)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: debian-s-1vcpu-1gb-ams3-01: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (4400.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   14:12:24 up 6 min,  1 user,  load average: 0.90, 0.34, 0.12; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Wed Dec 11 2019 14:12:24 - 14:40:37
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       18016981.9 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3729.2 MWIPS (10.1 s, 7 samples)
Execl Throughput                               2416.3 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        283772.5 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks           76113.2 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks        826905.7 KBps  (30.0 s, 2 samples)
Pipe Throughput                              461320.6 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  71653.1 lps   (10.0 s, 7 samples)
Process Creation                               5727.0 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   3489.5 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    445.3 lpm   (60.2 s, 2 samples)
System Call Overhead                         276916.2 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   18016981.9   1543.9
Double-Precision Whetstone                       55.0       3729.2    678.0
Execl Throughput                                 43.0       2416.3    561.9
File Copy 1024 bufsize 2000 maxblocks          3960.0     283772.5    716.6
File Copy 256 bufsize 500 maxblocks            1655.0      76113.2    459.9
File Copy 4096 bufsize 8000 maxblocks          5800.0     826905.7   1425.7
Pipe Throughput                               12440.0     461320.6    370.8
Pipe-based Context Switching                   4000.0      71653.1    179.1
Process Creation                                126.0       5727.0    454.5
Shell Scripts (1 concurrent)                     42.4       3489.5    823.0
Shell Scripts (8 concurrent)                      6.0        445.3    742.2
System Call Overhead                          15000.0     276916.2    184.6
                                                                   ========
System Benchmarks Index Score                                         559.5


Unixbench (Сингапур)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: debian-s-1vcpu-1gb-sgp1-01: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (4589.2 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   18:08:59 up 3 min,  1 user,  load average: 0.75, 0.27, 0.09; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Wed Dec 11 2019 18:08:59 - 18:37:08
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       26210094.9 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3688.2 MWIPS (9.9 s, 7 samples)
Execl Throughput                               3280.2 lps   (29.9 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        395957.3 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          105354.4 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1118194.8 KBps  (30.0 s, 2 samples)
Pipe Throughput                              627972.2 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  86900.0 lps   (10.0 s, 7 samples)
Process Creation                               7904.4 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   4941.1 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    631.0 lpm   (60.1 s, 2 samples)
System Call Overhead                         368192.7 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   26210094.9   2245.9
Double-Precision Whetstone                       55.0       3688.2    670.6
Execl Throughput                                 43.0       3280.2    762.8
File Copy 1024 bufsize 2000 maxblocks          3960.0     395957.3    999.9
File Copy 256 bufsize 500 maxblocks            1655.0     105354.4    636.6
File Copy 4096 bufsize 8000 maxblocks          5800.0    1118194.8   1927.9
Pipe Throughput                               12440.0     627972.2    504.8
Pipe-based Context Switching                   4000.0      86900.0    217.3
Process Creation                                126.0       7904.4    627.3
Shell Scripts (1 concurrent)                     42.4       4941.1   1165.3
Shell Scripts (8 concurrent)                      6.0        631.0   1051.6
System Call Overhead                          15000.0     368192.7    245.5
                                                                   ========
System Benchmarks Index Score                                         744.9


Unixbench (Великобритания, Лондон)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: debian-s-1vcpu-1gb-lon1-01: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (4589.2 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   12:30:15 up 8 min,  1 user,  load average: 0.71, 0.24, 0.08; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Thu Dec 12 2019 12:30:15 - 12:58:24
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       31577703.9 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4164.5 MWIPS (9.8 s, 7 samples)
Execl Throughput                               3746.2 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        449769.6 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          127650.7 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1238103.8 KBps  (30.0 s, 2 samples)
Pipe Throughput                              684221.5 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  97428.3 lps   (10.0 s, 7 samples)
Process Creation                               9107.5 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   5976.2 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    746.2 lpm   (60.0 s, 2 samples)
System Call Overhead                         405652.2 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   31577703.9   2705.9
Double-Precision Whetstone                       55.0       4164.5    757.2
Execl Throughput                                 43.0       3746.2    871.2
File Copy 1024 bufsize 2000 maxblocks          3960.0     449769.6   1135.8
File Copy 256 bufsize 500 maxblocks            1655.0     127650.7    771.3
File Copy 4096 bufsize 8000 maxblocks          5800.0    1238103.8   2134.7
Pipe Throughput                               12440.0     684221.5    550.0
Pipe-based Context Switching                   4000.0      97428.3    243.6
Process Creation                                126.0       9107.5    722.8
Shell Scripts (1 concurrent)                     42.4       5976.2   1409.5
Shell Scripts (8 concurrent)                      6.0        746.2   1243.6
System Call Overhead                          15000.0     405652.2    270.4
                                                                   ========
System Benchmarks Index Score                                         855.3


Unixbench (США, Нью-Йорк)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: debian-s-1vcpu-1gb-nyc3-01: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (4589.2 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   13:10:48 up 4 min,  1 user,  load average: 0.61, 0.24, 0.09; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Wed Dec 04 2019 13:10:48 - 13:38:54
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       31830632.4 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4067.9 MWIPS (9.9 s, 7 samples)
Execl Throughput                               3892.2 lps   (29.5 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        484700.4 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          143546.1 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1306057.4 KBps  (30.0 s, 2 samples)
Pipe Throughput                              782935.5 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 109443.7 lps   (10.0 s, 7 samples)
Process Creation                              10168.3 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   6770.7 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    864.3 lpm   (60.0 s, 2 samples)
System Call Overhead                         470032.5 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   31830632.4   2727.6
Double-Precision Whetstone                       55.0       4067.9    739.6
Execl Throughput                                 43.0       3892.2    905.2
File Copy 1024 bufsize 2000 maxblocks          3960.0     484700.4   1224.0
File Copy 256 bufsize 500 maxblocks            1655.0     143546.1    867.3
File Copy 4096 bufsize 8000 maxblocks          5800.0    1306057.4   2251.8
Pipe Throughput                               12440.0     782935.5    629.4
Pipe-based Context Switching                   4000.0     109443.7    273.6
Process Creation                                126.0      10168.3    807.0
Shell Scripts (1 concurrent)                     42.4       6770.7   1596.9
Shell Scripts (8 concurrent)                      6.0        864.3   1440.5
System Call Overhead                          15000.0     470032.5    313.4
                                                                   ========
System Benchmarks Index Score                                         933.4


Unixbench (США, Сан-Франциско)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: debian-s-1vcpu-1gb-sfo2-01: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (4400.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   07:32:35 up 4 min,  1 user,  load average: 0.85, 0.38, 0.15; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Thu Dec 05 2019 07:32:35 - 08:00:35
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       18731484.4 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3866.3 MWIPS (9.2 s, 7 samples)
Execl Throughput                               2473.3 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        289169.4 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks           81647.3 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks        839008.9 KBps  (30.0 s, 2 samples)
Pipe Throughput                              464945.7 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  70296.3 lps   (10.0 s, 7 samples)
Process Creation                               6041.0 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   3583.0 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    458.9 lpm   (60.2 s, 2 samples)
System Call Overhead                         272373.3 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   18731484.4   1605.1
Double-Precision Whetstone                       55.0       3866.3    703.0
Execl Throughput                                 43.0       2473.3    575.2
File Copy 1024 bufsize 2000 maxblocks          3960.0     289169.4    730.2
File Copy 256 bufsize 500 maxblocks            1655.0      81647.3    493.3
File Copy 4096 bufsize 8000 maxblocks          5800.0     839008.9   1446.6
Pipe Throughput                               12440.0     464945.7    373.8
Pipe-based Context Switching                   4000.0      70296.3    175.7
Process Creation                                126.0       6041.0    479.4
Shell Scripts (1 concurrent)                     42.4       3583.0    845.0
Shell Scripts (8 concurrent)                      6.0        458.9    764.9
System Call Overhead                          15000.0     272373.3    181.6
                                                                   ========
System Benchmarks Index Score                                         572.9


dd (Канада, Торонто)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.32407 s, 340 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 8.07427 s, 266 MB/s


dd (Германия, Франкфурт)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.45694 s, 482 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.40443 s, 631 MB/s


dd (Индия, Бангалор)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.84399 s, 755 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.98814 s, 538 MB/s


dd (Нидерланды, Амстердам)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.25153 s, 505 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.97376 s, 432 MB/s


dd (Сингапур)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.79863 s, 565 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.49007 s, 615 MB/s


dd (Великобритания, Лондон)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 5.75593 s, 373 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 5.54156 s, 388 MB/s


dd (США, Нью-Йорк)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.72736 s, 576 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.41343 s, 629 MB/s


dd (США, Сан-Франциско)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.30885 s, 649 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.53563 s, 607 MB/s


fio (Канада, Торонто)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28463: Sun Dec 15 16:21:05 2019
  read : io=810904KB, bw=13512KB/s, iops=3378, runt= 60013msec
    clat (usec): min=170, max=562215, avg=2329.21, stdev=7399.56
     lat (usec): min=171, max=562216, avg=2334.20, stdev=7405.70
    clat percentiles (usec):
     |  1.00th=[  245],  5.00th=[  314], 10.00th=[  402], 20.00th=[  540],
     | 30.00th=[  644], 40.00th=[  764], 50.00th=[  924], 60.00th=[ 1160],
     | 70.00th=[ 1544], 80.00th=[ 2288], 90.00th=[ 4320], 95.00th=[ 7904],
     | 99.00th=[29568], 99.50th=[37632], 99.90th=[50944], 99.95th=[67072],
     | 99.99th=[382976]
    bw (KB  /s): min=   20, max= 3280, per=12.65%, avg=1709.08, stdev=752.64
    lat (usec) : 250=1.18%, 500=15.24%, 750=22.38%, 1000=14.99%
    lat (msec) : 2=23.20%, 4=12.09%, 10=7.21%, 20=2.10%, 50=1.50%
    lat (msec) : 100=0.07%, 250=0.02%, 500=0.01%, 750=0.01%
  cpu          : usr=1.30%, sys=5.68%, ctx=197641, majf=0, minf=85
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=202726/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=810904KB, aggrb=13512KB/s, minb=13512KB/s, maxb=13512KB/s, mint=60013msec, maxt=60013msec

Disk stats (read/write):
  vda: ios=203045/12, merge=0/12, ticks=250920/612, in_queue=251312, util=74.66%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28501: Sun Dec 15 16:23:17 2019
  write: io=6566.3MB, bw=111997KB/s, iops=27999, runt= 60036msec
    clat (usec): min=2, max=508874, avg=164.65, stdev=3238.47
     lat (usec): min=3, max=508875, avg=181.01, stdev=3480.05
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    5], 80.00th=[    6], 90.00th=[    7], 95.00th=[    8],
     | 99.00th=[   48], 99.50th=[ 7392], 99.90th=[40192], 99.95th=[55040],
     | 99.99th=[117248]
    bw (KB  /s): min=    0, max=139593, per=14.20%, avg=15900.11, stdev=8999.85
    lat (usec) : 4=0.41%, 10=95.61%, 20=2.47%, 50=0.52%, 100=0.13%
    lat (usec) : 250=0.16%, 500=0.05%, 750=0.03%, 1000=0.01%
    lat (msec) : 2=0.03%, 4=0.03%, 10=0.10%, 20=0.16%, 50=0.23%
    lat (msec) : 100=0.05%, 250=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=3.51%, sys=4.83%, ctx=29270, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1680965/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=6566.3MB, aggrb=111997KB/s, minb=111997KB/s, maxb=111997KB/s, mint=60036msec, maxt=60036msec

Disk stats (read/write):
  vda: ios=653/312108, merge=0/1864, ticks=5244/3962028, in_queue=3970048, util=85.40%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28488: Sun Dec 15 16:22:17 2019
  read : io=926632KB, bw=15443KB/s, iops=3860, runt= 60003msec
    clat (usec): min=158, max=342030, avg=2041.48, stdev=5221.60
     lat (usec): min=159, max=342031, avg=2044.90, stdev=5224.66
    clat percentiles (usec):
     |  1.00th=[  247],  5.00th=[  322], 10.00th=[  406], 20.00th=[  524],
     | 30.00th=[  612], 40.00th=[  732], 50.00th=[  868], 60.00th=[ 1080],
     | 70.00th=[ 1416], 80.00th=[ 2096], 90.00th=[ 3888], 95.00th=[ 6752],
     | 99.00th=[22656], 99.50th=[34048], 99.90th=[47872], 99.95th=[52992],
     | 99.99th=[199680]
    bw (KB  /s): min=   81, max= 3704, per=12.60%, avg=1946.40, stdev=778.52
    lat (usec) : 250=1.16%, 500=16.28%, 750=24.32%, 1000=15.06%
    lat (msec) : 2=22.14%, 4=11.33%, 10=6.79%, 20=1.75%, 50=1.11%
    lat (msec) : 100=0.05%, 250=0.01%, 500=0.01%
  cpu          : usr=1.31%, sys=5.34%, ctx=227680, majf=0, minf=84
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=231658/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=926632KB, aggrb=15443KB/s, minb=15443KB/s, maxb=15443KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  vda: ios=230719/8, merge=0/7, ticks=268276/96, in_queue=268104, util=76.22%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28524: Sun Dec 15 16:24:19 2019
  write: io=124888KB, bw=2080.2KB/s, iops=520, runt= 60037msec
    clat (usec): min=653, max=401729, avg=15355.31, stdev=39324.23
     lat (usec): min=655, max=401731, avg=15358.38, stdev=39325.85
    clat percentiles (usec):
     |  1.00th=[  764],  5.00th=[  884], 10.00th=[  988], 20.00th=[ 1128],
     | 30.00th=[ 1288], 40.00th=[ 1448], 50.00th=[ 1640], 60.00th=[ 1848],
     | 70.00th=[ 2160], 80.00th=[ 2896], 90.00th=[94720], 95.00th=[124416],
     | 99.00th=[152576], 99.50th=[166912], 99.90th=[207872], 99.95th=[268288],
     | 99.99th=[395264]
    bw (KB  /s): min=   17, max=  584, per=12.52%, avg=260.41, stdev=87.52
    lat (usec) : 750=0.73%, 1000=10.17%
    lat (msec) : 2=54.34%, 4=19.08%, 10=3.99%, 20=0.49%, 50=0.27%
    lat (msec) : 100=1.36%, 250=9.51%, 500=0.06%
  cpu          : usr=0.23%, sys=0.89%, ctx=66162, majf=0, minf=80
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=31222/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=124888KB, aggrb=2080KB/s, minb=2080KB/s, maxb=2080KB/s, mint=60037msec, maxt=60037msec

Disk stats (read/write):
  vda: ios=569/31224, merge=0/1484, ticks=648/54976, in_queue=55592, util=91.32%


fio (Германия, Франкфурт)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=22841: Sat Dec 14 00:06:23 2019
  read : io=5529.7MB, bw=94371KB/s, iops=23592, runt= 60001msec
    clat (usec): min=2, max=208020, avg=332.04, stdev=793.12
     lat (usec): min=3, max=208021, avg=333.14, stdev=793.14
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    5], 10.00th=[  187], 20.00th=[  213],
     | 30.00th=[  229], 40.00th=[  245], 50.00th=[  262], 60.00th=[  286],
     | 70.00th=[  314], 80.00th=[  366], 90.00th=[  490], 95.00th=[  644],
     | 99.00th=[ 2096], 99.50th=[ 2960], 99.90th=[ 4128], 99.95th=[ 4896],
     | 99.99th=[10176]
    bw (KB  /s): min= 5640, max=19344, per=12.53%, avg=11821.28, stdev=2384.67
    lat (usec) : 4=2.92%, 10=2.71%, 20=0.28%, 50=0.05%, 100=0.01%
    lat (usec) : 250=36.74%, 500=47.83%, 750=5.95%, 1000=1.29%
    lat (msec) : 2=1.16%, 4=0.94%, 10=0.11%, 20=0.01%, 50=0.01%
    lat (msec) : 250=0.01%
  cpu          : usr=2.40%, sys=5.64%, ctx=1335613, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1415595/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=5529.7MB, aggrb=94371KB/s, minb=94371KB/s, maxb=94371KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1330026/16, merge=31/12, ticks=381688/32, in_queue=381284, util=98.86%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=22875: Sat Dec 14 00:08:28 2019
  write: io=23387MB, bw=399054KB/s, iops=99763, runt= 60013msec
    clat (usec): min=2, max=347823, avg=53.51, stdev=1396.60
     lat (usec): min=2, max=348279, avg=58.09, stdev=1444.65
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    5], 95.00th=[    5],
     | 99.00th=[   13], 99.50th=[   16], 99.90th=[16320], 99.95th=[22656],
     | 99.99th=[36096]
    bw (KB  /s): min=    0, max=69737, per=13.03%, avg=52000.63, stdev=13148.08
    lat (usec) : 4=46.65%, 10=51.87%, 20=1.04%, 50=0.08%, 100=0.02%
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.02%, 10=0.09%, 20=0.13%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=4.38%, sys=5.58%, ctx=48485, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5987113/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=23387MB, aggrb=399054KB/s, minb=399054KB/s, maxb=399054KB/s, mint=60013msec, maxt=60013msec

Disk stats (read/write):
  vda: ios=269/703950, merge=0/5166, ticks=376/6209928, in_queue=6210116, util=97.19%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=22862: Sat Dec 14 00:07:28 2019
  read : io=5254.4MB, bw=89673KB/s, iops=22418, runt= 60001msec
    clat (usec): min=128, max=189254, avg=349.55, stdev=876.11
     lat (usec): min=129, max=189255, avg=350.71, stdev=876.13
    clat percentiles (usec):
     |  1.00th=[  169],  5.00th=[  187], 10.00th=[  199], 20.00th=[  219],
     | 30.00th=[  233], 40.00th=[  249], 50.00th=[  266], 60.00th=[  290],
     | 70.00th=[  322], 80.00th=[  374], 90.00th=[  494], 95.00th=[  644],
     | 99.00th=[ 2064], 99.50th=[ 2992], 99.90th=[ 4192], 99.95th=[ 5472],
     | 99.99th=[12992]
    bw (KB  /s): min= 4986, max=14872, per=12.50%, avg=11210.81, stdev=1832.67
    lat (usec) : 250=40.19%, 500=50.04%, 750=6.37%, 1000=1.22%
    lat (msec) : 2=1.15%, 4=0.90%, 10=0.11%, 20=0.02%, 50=0.01%
    lat (msec) : 250=0.01%
  cpu          : usr=2.30%, sys=5.08%, ctx=1349121, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1345114/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=5254.4MB, aggrb=89672KB/s, minb=89672KB/s, maxb=89672KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1340578/8, merge=0/7, ticks=396628/20, in_queue=396256, util=99.82%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=22896: Sat Dec 14 00:09:29 2019
  write: io=181408KB, bw=3022.6KB/s, iops=755, runt= 60018msec
    clat (usec): min=606, max=535365, avg=10574.01, stdev=38305.93
     lat (usec): min=607, max=535367, avg=10575.65, stdev=38305.97
    clat percentiles (usec):
     |  1.00th=[  660],  5.00th=[  700], 10.00th=[  732], 20.00th=[  812],
     | 30.00th=[  900], 40.00th=[  980], 50.00th=[ 1048], 60.00th=[ 1144],
     | 70.00th=[ 1304], 80.00th=[ 1624], 90.00th=[ 3152], 95.00th=[123392],
     | 99.00th=[183296], 99.50th=[218112], 99.90th=[305152], 99.95th=[358400],
     | 99.99th=[460800]
    bw (KB  /s): min=   14, max=  736, per=12.61%, avg=381.11, stdev=155.64
    lat (usec) : 750=12.63%, 1000=30.54%
    lat (msec) : 2=41.81%, 4=6.44%, 10=2.36%, 20=0.17%, 50=0.04%
    lat (msec) : 100=0.01%, 250=5.74%, 500=0.26%, 750=0.01%
  cpu          : usr=0.15%, sys=0.62%, ctx=93571, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=45352/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=181408KB, aggrb=3022KB/s, minb=3022KB/s, maxb=3022KB/s, mint=60018msec, maxt=60018msec

Disk stats (read/write):
  vda: ios=254/45392, merge=0/1838, ticks=272/56992, in_queue=57236, util=94.58%


fio (Индия, Бангалор)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=16212: Sun Dec 15 19:21:57 2019
  read : io=6139.3MB, bw=104771KB/s, iops=26192, runt= 60003msec
    clat (usec): min=2, max=168209, avg=297.31, stdev=431.72
     lat (usec): min=3, max=168210, avg=298.56, stdev=431.78
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[  189], 20.00th=[  213],
     | 30.00th=[  229], 40.00th=[  245], 50.00th=[  262], 60.00th=[  282],
     | 70.00th=[  310], 80.00th=[  354], 90.00th=[  442], 95.00th=[  540],
     | 99.00th=[  980], 99.50th=[ 1592], 99.90th=[ 3504], 99.95th=[ 4128],
     | 99.99th=[ 6752]
    bw (KB  /s): min= 9088, max=17808, per=12.49%, avg=13085.74, stdev=1494.25
    lat (usec) : 4=0.94%, 10=5.00%, 20=0.42%, 50=0.14%, 100=0.01%
    lat (usec) : 250=37.12%, 500=49.91%, 750=4.81%, 1000=0.69%
    lat (msec) : 2=0.60%, 4=0.30%, 10=0.05%, 20=0.01%, 50=0.01%
    lat (msec) : 250=0.01%
  cpu          : usr=3.02%, sys=7.08%, ctx=1475390, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1571645/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=6139.3MB, aggrb=104771KB/s, minb=104771KB/s, maxb=104771KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  vda: ios=1464677/18, merge=0/9, ticks=324452/40, in_queue=324132, util=97.85%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=16261: Sun Dec 15 19:24:02 2019
  write: io=18895MB, bw=322434KB/s, iops=80608, runt= 60009msec
    clat (usec): min=2, max=242956, avg=66.55, stdev=1165.23
     lat (usec): min=2, max=242958, avg=72.17, stdev=1229.79
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    4], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    6], 95.00th=[    6],
     | 99.00th=[   16], 99.50th=[  159], 99.90th=[18304], 99.95th=[23424],
     | 99.99th=[36096]
    bw (KB  /s): min=    0, max=63440, per=12.78%, avg=41198.18, stdev=6458.96
    lat (usec) : 4=8.56%, 10=89.24%, 20=1.36%, 50=0.29%, 100=0.03%
    lat (usec) : 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.04%, 10=0.15%, 20=0.19%, 50=0.07%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=4.18%, sys=5.82%, ctx=54734, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=4837234/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=18895MB, aggrb=322433KB/s, minb=322433KB/s, maxb=322433KB/s, mint=60009msec, maxt=60009msec

Disk stats (read/write):
  vda: ios=689/1218295, merge=0/4077, ticks=448/5951228, in_queue=5952404, util=90.57%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=16234: Sun Dec 15 19:23:02 2019
  read : io=6596.6MB, bw=112569KB/s, iops=28142, runt= 60002msec
    clat (usec): min=121, max=184890, avg=276.48, stdev=539.90
     lat (usec): min=123, max=184891, avg=277.66, stdev=539.94
    clat percentiles (usec):
     |  1.00th=[  169],  5.00th=[  185], 10.00th=[  197], 20.00th=[  213],
     | 30.00th=[  225], 40.00th=[  237], 50.00th=[  251], 60.00th=[  266],
     | 70.00th=[  278], 80.00th=[  306], 90.00th=[  366], 95.00th=[  442],
     | 99.00th=[  660], 99.50th=[  868], 99.90th=[ 1832], 99.95th=[ 2224],
     | 99.99th=[ 3632]
    bw (KB  /s): min= 9216, max=15784, per=12.51%, avg=14079.06, stdev=854.30
    lat (usec) : 250=49.89%, 500=47.05%, 750=2.39%, 1000=0.27%
    lat (msec) : 2=0.32%, 4=0.07%, 10=0.01%, 20=0.01%, 250=0.01%
  cpu          : usr=3.13%, sys=6.76%, ctx=1693666, majf=0, minf=68
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1688590/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=6596.6MB, aggrb=112568KB/s, minb=112568KB/s, maxb=112568KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  vda: ios=1684306/17, merge=0/12, ticks=354396/0, in_queue=354000, util=99.71%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=16285: Sun Dec 15 19:25:03 2019
  write: io=254516KB, bw=4240.1KB/s, iops=1060, runt= 60014msec
    clat (usec): min=600, max=238513, avg=7533.21, stdev=25823.98
     lat (usec): min=601, max=238514, avg=7534.78, stdev=25824.02
    clat percentiles (usec):
     |  1.00th=[  644],  5.00th=[  668], 10.00th=[  684], 20.00th=[  708],
     | 30.00th=[  732], 40.00th=[  764], 50.00th=[  844], 60.00th=[  940],
     | 70.00th=[ 1004], 80.00th=[ 1096], 90.00th=[ 1608], 95.00th=[98816],
     | 99.00th=[114176], 99.50th=[120320], 99.90th=[150528], 99.95th=[179200],
     | 99.99th=[211968]
    bw (KB  /s): min=  161, max=  944, per=12.61%, avg=534.54, stdev=151.72
    lat (usec) : 750=36.66%, 1000=32.68%
    lat (msec) : 2=22.13%, 4=2.00%, 10=0.26%, 20=0.03%, 50=0.01%
    lat (msec) : 100=1.48%, 250=4.75%
  cpu          : usr=0.22%, sys=0.89%, ctx=131469, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=63629/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=254516KB, aggrb=4240KB/s, minb=4240KB/s, maxb=4240KB/s, mint=60014msec, maxt=60014msec

Disk stats (read/write):
  vda: ios=688/63671, merge=0/1910, ticks=548/55720, in_queue=56248, util=92.49%


fio (Нидерланды, Амстердам)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=5714: Wed Dec 11 14:41:53 2019
  read : io=1601.9MB, bw=27335KB/s, iops=6833, runt= 60009msec
    clat (usec): min=147, max=100526, avg=1153.46, stdev=1994.16
     lat (usec): min=148, max=100527, avg=1155.92, stdev=1997.60
    clat percentiles (usec):
     |  1.00th=[  209],  5.00th=[  245], 10.00th=[  270], 20.00th=[  338],
     | 30.00th=[  446], 40.00th=[  540], 50.00th=[  628], 60.00th=[  748],
     | 70.00th=[  940], 80.00th=[ 1272], 90.00th=[ 2192], 95.00th=[ 3728],
     | 99.00th=[10176], 99.50th=[13888], 99.90th=[23680], 99.95th=[27776],
     | 99.99th=[35584]
    bw (KB  /s): min= 1928, max= 7392, per=12.49%, avg=3415.22, stdev=835.60
    lat (usec) : 250=5.96%, 500=28.91%, 750=25.29%, 1000=12.39%
    lat (msec) : 2=16.24%, 4=6.70%, 10=3.48%, 20=0.85%, 50=0.18%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.45%, sys=4.77%, ctx=415144, majf=0, minf=90
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=410080/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=1601.9MB, aggrb=27334KB/s, minb=27334KB/s, maxb=27334KB/s, mint=60009msec, maxt=60009msec

Disk stats (read/write):
  vda: ios=409241/8, merge=0/6, ticks=311748/96, in_queue=311500, util=86.72%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=5746: Wed Dec 11 14:43:59 2019
  write: io=10688MB, bw=182382KB/s, iops=45595, runt= 60009msec
    clat (usec): min=2, max=384094, avg=108.20, stdev=2173.94
     lat (usec): min=3, max=384096, avg=118.33, stdev=2308.57
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    6], 95.00th=[    8],
     | 99.00th=[   34], 99.50th=[ 2608], 99.90th=[28032], 99.95th=[37632],
     | 99.99th=[76288]
    bw (KB  /s): min=    0, max=45740, per=13.54%, avg=24689.24, stdev=8568.82
    lat (usec) : 4=1.12%, 10=95.58%, 20=1.91%, 50=0.51%, 100=0.12%
    lat (usec) : 250=0.13%, 500=0.06%, 750=0.02%, 1000=0.01%
    lat (msec) : 2=0.03%, 4=0.04%, 10=0.13%, 20=0.17%, 50=0.15%
    lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%
  cpu          : usr=3.78%, sys=5.14%, ctx=39912, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=2736147/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=10688MB, aggrb=182382KB/s, minb=182382KB/s, maxb=182382KB/s, mint=60009msec, maxt=60009msec

Disk stats (read/write):
  vda: ios=170/452541, merge=0/4470, ticks=592/4649176, in_queue=4650320, util=90.11%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=5733: Wed Dec 11 14:42:59 2019
  read : io=1587.8MB, bw=27097KB/s, iops=6774, runt= 60003msec
    clat (usec): min=146, max=129531, avg=1163.40, stdev=1924.84
     lat (usec): min=147, max=129532, avg=1165.57, stdev=1926.44
    clat percentiles (usec):
     |  1.00th=[  215],  5.00th=[  258], 10.00th=[  298], 20.00th=[  402],
     | 30.00th=[  502], 40.00th=[  572], 50.00th=[  652], 60.00th=[  772],
     | 70.00th=[  956], 80.00th=[ 1288], 90.00th=[ 2192], 95.00th=[ 3600],
     | 99.00th=[ 9664], 99.50th=[12992], 99.90th=[22400], 99.95th=[26496],
     | 99.99th=[38656]
    bw (KB  /s): min= 1720, max= 5008, per=12.52%, avg=3391.80, stdev=509.87
    lat (usec) : 250=4.06%, 500=25.57%, 750=28.59%, 1000=13.68%
    lat (msec) : 2=16.84%, 4=6.99%, 10=3.33%, 20=0.78%, 50=0.15%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.64%, sys=4.62%, ctx=408046, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=406469/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=1587.8MB, aggrb=27096KB/s, minb=27096KB/s, maxb=27096KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  vda: ios=405647/8, merge=0/7, ticks=300840/196, in_queue=300772, util=84.98%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=5759: Wed Dec 11 14:45:00 2019
  write: io=188084KB, bw=3133.5KB/s, iops=783, runt= 60024msec
    clat (usec): min=639, max=249688, avg=10195.17, stdev=29490.37
     lat (usec): min=640, max=249690, avg=10197.12, stdev=29490.47
    clat percentiles (usec):
     |  1.00th=[  692],  5.00th=[  724], 10.00th=[  748], 20.00th=[  804],
     | 30.00th=[  892], 40.00th=[ 1004], 50.00th=[ 1096], 60.00th=[ 1192],
     | 70.00th=[ 1368], 80.00th=[ 1688], 90.00th=[ 4320], 95.00th=[102912],
     | 99.00th=[119296], 99.50th=[126464], 99.90th=[146432], 99.95th=[150528],
     | 99.99th=[246784]
    bw (KB  /s): min=  135, max=  728, per=12.57%, avg=393.66, stdev=111.76
    lat (usec) : 750=10.98%, 1000=28.18%
    lat (msec) : 2=44.97%, 4=5.64%, 10=1.47%, 20=0.11%, 50=0.13%
    lat (msec) : 100=2.51%, 250=6.02%
  cpu          : usr=0.28%, sys=0.93%, ctx=98320, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=47021/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=188084KB, aggrb=3133KB/s, minb=3133KB/s, maxb=3133KB/s, mint=60024msec, maxt=60024msec

Disk stats (read/write):
  vda: ios=660/47082, merge=0/1723, ticks=944/54796, in_queue=55720, util=90.97%


fio (Сингапур)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=26021: Wed Dec 11 18:38:21 2019
  read : io=4308.6MB, bw=73532KB/s, iops=18382, runt= 60001msec
    clat (usec): min=2, max=75156, avg=428.64, stdev=674.69
     lat (usec): min=2, max=75156, avg=429.66, stdev=674.74
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[  153], 10.00th=[  187], 20.00th=[  215],
     | 30.00th=[  237], 40.00th=[  262], 50.00th=[  298], 60.00th=[  350],
     | 70.00th=[  414], 80.00th=[  502], 90.00th=[  652], 95.00th=[  900],
     | 99.00th=[ 3056], 99.50th=[ 4320], 99.90th=[ 7648], 99.95th=[ 9792],
     | 99.99th=[14784]
    bw (KB  /s): min= 3133, max=13504, per=12.49%, avg=9181.87, stdev=1826.22
    lat (usec) : 4=1.88%, 10=2.65%, 20=0.22%, 50=0.03%, 100=0.01%
    lat (usec) : 250=30.28%, 500=44.58%, 750=13.23%, 1000=2.86%
    lat (msec) : 2=2.33%, 4=1.35%, 10=0.54%, 20=0.04%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.72%, sys=4.35%, ctx=1055602, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1102992/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4308.6MB, aggrb=73531KB/s, minb=73531KB/s, maxb=73531KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1047795/8, merge=0/6, ticks=390052/52, in_queue=389732, util=97.89%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=26051: Wed Dec 11 18:40:26 2019
  write: io=22990MB, bw=392328KB/s, iops=98081, runt= 60006msec
    clat (usec): min=1, max=369457, avg=55.42, stdev=1456.41
     lat (usec): min=2, max=369458, avg=59.90, stdev=1504.16
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    5], 95.00th=[    5],
     | 99.00th=[    9], 99.50th=[   22], 99.90th=[16768], 99.95th=[22656],
     | 99.99th=[37120]
    bw (KB  /s): min=    0, max=70700, per=13.09%, avg=51347.13, stdev=13217.67
    lat (usec) : 2=0.01%, 4=39.98%, 10=59.15%, 20=0.32%, 50=0.18%
    lat (usec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.02%, 10=0.09%, 20=0.13%, 50=0.07%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=3.92%, sys=5.85%, ctx=47216, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5885501/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=22990MB, aggrb=392327KB/s, minb=392327KB/s, maxb=392327KB/s, mint=60006msec, maxt=60006msec

Disk stats (read/write):
  vda: ios=253/695603, merge=0/4848, ticks=460/6040768, in_queue=6041620, util=94.92%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=26034: Wed Dec 11 18:39:26 2019
  read : io=4060.6MB, bw=69297KB/s, iops=17324, runt= 60002msec
    clat (usec): min=126, max=73471, avg=454.91, stdev=816.66
     lat (usec): min=127, max=73472, avg=455.93, stdev=816.68
    clat percentiles (usec):
     |  1.00th=[  161],  5.00th=[  179], 10.00th=[  191], 20.00th=[  211],
     | 30.00th=[  229], 40.00th=[  249], 50.00th=[  274], 60.00th=[  314],
     | 70.00th=[  374], 80.00th=[  466], 90.00th=[  636], 95.00th=[  996],
     | 99.00th=[ 4320], 99.50th=[ 5664], 99.90th=[ 8384], 99.95th=[ 9920],
     | 99.99th=[16320]
    bw (KB  /s): min= 3722, max=14064, per=12.47%, avg=8642.03, stdev=2176.97
    lat (usec) : 250=40.65%, 500=42.13%, 750=10.05%, 1000=2.20%
    lat (msec) : 2=2.13%, 4=1.68%, 10=1.11%, 20=0.04%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.66%, sys=3.86%, ctx=1045614, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1039489/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4060.6MB, aggrb=69296KB/s, minb=69296KB/s, maxb=69296KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  vda: ios=1035654/8, merge=0/7, ticks=408452/1488, in_queue=409680, util=98.90%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=26068: Wed Dec 11 18:41:27 2019
  write: io=131456KB, bw=2190.6KB/s, iops=547, runt= 60012msec
    clat (usec): min=613, max=1435.9K, avg=14593.78, stdev=50405.98
     lat (usec): min=614, max=1435.9K, avg=14595.36, stdev=50406.03
    clat percentiles (usec):
     |  1.00th=[  684],  5.00th=[  732], 10.00th=[  812], 20.00th=[  964],
     | 30.00th=[ 1064], 40.00th=[ 1192], 50.00th=[ 1336], 60.00th=[ 1560],
     | 70.00th=[ 1880], 80.00th=[ 2448], 90.00th=[ 6112], 95.00th=[132096],
     | 99.00th=[214016], 99.50th=[268288], 99.90th=[411648], 99.95th=[468992],
     | 99.99th=[1400832]
    bw (KB  /s): min=    2, max=  672, per=12.89%, avg=282.21, stdev=143.92
    lat (usec) : 750=6.13%, 1000=16.97%
    lat (msec) : 2=49.80%, 4=14.23%, 10=4.16%, 20=0.69%, 50=0.21%
    lat (msec) : 100=0.05%, 250=7.07%, 500=0.67%, 750=0.01%, 2000=0.02%
  cpu          : usr=0.13%, sys=0.58%, ctx=68429, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=32864/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=131456KB, aggrb=2190KB/s, minb=2190KB/s, maxb=2190KB/s, mint=60012msec, maxt=60012msec

Disk stats (read/write):
  vda: ios=261/32874, merge=0/1477, ticks=160/56864, in_queue=57012, util=94.50%


fio (Великобритания, Лондон)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=31964: Thu Dec 12 12:59:43 2019
  read : io=4469.1MB, bw=76286KB/s, iops=19071, runt= 60001msec
    clat (usec): min=1, max=57633, avg=412.24, stdev=553.46
     lat (usec): min=2, max=57635, avg=413.42, stdev=553.53
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    8], 10.00th=[  187], 20.00th=[  221],
     | 30.00th=[  249], 40.00th=[  278], 50.00th=[  318], 60.00th=[  362],
     | 70.00th=[  422], 80.00th=[  506], 90.00th=[  652], 95.00th=[  852],
     | 99.00th=[ 2480], 99.50th=[ 3152], 99.90th=[ 5472], 99.95th=[ 7712],
     | 99.99th=[17280]
    bw (KB  /s): min= 4344, max=15384, per=12.49%, avg=9524.23, stdev=1621.11
    lat (usec) : 2=0.01%, 4=2.21%, 10=2.89%, 20=0.24%, 50=0.08%
    lat (usec) : 100=0.01%, 250=25.20%, 500=48.91%, 750=13.74%, 1000=3.08%
    lat (msec) : 2=2.17%, 4=1.24%, 10=0.19%, 20=0.03%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=1.86%, sys=4.84%, ctx=1086469, majf=1, minf=88
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1144306/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4469.1MB, aggrb=76285KB/s, minb=76285KB/s, maxb=76285KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1078551/8, merge=0/6, ticks=391976/24, in_queue=391736, util=98.85%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=32007: Thu Dec 12 13:01:50 2019
  write: io=17383MB, bw=296475KB/s, iops=74118, runt= 60038msec
    clat (usec): min=1, max=634520, avg=80.43, stdev=2797.83
     lat (usec): min=2, max=634522, avg=84.20, stdev=2824.23
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    5], 95.00th=[    5],
     | 99.00th=[    9], 99.50th=[   18], 99.90th=[20096], 99.95th=[35072],
     | 99.99th=[87552]
    bw (KB  /s): min=    0, max=87481, per=13.99%, avg=41472.38, stdev=18398.46
    lat (usec) : 2=0.01%, 4=55.86%, 10=43.23%, 20=0.45%, 50=0.13%
    lat (usec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.02%, 10=0.07%, 20=0.10%, 50=0.08%
    lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=2.82%, sys=4.31%, ctx=36944, majf=0, minf=76
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=4449946/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=17383MB, aggrb=296475KB/s, minb=296475KB/s, maxb=296475KB/s, mint=60038msec, maxt=60038msec

Disk stats (read/write):
  vda: ios=262/323640, merge=0/6202, ticks=144/5185996, in_queue=5186620, util=96.41%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=31989: Thu Dec 12 13:00:50 2019
  read : io=4223.6MB, bw=72081KB/s, iops=18020, runt= 60001msec
    clat (usec): min=109, max=106784, avg=436.41, stdev=534.14
     lat (usec): min=110, max=106785, avg=437.62, stdev=534.20
    clat percentiles (usec):
     |  1.00th=[  165],  5.00th=[  189], 10.00th=[  207], 20.00th=[  235],
     | 30.00th=[  262], 40.00th=[  294], 50.00th=[  334], 60.00th=[  382],
     | 70.00th=[  442], 80.00th=[  524], 90.00th=[  676], 95.00th=[  884],
     | 99.00th=[ 2416], 99.50th=[ 2992], 99.90th=[ 4512], 99.95th=[ 5856],
     | 99.99th=[14528]
    bw (KB  /s): min= 5672, max=11072, per=12.50%, avg=9011.87, stdev=833.30
    lat (usec) : 250=25.62%, 500=51.60%, 750=15.32%, 1000=3.50%
    lat (msec) : 2=2.49%, 4=1.31%, 10=0.13%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.82%, sys=4.50%, ctx=1084188, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1081230/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4223.6MB, aggrb=72080KB/s, minb=72080KB/s, maxb=72080KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1078880/8, merge=0/7, ticks=402188/24, in_queue=401836, util=99.65%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=32033: Thu Dec 12 13:02:51 2019
  write: io=91328KB, bw=1521.7KB/s, iops=380, runt= 60019msec
    clat (usec): min=629, max=1370.1K, avg=21010.33, stdev=67213.73
     lat (usec): min=631, max=1370.1K, avg=21012.43, stdev=67213.79
    clat percentiles (usec):
     |  1.00th=[  868],  5.00th=[ 1032], 10.00th=[ 1144], 20.00th=[ 1304],
     | 30.00th=[ 1464], 40.00th=[ 1656], 50.00th=[ 1880], 60.00th=[ 2192],
     | 70.00th=[ 2672], 80.00th=[ 3888], 90.00th=[11072], 95.00th=[187392],
     | 99.00th=[272384], 99.50th=[366592], 99.90th=[569344], 99.95th=[643072],
     | 99.99th=[1318912]
    bw (KB  /s): min=   22, max=  360, per=12.77%, avg=194.30, stdev=56.49
    lat (usec) : 750=0.12%, 1000=3.99%
    lat (msec) : 2=50.22%, 4=26.11%, 10=9.22%, 20=1.34%, 50=0.39%
    lat (msec) : 100=0.14%, 250=6.92%, 500=1.36%, 750=0.18%, 1000=0.01%
    lat (msec) : 2000=0.03%
  cpu          : usr=0.10%, sys=0.45%, ctx=47653, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=22832/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=91328KB, aggrb=1521KB/s, minb=1521KB/s, maxb=1521KB/s, mint=60019msec, maxt=60019msec

Disk stats (read/write):
  vda: ios=699/22885, merge=0/1284, ticks=488/58240, in_queue=58728, util=96.24%


fio (США, Нью-Йорк)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=8043: Wed Dec  4 13:40:06 2019
  read : io=8650.3MB, bw=147629KB/s, iops=36907, runt= 60001msec
    clat (usec): min=1, max=12457, avg=212.16, stdev=168.73
     lat (usec): min=2, max=12458, avg=212.93, stdev=168.79
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[  131], 20.00th=[  171],
     | 30.00th=[  181], 40.00th=[  191], 50.00th=[  201], 60.00th=[  211],
     | 70.00th=[  227], 80.00th=[  253], 90.00th=[  302], 95.00th=[  358],
     | 99.00th=[  564], 99.50th=[  836], 99.90th=[ 2544], 99.95th=[ 2928],
     | 99.99th=[ 4640]
    bw (KB  /s): min=12872, max=25280, per=12.50%, avg=18457.17, stdev=2181.08
    lat (usec) : 2=0.01%, 4=7.82%, 10=1.21%, 20=0.37%, 50=0.02%
    lat (usec) : 100=0.04%, 250=69.71%, 500=19.41%, 750=0.83%, 1000=0.15%
    lat (msec) : 2=0.20%, 4=0.20%, 10=0.01%, 20=0.01%
  cpu          : usr=2.25%, sys=5.84%, ctx=2010544, majf=2, minf=96
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2214468/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=8650.3MB, aggrb=147628KB/s, minb=147628KB/s, maxb=147628KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=2002054/8, merge=0/6, ticks=374052/0, in_queue=373720, util=97.53%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=8088: Wed Dec  4 13:42:12 2019
  write: io=29272MB, bw=499455KB/s, iops=124863, runt= 60014msec
    clat (usec): min=1, max=365561, avg=44.38, stdev=1109.33
     lat (usec): min=2, max=365726, avg=47.80, stdev=1149.56
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    3], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   10], 99.50th=[   44], 99.90th=[14400], 99.95th=[19072],
     | 99.99th=[32128]
    bw (KB  /s): min=    0, max=80828, per=12.91%, avg=64495.24, stdev=14149.46
    lat (usec) : 2=0.01%, 4=80.00%, 10=18.95%, 20=0.40%, 50=0.16%
    lat (usec) : 100=0.12%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.03%, 10=0.11%, 20=0.13%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=3.87%, sys=6.39%, ctx=59719, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=7493570/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=29272MB, aggrb=499454KB/s, minb=499454KB/s, maxb=499454KB/s, mint=60014msec, maxt=60014msec

Disk stats (read/write):
  vda: ios=654/652218, merge=0/6105, ticks=1156/6405124, in_queue=6406188, util=97.12%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=8064: Wed Dec  4 13:41:11 2019
  read : io=8391.1MB, bw=143220KB/s, iops=35804, runt= 60001msec
    clat (usec): min=70, max=14982, avg=218.74, stdev=90.67
     lat (usec): min=71, max=14983, avg=219.51, stdev=90.78
    clat percentiles (usec):
     |  1.00th=[  141],  5.00th=[  157], 10.00th=[  165], 20.00th=[  175],
     | 30.00th=[  183], 40.00th=[  191], 50.00th=[  201], 60.00th=[  211],
     | 70.00th=[  225], 80.00th=[  247], 90.00th=[  290], 95.00th=[  338],
     | 99.00th=[  474], 99.50th=[  564], 99.90th=[ 1064], 99.95th=[ 1352],
     | 99.99th=[ 3056]
    bw (KB  /s): min=15832, max=19784, per=12.50%, avg=17904.29, stdev=655.80
    lat (usec) : 100=0.06%, 250=80.95%, 500=18.20%, 750=0.59%, 1000=0.09%
    lat (msec) : 2=0.09%, 4=0.02%, 10=0.01%, 20=0.01%
  cpu          : usr=2.32%, sys=5.26%, ctx=2152129, majf=0, minf=80
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2148329/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=8391.1MB, aggrb=143219KB/s, minb=143219KB/s, maxb=143219KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=2145398/17, merge=0/11, ticks=392640/0, in_queue=392288, util=99.31%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=8107: Wed Dec  4 13:43:12 2019
  write: io=340360KB, bw=5672.5KB/s, iops=1418, runt= 60007msec
    clat (usec): min=269, max=229762, avg=5632.93, stdev=22773.91
     lat (usec): min=270, max=229763, avg=5634.26, stdev=22773.96
    clat percentiles (usec):
     |  1.00th=[  314],  5.00th=[  354], 10.00th=[  402], 20.00th=[  466],
     | 30.00th=[  502], 40.00th=[  540], 50.00th=[  580], 60.00th=[  660],
     | 70.00th=[  820], 80.00th=[  988], 90.00th=[ 1240], 95.00th=[ 2448],
     | 99.00th=[115200], 99.50th=[121344], 99.90th=[140288], 99.95th=[162816],
     | 99.99th=[220160]
    bw (KB  /s): min=  155, max= 1314, per=12.53%, avg=710.70, stdev=210.32
    lat (usec) : 500=29.48%, 750=35.93%, 1000=14.99%
    lat (msec) : 2=14.39%, 4=0.50%, 10=0.11%, 20=0.02%, 50=0.01%
    lat (msec) : 100=1.05%, 250=3.52%
  cpu          : usr=0.23%, sys=0.85%, ctx=174273, majf=0, minf=65
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=85090/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=340360KB, aggrb=5672KB/s, minb=5672KB/s, maxb=5672KB/s, mint=60007msec, maxt=60007msec

Disk stats (read/write):
  vda: ios=598/85062, merge=0/2038, ticks=200/55736, in_queue=55928, util=92.53%


fio (США, Сан-Франциско)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29456: Thu Dec  5 08:01:48 2019
  read : io=1303.9MB, bw=22251KB/s, iops=5562, runt= 60003msec
    clat (usec): min=142, max=80183, avg=1420.21, stdev=2913.39
     lat (usec): min=142, max=80184, avg=1422.58, stdev=2915.85
    clat percentiles (usec):
     |  1.00th=[  213],  5.00th=[  243], 10.00th=[  270], 20.00th=[  338],
     | 30.00th=[  458], 40.00th=[  572], 50.00th=[  676], 60.00th=[  836],
     | 70.00th=[ 1048], 80.00th=[ 1496], 90.00th=[ 2736], 95.00th=[ 4832],
     | 99.00th=[13888], 99.50th=[19584], 99.90th=[36608], 99.95th=[43776],
     | 99.99th=[62720]
    bw (KB  /s): min= 1008, max= 7440, per=12.52%, avg=2785.28, stdev=1196.35
    lat (usec) : 250=6.47%, 500=26.73%, 750=22.17%, 1000=12.54%
    lat (msec) : 2=17.74%, 4=8.01%, 10=4.56%, 20=1.31%, 50=0.46%
    lat (msec) : 100=0.03%
  cpu          : usr=1.34%, sys=3.88%, ctx=338517, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=333779/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=1303.9MB, aggrb=22250KB/s, minb=22250KB/s, maxb=22250KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  vda: ios=333758/8, merge=0/6, ticks=326896/48, in_queue=326612, util=86.22%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29490: Thu Dec  5 08:03:54 2019
  write: io=12984MB, bw=221303KB/s, iops=55325, runt= 60079msec
    clat (usec): min=2, max=363723, avg=94.71, stdev=2166.04
     lat (usec): min=2, max=363724, avg=102.66, stdev=2264.24
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    4], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    6], 95.00th=[    7],
     | 99.00th=[   19], 99.50th=[  274], 99.90th=[24192], 99.95th=[32640],
     | 99.99th=[82432]
    bw (KB  /s): min=    0, max=52047, per=13.24%, avg=29293.56, stdev=11137.47
    lat (usec) : 4=2.76%, 10=94.87%, 20=1.39%, 50=0.32%, 100=0.06%
    lat (usec) : 250=0.09%, 500=0.03%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.03%, 10=0.11%, 20=0.16%, 50=0.12%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=3.89%, sys=5.30%, ctx=36713, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=3323911/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=12984MB, aggrb=221302KB/s, minb=221302KB/s, maxb=221302KB/s, mint=60079msec, maxt=60079msec

Disk stats (read/write):
  vda: ios=696/369874, merge=0/4081, ticks=1004/5449776, in_queue=5451104, util=93.26%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29473: Thu Dec  5 08:02:53 2019
  read : io=1645.1MB, bw=28089KB/s, iops=7022, runt= 60002msec
    clat (usec): min=116, max=80577, avg=1125.46, stdev=2401.59
     lat (usec): min=117, max=80578, avg=1127.23, stdev=2402.88
    clat percentiles (usec):
     |  1.00th=[  201],  5.00th=[  229], 10.00th=[  249], 20.00th=[  290],
     | 30.00th=[  362], 40.00th=[  466], 50.00th=[  564], 60.00th=[  668],
     | 70.00th=[  852], 80.00th=[ 1144], 90.00th=[ 1992], 95.00th=[ 3600],
     | 99.00th=[11456], 99.50th=[16768], 99.90th=[30080], 99.95th=[38144],
     | 99.99th=[50944]
    bw (KB  /s): min= 1464, max= 9208, per=12.50%, avg=3510.17, stdev=1352.72
    lat (usec) : 250=10.20%, 500=33.01%, 750=21.97%, 1000=10.86%
    lat (msec) : 2=14.03%, 4=5.53%, 10=3.17%, 20=0.91%, 50=0.32%
    lat (msec) : 100=0.01%
  cpu          : usr=1.37%, sys=3.08%, ctx=428535, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=421347/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=1645.1MB, aggrb=28088KB/s, minb=28088KB/s, maxb=28088KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  vda: ios=420438/8, merge=0/7, ticks=371708/188, in_queue=371760, util=92.14%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29514: Thu Dec  5 08:04:55 2019
  write: io=112088KB, bw=1867.3KB/s, iops=466, runt= 60030msec
    clat (usec): min=644, max=311332, avg=17114.08, stdev=44927.08
     lat (usec): min=645, max=311334, avg=17116.41, stdev=44927.87
    clat percentiles (usec):
     |  1.00th=[  716],  5.00th=[  772], 10.00th=[  828], 20.00th=[  980],
     | 30.00th=[ 1112], 40.00th=[ 1272], 50.00th=[ 1512], 60.00th=[ 1816],
     | 70.00th=[ 2288], 80.00th=[ 3632], 90.00th=[98816], 95.00th=[140288],
     | 99.00th=[189440], 99.50th=[207872], 99.90th=[254976], 99.95th=[280576],
     | 99.99th=[301056]
    bw (KB  /s): min=   53, max=  551, per=12.55%, avg=234.27, stdev=83.53
    lat (usec) : 750=3.25%, 1000=18.30%
    lat (msec) : 2=43.22%, 4=16.58%, 10=6.32%, 20=1.32%, 50=0.39%
    lat (msec) : 100=0.67%, 250=9.84%, 500=0.12%
  cpu          : usr=0.19%, sys=0.70%, ctx=59173, majf=0, minf=58
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=28022/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=112088KB, aggrb=1867KB/s, minb=1867KB/s, maxb=1867KB/s, mint=60030msec, maxt=60030msec

Disk stats (read/write):
  vda: ios=631/28086, merge=0/1468, ticks=780/56516, in_queue=57260, util=93.38%


Speedtest (Канада, Торонто)
Retrieving speedtest.net configuration...
Testing from Digital Ocean (159.203.34.131)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by FibreStream (Toronto, ON) [3.45 km]: 2.749 ms
Testing download speed................................................................................
Download: 336.52 Mbit/s
Testing upload speed................................................................................................
Upload: 601.00 Mbit/s


Speedtest (Германия, Франкфурт)
Retrieving speedtest.net configuration...
Testing from Digital Ocean (104.248.45.25)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by fdcservers.net (Frankfurt) [0.89 km]: 1.731 ms
Testing download speed................................................................................
Download: 2537.59 Mbit/s
Testing upload speed................................................................................................
Upload: 2018.71 Mbit/s


Speedtest (Индия, Бангалор)
Retrieving speedtest.net configuration...
Testing from Digital Ocean (139.59.17.112)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Bharti Airtel Ltd (Bangalore) [1.65 km]: 2.194 ms
Testing download speed................................................................................
Download: 672.15 Mbit/s
Testing upload speed................................................................................................
Upload: 865.85 Mbit/s


Speedtest (Нидерланды, Амстердам)
Retrieving speedtest.net configuration...
Testing from DigitalOcean (188.166.46.233)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Vancis (Amsterdam) [3.12 km]: 2.61 ms
Testing download speed................................................................................
Download: 290.28 Mbit/s
Testing upload speed................................................................................................
Upload: 874.21 Mbit/s


Speedtest (Сингапур)
Retrieving speedtest.net configuration...
Testing from Digital Ocean (68.183.190.84)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Viewqwest Pte Ltd (Singapore) [13.00 km]: 2.036 ms
Testing download speed................................................................................
Download: 3591.73 Mbit/s
Testing upload speed................................................................................................
Upload: 1560.91 Mbit/s


Speedtest (Великобритания, Лондон)
Retrieving speedtest.net configuration...
Testing from Digital Ocean (142.93.37.95)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Iomart (Maidenhead) [3.98 km]: 2.04 ms
Testing download speed................................................................................
Download: 858.28 Mbit/s
Testing upload speed................................................................................................
Upload: 876.08 Mbit/s


Speedtest (США, Нью-Йорк)
Retrieving speedtest.net configuration...
Testing from Digital Ocean (167.71.82.38)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Interserver, inc (Secaucus, NJ) [8.75 km]: 1.879 ms
Testing download speed................................................................................
Download: 3598.06 Mbit/s
Testing upload speed................................................................................................
Upload: 2021.05 Mbit/s


Speedtest (США, Сан-Франциско)
Retrieving speedtest.net configuration...
Testing from Digital Ocean (138.68.27.151)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Next Level Networks (Santa Clara, CA) [2.25 km]: 3.875 ms
Testing download speed................................................................................
Download: 574.97 Mbit/s
Testing upload speed................................................................................................
Upload: 261.58 Mbit/s



Виртуальный сервер от Vultr:




  • тариф Cloud Compute (25 GB SSD / 1x vCPU / 1 GB RAM / 10000 Mbit/s за $5 в месяц);
  • дата-центры в Австралии, Великобритании, Германии, Канаде, Нидерландах, Сингапуре, США, Франции, Японии.

Тесты:


Unixbench (Австралия, Сидней)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 523cbcdd6ca4 (4788.9 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   02:46:36 up 2 min,  1 user,  load average: 0.58, 0.19, 0.06; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 02:46:36 - 03:14:41
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       25102626.8 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4295.6 MWIPS (9.8 s, 7 samples)
Execl Throughput                               4230.2 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        604499.7 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          169295.0 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1277011.0 KBps  (30.0 s, 2 samples)
Pipe Throughput                              959022.0 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 140482.6 lps   (10.0 s, 7 samples)
Process Creation                               9784.3 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7107.8 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    906.8 lpm   (60.0 s, 2 samples)
System Call Overhead                         673720.5 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   25102626.8   2151.0
Double-Precision Whetstone                       55.0       4295.6    781.0
Execl Throughput                                 43.0       4230.2    983.8
File Copy 1024 bufsize 2000 maxblocks          3960.0     604499.7   1526.5
File Copy 256 bufsize 500 maxblocks            1655.0     169295.0   1022.9
File Copy 4096 bufsize 8000 maxblocks          5800.0    1277011.0   2201.7
Pipe Throughput                               12440.0     959022.0    770.9
Pipe-based Context Switching                   4000.0     140482.6    351.2
Process Creation                                126.0       9784.3    776.5
Shell Scripts (1 concurrent)                     42.4       7107.8   1676.4
Shell Scripts (8 concurrent)                      6.0        906.8   1511.4
System Call Overhead                          15000.0     673720.5    449.1
                                                                   ========
System Benchmarks Index Score                                        1025.9


Unixbench (Канада, Торонто)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 82d9ed4018dd (5187.8 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   02:42:53 up 2 min,  1 user,  load average: 0.64, 0.17, 0.06; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 02:42:53 - 03:11:00
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       38699273.9 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4619.5 MWIPS (10.2 s, 7 samples)
Execl Throughput                               6768.4 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        705978.4 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          192749.4 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1751883.6 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1012385.6 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 179610.0 lps   (10.0 s, 7 samples)
Process Creation                              15081.7 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                  10739.0 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   1378.7 lpm   (60.0 s, 2 samples)
System Call Overhead                         570458.3 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   38699273.9   3316.1
Double-Precision Whetstone                       55.0       4619.5    839.9
Execl Throughput                                 43.0       6768.4   1574.0
File Copy 1024 bufsize 2000 maxblocks          3960.0     705978.4   1782.8
File Copy 256 bufsize 500 maxblocks            1655.0     192749.4   1164.6
File Copy 4096 bufsize 8000 maxblocks          5800.0    1751883.6   3020.5
Pipe Throughput                               12440.0    1012385.6    813.8
Pipe-based Context Switching                   4000.0     179610.0    449.0
Process Creation                                126.0      15081.7   1197.0
Shell Scripts (1 concurrent)                     42.4      10739.0   2532.8
Shell Scripts (8 concurrent)                      6.0       1378.7   2297.8
System Call Overhead                          15000.0     570458.3    380.3
                                                                   ========
System Benchmarks Index Score                                        1314.5


Unixbench (Германия, Франкфурт)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 714389bda930 (4800.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   23:41:12 up 3 min,  1 user,  load average: 0.49, 0.17, 0.06; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Fri Dec 13 2019 23:41:12 - 00:09:21
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       26077546.4 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3561.5 MWIPS (9.9 s, 7 samples)
Execl Throughput                               4233.5 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        495359.7 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          130299.0 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1347940.9 KBps  (30.0 s, 2 samples)
Pipe Throughput                              657753.9 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 117057.6 lps   (10.0 s, 7 samples)
Process Creation                               9390.0 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   6618.6 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    870.7 lpm   (60.1 s, 2 samples)
System Call Overhead                         359105.0 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   26077546.4   2234.6
Double-Precision Whetstone                       55.0       3561.5    647.5
Execl Throughput                                 43.0       4233.5    984.5
File Copy 1024 bufsize 2000 maxblocks          3960.0     495359.7   1250.9
File Copy 256 bufsize 500 maxblocks            1655.0     130299.0    787.3
File Copy 4096 bufsize 8000 maxblocks          5800.0    1347940.9   2324.0
Pipe Throughput                               12440.0     657753.9    528.7
Pipe-based Context Switching                   4000.0     117057.6    292.6
Process Creation                                126.0       9390.0    745.2
Shell Scripts (1 concurrent)                     42.4       6618.6   1561.0
Shell Scripts (8 concurrent)                      6.0        870.7   1451.2
System Call Overhead                          15000.0     359105.0    239.4
                                                                   ========
System Benchmarks Index Score                                         875.9


Unixbench (Франция, Париж)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 523cbcdd6ca4 (4788.9 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   12:37:38 up 5 min,  1 user,  load average: 0.85, 0.48, 0.21; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Thu Dec 12 2019 12:37:38 - 13:05:33
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables        8896034.1 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     1762.9 MWIPS (9.5 s, 7 samples)
Execl Throughput                               1488.3 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        170322.0 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks           46997.0 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks        499490.5 KBps  (30.0 s, 2 samples)
Pipe Throughput                              253632.7 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  47089.9 lps   (10.0 s, 7 samples)
Process Creation                               3571.1 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   2557.8 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    339.8 lpm   (60.1 s, 2 samples)
System Call Overhead                         149530.8 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0    8896034.1    762.3
Double-Precision Whetstone                       55.0       1762.9    320.5
Execl Throughput                                 43.0       1488.3    346.1
File Copy 1024 bufsize 2000 maxblocks          3960.0     170322.0    430.1
File Copy 256 bufsize 500 maxblocks            1655.0      46997.0    284.0
File Copy 4096 bufsize 8000 maxblocks          5800.0     499490.5    861.2
Pipe Throughput                               12440.0     253632.7    203.9
Pipe-based Context Switching                   4000.0      47089.9    117.7
Process Creation                                126.0       3571.1    283.4
Shell Scripts (1 concurrent)                     42.4       2557.8    603.3
Shell Scripts (8 concurrent)                      6.0        339.8    566.3
System Call Overhead                          15000.0     149530.8     99.7
                                                                   ========
System Benchmarks Index Score                                         335.9


Unixbench (Япония, Токио)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 523cbcdd6ca4 (4788.9 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   07:37:39 up 3 min,  1 user,  load average: 0.65, 0.31, 0.12; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Thu Dec 05 2019 07:37:39 - 08:05:44
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       26212655.7 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4429.9 MWIPS (10.0 s, 7 samples)
Execl Throughput                               4627.0 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        676411.8 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          181937.6 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1590192.7 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1008509.9 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 147408.3 lps   (10.0 s, 7 samples)
Process Creation                              10743.2 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7856.9 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   1025.9 lpm   (60.0 s, 2 samples)
System Call Overhead                         689942.2 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   26212655.7   2246.2
Double-Precision Whetstone                       55.0       4429.9    805.4
Execl Throughput                                 43.0       4627.0   1076.0
File Copy 1024 bufsize 2000 maxblocks          3960.0     676411.8   1708.1
File Copy 256 bufsize 500 maxblocks            1655.0     181937.6   1099.3
File Copy 4096 bufsize 8000 maxblocks          5800.0    1590192.7   2741.7
Pipe Throughput                               12440.0    1008509.9    810.7
Pipe-based Context Switching                   4000.0     147408.3    368.5
Process Creation                                126.0      10743.2    852.6
Shell Scripts (1 concurrent)                     42.4       7856.9   1853.0
Shell Scripts (8 concurrent)                      6.0       1025.9   1709.8
System Call Overhead                          15000.0     689942.2    460.0
                                                                   ========
System Benchmarks Index Score                                        1115.7


Unixbench (Нидерланды, Амстердам)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 523cbcdd6ca4 (4788.9 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   18:14:32 up 4 min,  1 user,  load average: 0.54, 0.21, 0.07; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Wed Dec 11 2019 18:14:32 - 18:42:36
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       24400966.1 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4307.7 MWIPS (10.0 s, 7 samples)
Execl Throughput                               4137.2 lps   (29.7 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        563366.3 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          166725.8 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1351284.6 KBps  (30.0 s, 2 samples)
Pipe Throughput                              947831.0 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 127169.9 lps   (10.0 s, 7 samples)
Process Creation                               9682.9 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   6961.7 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    917.2 lpm   (60.0 s, 2 samples)
System Call Overhead                         660482.8 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   24400966.1   2090.9
Double-Precision Whetstone                       55.0       4307.7    783.2
Execl Throughput                                 43.0       4137.2    962.1
File Copy 1024 bufsize 2000 maxblocks          3960.0     563366.3   1422.6
File Copy 256 bufsize 500 maxblocks            1655.0     166725.8   1007.4
File Copy 4096 bufsize 8000 maxblocks          5800.0    1351284.6   2329.8
Pipe Throughput                               12440.0     947831.0    761.9
Pipe-based Context Switching                   4000.0     127169.9    317.9
Process Creation                                126.0       9682.9    768.5
Shell Scripts (1 concurrent)                     42.4       6961.7   1641.9
Shell Scripts (8 concurrent)                      6.0        917.2   1528.7
System Call Overhead                          15000.0     660482.8    440.3
                                                                   ========
System Benchmarks Index Score                                        1006.6


Unixbench (Сингапур)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 523cbcdd6ca4 (4788.9 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   14:17:01 up 4 min,  1 user,  load average: 0.68, 0.23, 0.08; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Wed Dec 11 2019 14:17:01 - 14:45:06
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       25144861.2 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4260.9 MWIPS (9.9 s, 7 samples)
Execl Throughput                               4040.1 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        427185.8 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          116507.8 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1148414.3 KBps  (30.0 s, 2 samples)
Pipe Throughput                              626572.7 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 115253.6 lps   (10.0 s, 7 samples)
Process Creation                               9620.9 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   6846.7 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    881.5 lpm   (60.0 s, 2 samples)
System Call Overhead                         344955.6 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   25144861.2   2154.7
Double-Precision Whetstone                       55.0       4260.9    774.7
Execl Throughput                                 43.0       4040.1    939.5
File Copy 1024 bufsize 2000 maxblocks          3960.0     427185.8   1078.8
File Copy 256 bufsize 500 maxblocks            1655.0     116507.8    704.0
File Copy 4096 bufsize 8000 maxblocks          5800.0    1148414.3   1980.0
Pipe Throughput                               12440.0     626572.7    503.7
Pipe-based Context Switching                   4000.0     115253.6    288.1
Process Creation                                126.0       9620.9    763.6
Shell Scripts (1 concurrent)                     42.4       6846.7   1614.8
Shell Scripts (8 concurrent)                      6.0        881.5   1469.2
System Call Overhead                          15000.0     344955.6    230.0
                                                                   ========
System Benchmarks Index Score                                         850.2


Unixbench (Великобритания, Лондон)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel Core Processor (Haswell, no TSX, IBRS) (4788.9 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   15:54:53 up 2 min,  1 user,  load average: 0.62, 0.21, 0.07; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Sun Dec 15 2019 15:54:53 - 16:22:57
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       26199695.8 lps   (10.1 s, 7 samples)
Double-Precision Whetstone                     4515.1 MWIPS (9.9 s, 7 samples)
Execl Throughput                               4408.5 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        595220.1 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          166098.6 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1395372.9 KBps  (30.0 s, 2 samples)
Pipe Throughput                              993618.1 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 141759.0 lps   (10.0 s, 7 samples)
Process Creation                              10582.5 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7349.8 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    939.8 lpm   (60.1 s, 2 samples)
System Call Overhead                         689617.2 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   26199695.8   2245.0
Double-Precision Whetstone                       55.0       4515.1    820.9
Execl Throughput                                 43.0       4408.5   1025.2
File Copy 1024 bufsize 2000 maxblocks          3960.0     595220.1   1503.1
File Copy 256 bufsize 500 maxblocks            1655.0     166098.6   1003.6
File Copy 4096 bufsize 8000 maxblocks          5800.0    1395372.9   2405.8
Pipe Throughput                               12440.0     993618.1    798.7
Pipe-based Context Switching                   4000.0     141759.0    354.4
Process Creation                                126.0      10582.5    839.9
Shell Scripts (1 concurrent)                     42.4       7349.8   1733.4
Shell Scripts (8 concurrent)                      6.0        939.8   1566.4
System Call Overhead                          15000.0     689617.2    459.7
                                                                   ========
System Benchmarks Index Score                                        1061.0


Unixbench (США, Атланта)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 523cbcdd6ca4 (4788.9 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   18:59:49 up 2 min,  1 user,  load average: 0.65, 0.22, 0.08; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Sun Dec 15 2019 18:59:49 - 19:27:54
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       25425673.3 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4296.1 MWIPS (9.9 s, 7 samples)
Execl Throughput                               4230.5 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        463467.3 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          126388.6 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1186982.0 KBps  (30.0 s, 2 samples)
Pipe Throughput                              646774.2 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 122358.8 lps   (10.0 s, 7 samples)
Process Creation                               9800.5 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7261.6 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    938.8 lpm   (60.0 s, 2 samples)
System Call Overhead                         357385.0 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   25425673.3   2178.7
Double-Precision Whetstone                       55.0       4296.1    781.1
Execl Throughput                                 43.0       4230.5    983.8
File Copy 1024 bufsize 2000 maxblocks          3960.0     463467.3   1170.4
File Copy 256 bufsize 500 maxblocks            1655.0     126388.6    763.7
File Copy 4096 bufsize 8000 maxblocks          5800.0    1186982.0   2046.5
Pipe Throughput                               12440.0     646774.2    519.9
Pipe-based Context Switching                   4000.0     122358.8    305.9
Process Creation                                126.0       9800.5    777.8
Shell Scripts (1 concurrent)                     42.4       7261.6   1712.6
Shell Scripts (8 concurrent)                      6.0        938.8   1564.6
System Call Overhead                          15000.0     357385.0    238.3
                                                                   ========
System Benchmarks Index Score                                         888.4


Unixbench (США, Чикаго)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 523cbcdd6ca4 (4788.9 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   01:04:10 up 2 min,  1 user,  load average: 0.60, 0.20, 0.07; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 01:04:10 - 01:32:14
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       25523985.5 lps   (10.1 s, 7 samples)
Double-Precision Whetstone                     4315.5 MWIPS (9.9 s, 7 samples)
Execl Throughput                               4148.4 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        452161.7 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          123919.4 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1196969.9 KBps  (30.0 s, 2 samples)
Pipe Throughput                              642700.8 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 124108.9 lps   (10.0 s, 7 samples)
Process Creation                               9849.9 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7299.6 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    941.0 lpm   (60.0 s, 2 samples)
System Call Overhead                         355273.2 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   25523985.5   2187.1
Double-Precision Whetstone                       55.0       4315.5    784.6
Execl Throughput                                 43.0       4148.4    964.8
File Copy 1024 bufsize 2000 maxblocks          3960.0     452161.7   1141.8
File Copy 256 bufsize 500 maxblocks            1655.0     123919.4    748.8
File Copy 4096 bufsize 8000 maxblocks          5800.0    1196969.9   2063.7
Pipe Throughput                               12440.0     642700.8    516.6
Pipe-based Context Switching                   4000.0     124108.9    310.3
Process Creation                                126.0       9849.9    781.7
Shell Scripts (1 concurrent)                     42.4       7299.6   1721.6
Shell Scripts (8 concurrent)                      6.0        941.0   1568.4
System Call Overhead                          15000.0     355273.2    236.8
                                                                   ========
System Benchmarks Index Score                                         886.0


Unixbench (США, Даллас)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 523cbcdd6ca4 (4800.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   01:07:22 up 2 min,  1 user,  load average: 0.62, 0.19, 0.07; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 01:07:22 - 01:35:27
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       26030426.9 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4443.2 MWIPS (10.0 s, 7 samples)
Execl Throughput                               4537.8 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        634719.5 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          176552.9 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1486392.3 KBps  (30.0 s, 2 samples)
Pipe Throughput                              990514.7 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 144687.4 lps   (10.0 s, 7 samples)
Process Creation                              10880.8 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7575.3 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    992.9 lpm   (60.0 s, 2 samples)
System Call Overhead                         688433.6 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   26030426.9   2230.5
Double-Precision Whetstone                       55.0       4443.2    807.9
Execl Throughput                                 43.0       4537.8   1055.3
File Copy 1024 bufsize 2000 maxblocks          3960.0     634719.5   1602.8
File Copy 256 bufsize 500 maxblocks            1655.0     176552.9   1066.8
File Copy 4096 bufsize 8000 maxblocks          5800.0    1486392.3   2562.7
Pipe Throughput                               12440.0     990514.7    796.2
Pipe-based Context Switching                   4000.0     144687.4    361.7
Process Creation                                126.0      10880.8    863.6
Shell Scripts (1 concurrent)                     42.4       7575.3   1786.6
Shell Scripts (8 concurrent)                      6.0        992.9   1654.8
System Call Overhead                          15000.0     688433.6    459.0
                                                                   ========
System Benchmarks Index Score                                        1090.0


Unixbench (США, Лос-Анджелес)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 523cbcdd6ca4 (4788.9 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   01:11:21 up 3 min,  1 user,  load average: 0.71, 0.29, 0.11; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 01:11:21 - 01:39:26
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       25267089.5 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4246.4 MWIPS (9.9 s, 7 samples)
Execl Throughput                               4112.8 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        456196.7 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          122812.5 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1192809.7 KBps  (30.0 s, 2 samples)
Pipe Throughput                              636446.5 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 116546.4 lps   (10.0 s, 7 samples)
Process Creation                               9696.7 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7097.2 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    919.9 lpm   (60.0 s, 2 samples)
System Call Overhead                         352701.1 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   25267089.5   2165.1
Double-Precision Whetstone                       55.0       4246.4    772.1
Execl Throughput                                 43.0       4112.8    956.5
File Copy 1024 bufsize 2000 maxblocks          3960.0     456196.7   1152.0
File Copy 256 bufsize 500 maxblocks            1655.0     122812.5    742.1
File Copy 4096 bufsize 8000 maxblocks          5800.0    1192809.7   2056.6
Pipe Throughput                               12440.0     636446.5    511.6
Pipe-based Context Switching                   4000.0     116546.4    291.4
Process Creation                                126.0       9696.7    769.6
Shell Scripts (1 concurrent)                     42.4       7097.2   1673.9
Shell Scripts (8 concurrent)                      6.0        919.9   1533.2
System Call Overhead                          15000.0     352701.1    235.1
                                                                   ========
System Benchmarks Index Score                                         872.5


Unixbench (США, Майами)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 523cbcdd6ca4 (4788.9 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   01:15:03 up 2 min,  1 user,  load average: 0.60, 0.21, 0.07; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 01:15:03 - 01:43:08
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       26582764.1 lps   (10.1 s, 7 samples)
Double-Precision Whetstone                     4412.8 MWIPS (10.0 s, 7 samples)
Execl Throughput                               4634.7 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        669481.5 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          179719.1 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1533680.6 KBps  (30.0 s, 2 samples)
Pipe Throughput                              997113.7 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 145060.8 lps   (10.0 s, 7 samples)
Process Creation                              11030.6 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7915.2 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   1025.9 lpm   (60.0 s, 2 samples)
System Call Overhead                         686188.0 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   26582764.1   2277.9
Double-Precision Whetstone                       55.0       4412.8    802.3
Execl Throughput                                 43.0       4634.7   1077.8
File Copy 1024 bufsize 2000 maxblocks          3960.0     669481.5   1690.6
File Copy 256 bufsize 500 maxblocks            1655.0     179719.1   1085.9
File Copy 4096 bufsize 8000 maxblocks          5800.0    1533680.6   2644.3
Pipe Throughput                               12440.0     997113.7    801.5
Pipe-based Context Switching                   4000.0     145060.8    362.7
Process Creation                                126.0      11030.6    875.4
Shell Scripts (1 concurrent)                     42.4       7915.2   1866.8
Shell Scripts (8 concurrent)                      6.0       1025.9   1709.8
System Call Overhead                          15000.0     686188.0    457.5
                                                                   ========
System Benchmarks Index Score                                        1111.4


Unixbench (США, Нью-Йорк)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 714389bda930 (4800.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   00:55:26 up 2 min,  1 user,  load average: 0.56, 0.23, 0.09; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 00:55:26 - 01:23:34
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       25633639.9 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3533.8 MWIPS (9.9 s, 7 samples)
Execl Throughput                               4070.4 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        474437.0 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          126743.5 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1199905.9 KBps  (30.0 s, 2 samples)
Pipe Throughput                              647341.0 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  88978.2 lps   (10.0 s, 7 samples)
Process Creation                               7295.0 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   6352.0 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    826.7 lpm   (60.0 s, 2 samples)
System Call Overhead                         353376.1 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   25633639.9   2196.5
Double-Precision Whetstone                       55.0       3533.8    642.5
Execl Throughput                                 43.0       4070.4    946.6
File Copy 1024 bufsize 2000 maxblocks          3960.0     474437.0   1198.1
File Copy 256 bufsize 500 maxblocks            1655.0     126743.5    765.8
File Copy 4096 bufsize 8000 maxblocks          5800.0    1199905.9   2068.8
Pipe Throughput                               12440.0     647341.0    520.4
Pipe-based Context Switching                   4000.0      88978.2    222.4
Process Creation                                126.0       7295.0    579.0
Shell Scripts (1 concurrent)                     42.4       6352.0   1498.1
Shell Scripts (8 concurrent)                      6.0        826.7   1377.9
System Call Overhead                          15000.0     353376.1    235.6
                                                                   ========
System Benchmarks Index Score                                         812.4


Unixbench (США, Сиэтл)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 714389bda930 (4800.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   01:18:39 up 2 min,  1 user,  load average: 0.73, 0.25, 0.09; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 01:18:39 - 01:46:47
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       26700878.9 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3622.7 MWIPS (9.9 s, 7 samples)
Execl Throughput                               3907.5 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        486945.0 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          130046.2 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1283902.3 KBps  (30.0 s, 2 samples)
Pipe Throughput                              662209.3 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 113981.0 lps   (10.0 s, 7 samples)
Process Creation                               8765.1 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7075.8 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    905.1 lpm   (60.1 s, 2 samples)
System Call Overhead                         364946.5 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   26700878.9   2288.0
Double-Precision Whetstone                       55.0       3622.7    658.7
Execl Throughput                                 43.0       3907.5    908.7
File Copy 1024 bufsize 2000 maxblocks          3960.0     486945.0   1229.7
File Copy 256 bufsize 500 maxblocks            1655.0     130046.2    785.8
File Copy 4096 bufsize 8000 maxblocks          5800.0    1283902.3   2213.6
Pipe Throughput                               12440.0     662209.3    532.3
Pipe-based Context Switching                   4000.0     113981.0    285.0
Process Creation                                126.0       8765.1    695.6
Shell Scripts (1 concurrent)                     42.4       7075.8   1668.8
Shell Scripts (8 concurrent)                      6.0        905.1   1508.5
System Call Overhead                          15000.0     364946.5    243.3
                                                                   ========
System Benchmarks Index Score                                         870.5


Unixbench (США, Сан-Хосе)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: vultr.guest: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Virtual CPU 523cbcdd6ca4 (4788.9 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   02:40:36 up 3 min,  1 user,  load average: 0.62, 0.21, 0.07; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 02:40:36 - 03:08:40
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       26677842.8 lps   (10.1 s, 7 samples)
Double-Precision Whetstone                     4517.5 MWIPS (10.0 s, 7 samples)
Execl Throughput                               4679.7 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        658109.9 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          179078.5 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1513335.5 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1011954.1 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 147993.7 lps   (10.0 s, 7 samples)
Process Creation                              11034.6 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7761.3 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   1007.2 lpm   (60.0 s, 2 samples)
System Call Overhead                         686496.1 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   26677842.8   2286.0
Double-Precision Whetstone                       55.0       4517.5    821.4
Execl Throughput                                 43.0       4679.7   1088.3
File Copy 1024 bufsize 2000 maxblocks          3960.0     658109.9   1661.9
File Copy 256 bufsize 500 maxblocks            1655.0     179078.5   1082.0
File Copy 4096 bufsize 8000 maxblocks          5800.0    1513335.5   2609.2
Pipe Throughput                               12440.0    1011954.1    813.5
Pipe-based Context Switching                   4000.0     147993.7    370.0
Process Creation                                126.0      11034.6    875.8
Shell Scripts (1 concurrent)                     42.4       7761.3   1830.5
Shell Scripts (8 concurrent)                      6.0       1007.2   1678.6
System Call Overhead                          15000.0     686496.1    457.7
                                                                   ========
System Benchmarks Index Score                                        1111.5


dd (Австралия, Сидней)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.00964 s, 357 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 12.2036 s, 176 MB/s


dd (Канада, Торонто)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.20408 s, 511 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.97048 s, 432 MB/s


dd (Германия, Франкфурт)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.49292 s, 478 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.66099 s, 461 MB/s


dd (Франция, Париж)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.42477 s, 334 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.17672 s, 348 MB/s


dd (Япония, Токио)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 5.98321 s, 359 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 5.74079 s, 374 MB/s


dd (Нидерланды, Амстердам)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.00055 s, 358 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 7.24659 s, 296 MB/s


dd (Сингапур)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.21242 s, 346 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.38259 s, 336 MB/s


dd (Великобритания, Лондон)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 7.77476 s, 276 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 7.65941 s, 280 MB/s


dd (США, Атланта)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.17663 s, 348 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.14213 s, 350 MB/s


dd (США, Чикаго)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 5.75075 s, 373 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 5.78569 s, 371 MB/s


dd (США, Даллас)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 5.7069 s, 376 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 5.60498 s, 383 MB/s


dd (США, Лос-Анджелес)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 12.9305 s, 166 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.29062 s, 341 MB/s


dd (США, Майами)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 5.89201 s, 364 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 5.74615 s, 374 MB/s


dd (США, Нью-Йорк)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.52842 s, 474 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.94189 s, 435 MB/s


dd (США, Сиэтл)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.54416 s, 473 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 5.68025 s, 378 MB/s


dd (США, Сан-Хосе)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.92419 s, 310 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 8.11701 s, 265 MB/s


fio (Австралия, Сидней)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28589: Mon Dec 16 03:16:07 2019
  read : io=7674.1MB, bw=130983KB/s, iops=32745, runt= 60001msec
    clat (usec): min=1, max=13364, avg=240.17, stdev=183.78
     lat (usec): min=2, max=13365, avg=240.76, stdev=183.79
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[  171], 20.00th=[  193],
     | 30.00th=[  207], 40.00th=[  221], 50.00th=[  233], 60.00th=[  247],
     | 70.00th=[  266], 80.00th=[  290], 90.00th=[  334], 95.00th=[  374],
     | 99.00th=[  470], 99.50th=[  540], 99.90th=[ 2864], 99.95th=[ 3408],
     | 99.99th=[ 7008]
    bw (KB  /s): min=10904, max=23080, per=12.50%, avg=16375.51, stdev=1940.14
    lat (usec) : 2=0.02%, 4=5.24%, 10=1.36%, 20=0.94%, 50=0.29%
    lat (usec) : 100=0.02%, 250=53.43%, 500=37.99%, 750=0.42%, 1000=0.04%
    lat (msec) : 2=0.05%, 4=0.18%, 10=0.03%, 20=0.01%
  cpu          : usr=1.87%, sys=5.83%, ctx=1816331, majf=2, minf=94
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1964785/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7674.1MB, aggrb=130983KB/s, minb=130983KB/s, maxb=130983KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1809446/9, merge=0/4, ticks=391072/0, in_queue=391128, util=98.07%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28618: Mon Dec 16 03:18:14 2019
  write: io=31535MB, bw=537932KB/s, iops=134482, runt= 60029msec
    clat (usec): min=1, max=512047, avg=44.70, stdev=1034.85
     lat (usec): min=1, max=512048, avg=47.33, stdev=1074.92
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    4], 90.00th=[    4], 95.00th=[    6],
     | 99.00th=[   18], 99.50th=[   28], 99.90th=[15296], 99.95th=[19840],
     | 99.99th=[30848]
    bw (KB  /s): min=    1, max=85640, per=12.68%, avg=68203.30, stdev=11017.32
    lat (usec) : 2=0.12%, 4=77.58%, 10=19.53%, 20=2.07%, 50=0.27%
    lat (usec) : 100=0.04%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.03%, 10=0.10%, 20=0.13%, 50=0.05%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=2.64%, sys=7.81%, ctx=76430, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=8072876/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=31535MB, aggrb=537931KB/s, minb=537931KB/s, maxb=537931KB/s, mint=60029msec, maxt=60029msec

Disk stats (read/write):
  vda: ios=169/782742, merge=0/8447, ticks=1264/6814944, in_queue=6818788, util=98.98%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28602: Mon Dec 16 03:17:14 2019
  read : io=7471.5MB, bw=127506KB/s, iops=31876, runt= 60003msec
    clat (usec): min=132, max=10029, avg=246.78, stdev=107.92
     lat (usec): min=133, max=10030, avg=247.38, stdev=107.95
    clat percentiles (usec):
     |  1.00th=[  159],  5.00th=[  173], 10.00th=[  183], 20.00th=[  197],
     | 30.00th=[  209], 40.00th=[  219], 50.00th=[  231], 60.00th=[  245],
     | 70.00th=[  262], 80.00th=[  286], 90.00th=[  326], 95.00th=[  362],
     | 99.00th=[  446], 99.50th=[  494], 99.90th=[  884], 99.95th=[ 1752],
     | 99.99th=[ 4960]
    bw (KB  /s): min=13352, max=17536, per=12.50%, avg=15943.78, stdev=782.58
    lat (usec) : 250=63.11%, 500=36.43%, 750=0.33%, 1000=0.04%
    lat (msec) : 2=0.04%, 4=0.02%, 10=0.03%, 20=0.01%
  cpu          : usr=1.84%, sys=4.97%, ctx=1916371, majf=0, minf=80
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1912681/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7471.5MB, aggrb=127505KB/s, minb=127505KB/s, maxb=127505KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  vda: ios=1911024/10, merge=0/14, ticks=387936/0, in_queue=387700, util=99.79%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28631: Mon Dec 16 03:19:15 2019
  write: io=2116.3MB, bw=36114KB/s, iops=9028, runt= 60005msec
    clat (usec): min=60, max=52275, avg=881.45, stdev=5188.98
     lat (usec): min=60, max=52275, avg=882.13, stdev=5189.03
    clat percentiles (usec):
     |  1.00th=[   67],  5.00th=[   71], 10.00th=[   74], 20.00th=[   80],
     | 30.00th=[   85], 40.00th=[   90], 50.00th=[   95], 60.00th=[  101],
     | 70.00th=[  109], 80.00th=[  121], 90.00th=[  161], 95.00th=[  215],
     | 99.00th=[36096], 99.50th=[38144], 99.90th=[41216], 99.95th=[42240],
     | 99.99th=[44800]
    bw (KB  /s): min= 3169, max= 5891, per=12.51%, avg=4516.12, stdev=452.68
    lat (usec) : 100=57.43%, 250=39.03%, 500=1.24%, 750=0.04%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.01%, 50=2.21%
    lat (msec) : 100=0.01%
  cpu          : usr=0.61%, sys=2.76%, ctx=1095749, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=541749/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2116.3MB, aggrb=36113KB/s, minb=36113KB/s, maxb=36113KB/s, mint=60005msec, maxt=60005msec

Disk stats (read/write):
  vda: ios=182/541139, merge=0/1556, ticks=100/47652, in_queue=47640, util=79.23%


fio (Канада, Торонто)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=4579: Mon Dec 16 03:12:15 2019
  read : io=12498MB, bw=213300KB/s, iops=53324, runt= 60001msec
    clat (usec): min=1, max=12335, avg=146.68, stdev=110.17
     lat (usec): min=1, max=12335, avg=147.24, stdev=110.17
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    4], 20.00th=[  135],
     | 30.00th=[  141], 40.00th=[  147], 50.00th=[  151], 60.00th=[  155],
     | 70.00th=[  161], 80.00th=[  169], 90.00th=[  183], 95.00th=[  201],
     | 99.00th=[  318], 99.50th=[  422], 99.90th=[ 1832], 99.95th=[ 2024],
     | 99.99th=[ 2672]
    bw (KB  /s): min=17680, max=35552, per=12.47%, avg=26605.05, stdev=3053.39
    lat (usec) : 2=0.92%, 4=8.94%, 10=0.77%, 20=0.02%, 50=0.01%
    lat (usec) : 100=0.13%, 250=87.61%, 500=1.28%, 750=0.05%, 1000=0.02%
    lat (msec) : 2=0.21%, 4=0.05%, 10=0.01%, 20=0.01%
  cpu          : usr=2.33%, sys=5.24%, ctx=2863832, majf=2, minf=87
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=3199546/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=12498MB, aggrb=213299KB/s, minb=213299KB/s, maxb=213299KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=2853531/9, merge=0/4, ticks=413816/0, in_queue=413848, util=97.93%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=4606: Mon Dec 16 03:14:20 2019
  write: io=40568MB, bw=692123KB/s, iops=173030, runt= 60020msec
    clat (usec): min=1, max=168030, avg=30.86, stdev=724.13
     lat (usec): min=1, max=168030, avg=33.62, stdev=769.11
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    2], 50.00th=[    2], 60.00th=[    2],
     | 70.00th=[    3], 80.00th=[    3], 90.00th=[    3], 95.00th=[    3],
     | 99.00th=[    4], 99.50th=[    8], 99.90th=[11712], 99.95th=[16192],
     | 99.99th=[27776]
    bw (KB  /s): min=    1, max=102488, per=12.60%, avg=87188.15, stdev=9025.40
    lat (usec) : 2=0.15%, 4=98.40%, 10=1.00%, 20=0.07%, 50=0.06%
    lat (usec) : 100=0.02%, 250=0.01%, 500=0.02%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.03%, 10=0.09%, 20=0.09%, 50=0.03%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=4.36%, sys=6.42%, ctx=83149, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=10385305/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=40568MB, aggrb=692122KB/s, minb=692122KB/s, maxb=692122KB/s, mint=60020msec, maxt=60020msec

Disk stats (read/write):
  vda: ios=625/1328807, merge=0/8697, ticks=3072/7005192, in_queue=7008748, util=99.27%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=4593: Mon Dec 16 03:13:20 2019
  read : io=11311MB, bw=193037KB/s, iops=48259, runt= 60001msec
    clat (usec): min=75, max=8391, avg=162.37, stdev=140.17
     lat (usec): min=76, max=8391, avg=162.94, stdev=140.17
    clat percentiles (usec):
     |  1.00th=[  123],  5.00th=[  131], 10.00th=[  135], 20.00th=[  141],
     | 30.00th=[  145], 40.00th=[  149], 50.00th=[  151], 60.00th=[  157],
     | 70.00th=[  161], 80.00th=[  169], 90.00th=[  183], 95.00th=[  199],
     | 99.00th=[  290], 99.50th=[  406], 99.90th=[ 2192], 99.95th=[ 3120],
     | 99.99th=[ 6432]
    bw (KB  /s): min=17560, max=26000, per=12.50%, avg=24134.45, stdev=1483.61
    lat (usec) : 100=0.12%, 250=98.48%, 500=1.10%, 750=0.05%, 1000=0.02%
    lat (msec) : 2=0.11%, 4=0.08%, 10=0.04%
  cpu          : usr=2.16%, sys=4.65%, ctx=2903321, majf=0, minf=73
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2895610/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=11311MB, aggrb=193037KB/s, minb=193037KB/s, maxb=193037KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=2892534/8, merge=0/5, ticks=434584/0, in_queue=434828, util=99.87%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=4621: Mon Dec 16 03:15:21 2019
  write: io=2264.6MB, bw=38648KB/s, iops=9661, runt= 60002msec
    clat (usec): min=51, max=256774, avg=824.62, stdev=6716.44
     lat (usec): min=51, max=256775, avg=825.21, stdev=6716.44
    clat percentiles (usec):
     |  1.00th=[   56],  5.00th=[   57], 10.00th=[   58], 20.00th=[   60],
     | 30.00th=[   61], 40.00th=[   63], 50.00th=[   65], 60.00th=[   67],
     | 70.00th=[   71], 80.00th=[   89], 90.00th=[  139], 95.00th=[  203],
     | 99.00th=[43264], 99.50th=[55552], 99.90th=[81408], 99.95th=[96768],
     | 99.99th=[134144]
    bw (KB  /s): min= 1323, max= 8856, per=12.51%, avg=4835.55, stdev=1290.36
    lat (usec) : 100=81.77%, 250=14.40%, 500=1.52%, 750=0.17%, 1000=0.17%
    lat (msec) : 2=0.37%, 4=0.21%, 10=0.09%, 20=0.01%, 50=0.58%
    lat (msec) : 100=0.66%, 250=0.04%, 500=0.01%
  cpu          : usr=0.45%, sys=1.61%, ctx=1167262, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=579733/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2264.6MB, aggrb=38647KB/s, minb=38647KB/s, maxb=38647KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  vda: ios=173/578889, merge=0/1547, ticks=252/52532, in_queue=52724, util=87.45%


fio (Германия, Франкфурт)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=21959: Sat Dec 14 00:10:37 2019
  read : io=7915.7MB, bw=135091KB/s, iops=33772, runt= 60001msec
    clat (usec): min=1, max=15947, avg=231.15, stdev=202.63
     lat (usec): min=2, max=15947, avg=232.04, stdev=202.68
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    9], 10.00th=[  153], 20.00th=[  175],
     | 30.00th=[  189], 40.00th=[  201], 50.00th=[  215], 60.00th=[  229],
     | 70.00th=[  249], 80.00th=[  278], 90.00th=[  330], 95.00th=[  386],
     | 99.00th=[  588], 99.50th=[  940], 99.90th=[ 2704], 99.95th=[ 3152],
     | 99.99th=[ 8256]
    bw (KB  /s): min=13072, max=22224, per=12.50%, avg=16885.51, stdev=1416.82
    lat (usec) : 2=0.01%, 4=2.12%, 10=3.30%, 20=1.99%, 50=0.58%
    lat (usec) : 100=0.02%, 250=62.14%, 500=28.13%, 750=1.10%, 1000=0.15%
    lat (msec) : 2=0.21%, 4=0.23%, 10=0.02%, 20=0.01%
  cpu          : usr=2.90%, sys=7.72%, ctx=1870975, majf=2, minf=94
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2026398/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7915.7MB, aggrb=135090KB/s, minb=135090KB/s, maxb=135090KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1861991/9, merge=0/4, ticks=329328/0, in_queue=328932, util=98.13%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=21985: Sat Dec 14 00:12:43 2019
  write: io=26172MB, bw=446593KB/s, iops=111648, runt= 60009msec
    clat (usec): min=1, max=104202, avg=46.04, stdev=817.78
     lat (usec): min=2, max=104203, avg=50.84, stdev=880.51
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    4], 90.00th=[    5], 95.00th=[    7],
     | 99.00th=[   18], 99.50th=[   95], 99.90th=[14272], 99.95th=[18816],
     | 99.99th=[28032]
    bw (KB  /s): min=    1, max=66344, per=12.64%, avg=56439.90, stdev=6246.70
    lat (usec) : 2=0.01%, 4=74.88%, 10=21.89%, 20=2.34%, 50=0.31%
    lat (usec) : 100=0.08%, 250=0.03%, 500=0.02%, 750=0.02%, 1000=0.01%
    lat (msec) : 2=0.03%, 4=0.05%, 10=0.15%, 20=0.14%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=4.84%, sys=5.78%, ctx=92604, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=6699905/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=26172MB, aggrb=446593KB/s, minb=446593KB/s, maxb=446593KB/s, mint=60009msec, maxt=60009msec

Disk stats (read/write):
  vda: ios=169/1517807, merge=0/10554, ticks=652/6860812, in_queue=6861820, util=98.12%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=21972: Sat Dec 14 00:11:43 2019
  read : io=8536.2MB, bw=145672KB/s, iops=36417, runt= 60004msec
    clat (usec): min=118, max=11287, avg=214.02, stdev=81.60
     lat (usec): min=118, max=11288, avg=214.87, stdev=81.67
    clat percentiles (usec):
     |  1.00th=[  143],  5.00th=[  155], 10.00th=[  163], 20.00th=[  173],
     | 30.00th=[  183], 40.00th=[  191], 50.00th=[  201], 60.00th=[  213],
     | 70.00th=[  227], 80.00th=[  247], 90.00th=[  278], 95.00th=[  310],
     | 99.00th=[  398], 99.50th=[  442], 99.90th=[  628], 99.95th=[  804],
     | 99.99th=[ 3664]
    bw (KB  /s): min=14304, max=19464, per=12.50%, avg=18209.07, stdev=907.94
    lat (usec) : 250=81.57%, 500=18.17%, 750=0.20%, 1000=0.03%
    lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=3.16%, sys=6.72%, ctx=2190379, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2185219/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=8536.2MB, aggrb=145671KB/s, minb=145671KB/s, maxb=145671KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  vda: ios=2182468/8, merge=0/5, ticks=399108/0, in_queue=398872, util=99.82%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=21998: Sat Dec 14 00:13:43 2019
  write: io=2255.2MB, bw=38485KB/s, iops=9621, runt= 60005msec
    clat (usec): min=65, max=42951, avg=825.62, stdev=4606.39
     lat (usec): min=65, max=42951, avg=826.56, stdev=4606.42
    clat percentiles (usec):
     |  1.00th=[   71],  5.00th=[   74], 10.00th=[   75], 20.00th=[   79],
     | 30.00th=[   83], 40.00th=[   88], 50.00th=[   93], 60.00th=[   98],
     | 70.00th=[  105], 80.00th=[  114], 90.00th=[  133], 95.00th=[  161],
     | 99.00th=[30336], 99.50th=[31616], 99.90th=[34048], 99.95th=[35072],
     | 99.99th=[38144]
    bw (KB  /s): min= 3420, max= 6336, per=12.51%, avg=4815.62, stdev=460.55
    lat (usec) : 100=62.03%, 250=35.23%, 500=0.26%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=2.45%
  cpu          : usr=1.02%, sys=3.03%, ctx=1169418, majf=0, minf=67
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=577318/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2255.2MB, aggrb=38484KB/s, minb=38484KB/s, maxb=38484KB/s, mint=60005msec, maxt=60005msec

Disk stats (read/write):
  vda: ios=177/576692, merge=0/1614, ticks=108/44140, in_queue=44096, util=73.32%


fio (Франция, Париж)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=21844: Thu Dec 12 13:06:55 2019
  read : io=3057.1MB, bw=52185KB/s, iops=13046, runt= 60004msec
    clat (usec): min=4, max=52368, avg=596.01, stdev=581.93
     lat (usec): min=5, max=52370, avg=598.59, stdev=582.58
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[  251], 10.00th=[  306], 20.00th=[  354],
     | 30.00th=[  398], 40.00th=[  446], 50.00th=[  510], 60.00th=[  572],
     | 70.00th=[  652], 80.00th=[  756], 90.00th=[  940], 95.00th=[ 1176],
     | 99.00th=[ 1992], 99.50th=[ 2928], 99.90th=[ 8032], 99.95th=[ 9536],
     | 99.99th=[18048]
    bw (KB  /s): min= 2432, max=10000, per=12.51%, avg=6526.19, stdev=974.23
    lat (usec) : 10=3.48%, 20=0.46%, 50=0.34%, 100=0.03%, 250=0.64%
    lat (usec) : 500=43.85%, 750=30.97%, 1000=12.02%
    lat (msec) : 2=7.22%, 4=0.67%, 10=0.28%, 20=0.03%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=3.03%, sys=6.50%, ctx=754863, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=782820/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=3057.1MB, aggrb=52184KB/s, minb=52184KB/s, maxb=52184KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  vda: ios=748009/9, merge=0/4, ticks=297144/0, in_queue=296676, util=97.38%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=21880: Thu Dec 12 13:09:04 2019
  write: io=9695.7MB, bw=165439KB/s, iops=41359, runt= 60012msec
    clat (usec): min=4, max=272163, avg=120.45, stdev=1565.02
     lat (usec): min=5, max=272194, avg=133.49, stdev=1688.89
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    6],
     | 30.00th=[    7], 40.00th=[    8], 50.00th=[    8], 60.00th=[    8],
     | 70.00th=[    8], 80.00th=[    9], 90.00th=[   11], 95.00th=[   15],
     | 99.00th=[  129], 99.50th=[ 8768], 99.90th=[24192], 99.95th=[30336],
     | 99.99th=[41728]
    bw (KB  /s): min=    0, max=28015, per=12.73%, avg=21062.09, stdev=3164.09
    lat (usec) : 10=85.36%, 20=11.94%, 50=1.55%, 100=0.11%, 250=0.12%
    lat (usec) : 500=0.04%, 750=0.02%, 1000=0.02%
    lat (msec) : 2=0.06%, 4=0.09%, 10=0.23%, 20=0.30%, 50=0.17%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=4.62%, sys=5.28%, ctx=55211, majf=0, minf=91
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=2482084/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=9695.7MB, aggrb=165439KB/s, minb=165439KB/s, maxb=165439KB/s, mint=60012msec, maxt=60012msec

Disk stats (read/write):
  vda: ios=570/734015, merge=0/6974, ticks=960/4050752, in_queue=4051320, util=67.62%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=21859: Thu Dec 12 13:08:04 2019
  read : io=3183.3MB, bw=54325KB/s, iops=13581, runt= 60003msec
    clat (usec): min=177, max=31523, avg=571.81, stdev=426.10
     lat (usec): min=178, max=31525, avg=574.34, stdev=426.65
    clat percentiles (usec):
     |  1.00th=[  251],  5.00th=[  290], 10.00th=[  318], 20.00th=[  354],
     | 30.00th=[  394], 40.00th=[  438], 50.00th=[  494], 60.00th=[  556],
     | 70.00th=[  620], 80.00th=[  708], 90.00th=[  860], 95.00th=[ 1032],
     | 99.00th=[ 1704], 99.50th=[ 2192], 99.90th=[ 7456], 99.95th=[ 8768],
     | 99.99th=[10944]
    bw (KB  /s): min= 2080, max= 8136, per=12.52%, avg=6803.39, stdev=713.56
    lat (usec) : 250=0.91%, 500=49.97%, 750=32.72%, 1000=10.66%
    lat (msec) : 2=5.10%, 4=0.44%, 10=0.18%, 20=0.01%, 50=0.01%
  cpu          : usr=3.21%, sys=6.17%, ctx=821085, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=814921/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=3183.3MB, aggrb=54325KB/s, minb=54325KB/s, maxb=54325KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  vda: ios=814160/25, merge=0/21, ticks=309004/0, in_queue=308492, util=98.07%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=21905: Thu Dec 12 13:10:05 2019
  write: io=513932KB, bw=8564.1KB/s, iops=2141, runt= 60004msec
    clat (usec): min=124, max=75798, avg=3712.21, stdev=9522.14
     lat (usec): min=125, max=75801, avg=3715.65, stdev=9522.72
    clat percentiles (usec):
     |  1.00th=[  201],  5.00th=[  237], 10.00th=[  262], 20.00th=[  298],
     | 30.00th=[  330], 40.00th=[  366], 50.00th=[  418], 60.00th=[  482],
     | 70.00th=[  556], 80.00th=[  668], 90.00th=[25728], 95.00th=[30592],
     | 99.00th=[36096], 99.50th=[39168], 99.90th=[44800], 99.95th=[48896],
     | 99.99th=[62720]
    bw (KB  /s): min=  361, max= 1687, per=12.50%, avg=1070.72, stdev=161.85
    lat (usec) : 250=7.32%, 500=55.42%, 750=20.91%, 1000=3.87%
    lat (msec) : 2=1.37%, 4=0.23%, 10=0.16%, 20=0.05%, 50=10.64%
    lat (msec) : 100=0.04%
  cpu          : usr=0.84%, sys=3.45%, ctx=269617, majf=0, minf=64
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=128483/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=513932KB, aggrb=8564KB/s, minb=8564KB/s, maxb=8564KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  vda: ios=563/128541, merge=0/2127, ticks=392/44204, in_queue=44448, util=73.41%


fio (Япония, Токио)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=27596: Thu Dec  5 08:07:04 2019
  read : io=8536.6MB, bw=145688KB/s, iops=36421, runt= 60001msec
    clat (usec): min=1, max=8864, avg=215.91, stdev=140.30
     lat (usec): min=1, max=8865, avg=216.46, stdev=140.31
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[  159], 20.00th=[  185],
     | 30.00th=[  197], 40.00th=[  209], 50.00th=[  217], 60.00th=[  229],
     | 70.00th=[  241], 80.00th=[  258], 90.00th=[  286], 95.00th=[  314],
     | 99.00th=[  398], 99.50th=[  540], 99.90th=[ 2224], 99.95th=[ 2736],
     | 99.99th=[ 3632]
    bw (KB  /s): min=14880, max=25576, per=12.49%, avg=18199.00, stdev=2075.46
    lat (usec) : 2=0.29%, 4=7.93%, 10=0.72%, 20=0.12%, 50=0.01%
    lat (usec) : 100=0.01%, 250=66.79%, 500=23.59%, 750=0.13%, 1000=0.06%
    lat (msec) : 2=0.19%, 4=0.15%, 10=0.01%
  cpu          : usr=1.91%, sys=5.45%, ctx=1991593, majf=2, minf=94
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2185355/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=8536.6MB, aggrb=145687KB/s, minb=145687KB/s, maxb=145687KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1983091/10, merge=0/3, ticks=408440/0, in_queue=408168, util=98.38%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=27622: Thu Dec  5 08:09:12 2019
  write: io=35740MB, bw=609892KB/s, iops=152473, runt= 60007msec
    clat (usec): min=1, max=206966, avg=37.85, stdev=961.08
     lat (usec): min=1, max=206981, avg=40.42, stdev=1001.48
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    2], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    3], 90.00th=[    3], 95.00th=[    4],
     | 99.00th=[    9], 99.50th=[   14], 99.90th=[13248], 99.95th=[19584],
     | 99.99th=[33536]
    bw (KB  /s): min=    1, max=99528, per=12.64%, avg=77112.48, stdev=10362.96
    lat (usec) : 2=0.26%, 4=90.58%, 10=8.18%, 20=0.62%, 50=0.06%
    lat (usec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.02%, 10=0.08%, 20=0.10%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=2.89%, sys=7.40%, ctx=68728, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=9149450/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=35740MB, aggrb=609892KB/s, minb=609892KB/s, maxb=609892KB/s, mint=60007msec, maxt=60007msec

Disk stats (read/write):
  vda: ios=180/643486, merge=0/8512, ticks=1204/6913368, in_queue=6916056, util=99.07%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=27609: Thu Dec  5 08:08:11 2019
  read : io=7775.5MB, bw=132697KB/s, iops=33174, runt= 60002msec
    clat (usec): min=132, max=4782, avg=237.29, stdev=83.99
     lat (usec): min=133, max=4783, avg=237.83, stdev=84.00
    clat percentiles (usec):
     |  1.00th=[  161],  5.00th=[  175], 10.00th=[  185], 20.00th=[  197],
     | 30.00th=[  209], 40.00th=[  219], 50.00th=[  227], 60.00th=[  239],
     | 70.00th=[  251], 80.00th=[  266], 90.00th=[  294], 95.00th=[  318],
     | 99.00th=[  390], 99.50th=[  450], 99.90th=[ 1400], 99.95th=[ 1784],
     | 99.99th=[ 3184]
    bw (KB  /s): min=14368, max=18584, per=12.50%, avg=16593.13, stdev=701.89
    lat (usec) : 250=69.35%, 500=30.27%, 750=0.13%, 1000=0.07%
    lat (msec) : 2=0.15%, 4=0.03%, 10=0.01%
  cpu          : usr=1.77%, sys=4.65%, ctx=1994084, majf=0, minf=57
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1990521/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7775.5MB, aggrb=132696KB/s, minb=132696KB/s, maxb=132696KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  vda: ios=1983437/8, merge=0/5, ticks=409560/0, in_queue=409268, util=99.87%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=27635: Thu Dec  5 08:10:12 2019
  write: io=2355.4MB, bw=40196KB/s, iops=10048, runt= 60003msec
    clat (usec): min=59, max=50972, avg=792.04, stdev=4971.28
     lat (usec): min=60, max=50973, avg=792.63, stdev=4971.29
    clat percentiles (usec):
     |  1.00th=[   66],  5.00th=[   69], 10.00th=[   73], 20.00th=[   78],
     | 30.00th=[   83], 40.00th=[   87], 50.00th=[   92], 60.00th=[   96],
     | 70.00th=[  101], 80.00th=[  108], 90.00th=[  124], 95.00th=[  149],
     | 99.00th=[36096], 99.50th=[38144], 99.90th=[41216], 99.95th=[42240],
     | 99.99th=[44288]
    bw (KB  /s): min= 3600, max= 6696, per=12.51%, avg=5026.41, stdev=488.82
    lat (usec) : 100=67.89%, 250=29.95%, 500=0.18%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=1.94%
    lat (msec) : 100=0.01%
  cpu          : usr=0.59%, sys=2.72%, ctx=1218150, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=602963/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2355.4MB, aggrb=40195KB/s, minb=40195KB/s, maxb=40195KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  vda: ios=169/602207, merge=0/1593, ticks=12/48036, in_queue=47908, util=79.85%


fio (Нидерланды, Амстердам)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=15909: Wed Dec 11 18:43:57 2019
  read : io=7981.3MB, bw=136211KB/s, iops=34052, runt= 60001msec
    clat (usec): min=1, max=5923, avg=230.65, stdev=151.37
     lat (usec): min=1, max=5924, avg=231.24, stdev=151.40
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[  171], 20.00th=[  195],
     | 30.00th=[  209], 40.00th=[  219], 50.00th=[  229], 60.00th=[  241],
     | 70.00th=[  258], 80.00th=[  274], 90.00th=[  306], 95.00th=[  338],
     | 99.00th=[  434], 99.50th=[  524], 99.90th=[ 2672], 99.95th=[ 3312],
     | 99.99th=[ 4320]
    bw (KB  /s): min=13288, max=23776, per=12.50%, avg=17025.53, stdev=1846.05
    lat (usec) : 2=0.08%, 4=7.03%, 10=0.82%, 20=0.18%, 50=0.03%
    lat (usec) : 100=0.01%, 250=57.74%, 500=33.54%, 750=0.26%, 1000=0.06%
    lat (msec) : 2=0.07%, 4=0.17%, 10=0.02%
  cpu          : usr=1.94%, sys=6.08%, ctx=1883534, majf=2, minf=94
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2043203/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7981.3MB, aggrb=136211KB/s, minb=136211KB/s, maxb=136211KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1873042/9, merge=0/4, ticks=388148/0, in_queue=387944, util=98.33%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=15935: Wed Dec 11 18:46:05 2019
  write: io=32167MB, bw=548938KB/s, iops=137234, runt= 60005msec
    clat (usec): min=1, max=179157, avg=40.91, stdev=950.87
     lat (usec): min=1, max=179157, avg=43.74, stdev=995.56
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    3], 90.00th=[    4], 95.00th=[    5],
     | 99.00th=[    8], 99.50th=[   16], 99.90th=[14400], 99.95th=[20096],
     | 99.99th=[32384]
    bw (KB  /s): min=    0, max=94521, per=12.76%, avg=70066.25, stdev=10498.62
    lat (usec) : 2=0.09%, 4=80.81%, 10=18.19%, 20=0.47%, 50=0.08%
    lat (usec) : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.03%, 10=0.09%, 20=0.11%, 50=0.05%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=3.05%, sys=7.35%, ctx=68257, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=8234759/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=32167MB, aggrb=548938KB/s, minb=548938KB/s, maxb=548938KB/s, mint=60005msec, maxt=60005msec

Disk stats (read/write):
  vda: ios=180/704167, merge=0/7607, ticks=1196/6839992, in_queue=6841764, util=98.87%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=15922: Wed Dec 11 18:45:04 2019
  read : io=7375.3MB, bw=125867KB/s, iops=31466, runt= 60002msec
    clat (usec): min=136, max=10916, avg=250.04, stdev=101.92
     lat (usec): min=136, max=10916, avg=250.63, stdev=101.96
    clat percentiles (usec):
     |  1.00th=[  165],  5.00th=[  179], 10.00th=[  189], 20.00th=[  203],
     | 30.00th=[  215], 40.00th=[  225], 50.00th=[  235], 60.00th=[  247],
     | 70.00th=[  262], 80.00th=[  278], 90.00th=[  314], 95.00th=[  354],
     | 99.00th=[  564], 99.50th=[  676], 99.90th=[ 1032], 99.95th=[ 1512],
     | 99.99th=[ 3920]
    bw (KB  /s): min= 8752, max=18208, per=12.51%, avg=15747.04, stdev=1769.43
    lat (usec) : 250=62.53%, 500=35.96%, 750=1.19%, 1000=0.21%
    lat (msec) : 2=0.08%, 4=0.03%, 10=0.01%, 20=0.01%
  cpu          : usr=1.77%, sys=5.13%, ctx=1891322, majf=0, minf=88
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1888072/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7375.3MB, aggrb=125867KB/s, minb=125867KB/s, maxb=125867KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  vda: ios=1886265/8, merge=0/5, ticks=420800/0, in_queue=420836, util=99.87%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=15948: Wed Dec 11 18:47:05 2019
  write: io=1885.3MB, bw=32172KB/s, iops=8043, runt= 60004msec
    clat (usec): min=63, max=64992, avg=989.65, stdev=5493.21
     lat (usec): min=64, max=64993, avg=990.38, stdev=5493.23
    clat percentiles (usec):
     |  1.00th=[   73],  5.00th=[   81], 10.00th=[   86], 20.00th=[   92],
     | 30.00th=[   97], 40.00th=[  102], 50.00th=[  107], 60.00th=[  114],
     | 70.00th=[  123], 80.00th=[  139], 90.00th=[  171], 95.00th=[  221],
     | 99.00th=[35584], 99.50th=[37632], 99.90th=[42752], 99.95th=[44800],
     | 99.99th=[55552]
    bw (KB  /s): min= 2203, max= 5426, per=12.52%, avg=4027.61, stdev=480.41
    lat (usec) : 100=34.99%, 250=61.01%, 500=1.28%, 750=0.09%, 1000=0.05%
    lat (msec) : 2=0.06%, 4=0.03%, 10=0.01%, 20=0.01%, 50=2.46%
    lat (msec) : 100=0.02%
  cpu          : usr=0.60%, sys=2.78%, ctx=977664, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=482614/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=1885.3MB, aggrb=32172KB/s, minb=32172KB/s, maxb=32172KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  vda: ios=170/482245, merge=0/1678, ticks=132/47556, in_queue=47540, util=79.02%


fio (Сингапур)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30926: Wed Dec 11 14:46:27 2019
  read : io=7697.7MB, bw=131370KB/s, iops=32842, runt= 60001msec
    clat (usec): min=1, max=34126, avg=238.32, stdev=232.97
     lat (usec): min=2, max=34127, avg=239.17, stdev=232.98
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[  165], 20.00th=[  189],
     | 30.00th=[  201], 40.00th=[  213], 50.00th=[  227], 60.00th=[  241],
     | 70.00th=[  262], 80.00th=[  290], 90.00th=[  334], 95.00th=[  378],
     | 99.00th=[  540], 99.50th=[  844], 99.90th=[ 2512], 99.95th=[ 3056],
     | 99.99th=[ 4640]
    bw (KB  /s): min=11272, max=22698, per=12.50%, avg=16416.34, stdev=1953.87
    lat (usec) : 2=0.01%, 4=6.34%, 10=1.30%, 20=0.20%, 50=0.02%
    lat (usec) : 100=0.01%, 250=56.94%, 500=33.94%, 750=0.68%, 1000=0.16%
    lat (msec) : 2=0.20%, 4=0.19%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=2.34%, sys=5.49%, ctx=1820233, majf=2, minf=95
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1970586/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7697.7MB, aggrb=131370KB/s, minb=131370KB/s, maxb=131370KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1811866/16, merge=0/8, ticks=385580/0, in_queue=385364, util=98.23%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30954: Wed Dec 11 14:48:35 2019
  write: io=26015MB, bw=443885KB/s, iops=110971, runt= 60015msec
    clat (usec): min=1, max=192632, avg=45.41, stdev=880.56
     lat (usec): min=2, max=192634, avg=50.34, stdev=950.31
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    5],
     | 99.00th=[   18], 99.50th=[   35], 99.90th=[14656], 99.95th=[20096],
     | 99.99th=[32128]
    bw (KB  /s): min=    1, max=66360, per=12.66%, avg=56217.23, stdev=5810.74
    lat (usec) : 2=0.01%, 4=66.86%, 10=30.89%, 20=1.55%, 50=0.21%
    lat (usec) : 100=0.05%, 250=0.03%, 500=0.01%, 750=0.02%, 1000=0.01%
    lat (msec) : 2=0.03%, 4=0.04%, 10=0.12%, 20=0.13%, 50=0.05%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=4.75%, sys=5.94%, ctx=77362, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=6659941/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=26015MB, aggrb=443885KB/s, minb=443885KB/s, maxb=443885KB/s, mint=60015msec, maxt=60015msec

Disk stats (read/write):
  vda: ios=171/1050982, merge=0/9077, ticks=640/6829808, in_queue=6830932, util=98.80%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30940: Wed Dec 11 14:47:34 2019
  read : io=7409.7MB, bw=126453KB/s, iops=31613, runt= 60002msec
    clat (usec): min=130, max=24402, avg=247.64, stdev=134.86
     lat (usec): min=131, max=24403, avg=248.50, stdev=134.89
    clat percentiles (usec):
     |  1.00th=[  157],  5.00th=[  171], 10.00th=[  181], 20.00th=[  193],
     | 30.00th=[  205], 40.00th=[  215], 50.00th=[  225], 60.00th=[  239],
     | 70.00th=[  258], 80.00th=[  282], 90.00th=[  326], 95.00th=[  370],
     | 99.00th=[  556], 99.50th=[  804], 99.90th=[ 1720], 99.95th=[ 2480],
     | 99.99th=[ 3952]
    bw (KB  /s): min= 4128, max=17392, per=12.50%, avg=15809.98, stdev=1368.84
    lat (usec) : 250=66.66%, 500=32.07%, 750=0.69%, 1000=0.28%
    lat (msec) : 2=0.23%, 4=0.06%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=2.51%, sys=4.77%, ctx=1900108, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1896859/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7409.7MB, aggrb=126453KB/s, minb=126453KB/s, maxb=126453KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  vda: ios=1893026/8, merge=0/5, ticks=404240/0, in_queue=404112, util=99.88%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30967: Wed Dec 11 14:49:35 2019
  write: io=2091.3MB, bw=35686KB/s, iops=8921, runt= 60002msec
    clat (usec): min=60, max=76654, avg=890.95, stdev=5124.75
     lat (usec): min=61, max=76655, avg=891.89, stdev=5124.77
    clat percentiles (usec):
     |  1.00th=[   66],  5.00th=[   69], 10.00th=[   72], 20.00th=[   79],
     | 30.00th=[   84], 40.00th=[   89], 50.00th=[   95], 60.00th=[  100],
     | 70.00th=[  108], 80.00th=[  121], 90.00th=[  167], 95.00th=[  233],
     | 99.00th=[34560], 99.50th=[36608], 99.90th=[39680], 99.95th=[42240],
     | 99.99th=[56576]
    bw (KB  /s): min= 2831, max= 5484, per=12.50%, avg=4461.68, stdev=440.44
    lat (usec) : 100=58.61%, 250=37.16%, 500=1.74%, 750=0.08%, 1000=0.02%
    lat (msec) : 2=0.03%, 4=0.02%, 10=0.01%, 20=0.01%, 50=2.31%
    lat (msec) : 100=0.02%
  cpu          : usr=0.66%, sys=2.82%, ctx=1083156, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=535303/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2091.3MB, aggrb=35685KB/s, minb=35685KB/s, maxb=35685KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  vda: ios=170/534669, merge=0/1610, ticks=132/49496, in_queue=49572, util=82.41%


fio (Великобритания, Лондон)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=20488: Sun Dec 15 16:24:24 2019
  read : io=743988KB, bw=12399KB/s, iops=3099, runt= 60004msec
    clat (usec): min=142, max=15787, avg=2574.70, stdev=1413.98
     lat (usec): min=143, max=15788, avg=2575.38, stdev=1414.09
    clat percentiles (usec):
     |  1.00th=[  185],  5.00th=[  221], 10.00th=[  266], 20.00th=[ 1256],
     | 30.00th=[ 2128], 40.00th=[ 2480], 50.00th=[ 2768], 60.00th=[ 3024],
     | 70.00th=[ 3280], 80.00th=[ 3600], 90.00th=[ 4128], 95.00th=[ 4640],
     | 99.00th=[ 6176], 99.50th=[ 6880], 99.90th=[ 8256], 99.95th=[ 8896],
     | 99.99th=[10304]
    bw (KB  /s): min= 1136, max= 4691, per=12.52%, avg=1551.89, stdev=544.41
    lat (usec) : 250=8.44%, 500=8.36%, 750=1.31%, 1000=0.96%
    lat (msec) : 2=8.13%, 4=61.25%, 10=11.53%, 20=0.01%
  cpu          : usr=0.30%, sys=0.92%, ctx=186568, majf=0, minf=80
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=185997/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=743988KB, aggrb=12398KB/s, minb=12398KB/s, maxb=12398KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  vda: ios=186162/8, merge=0/4, ticks=470968/44, in_queue=470988, util=99.87%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=20514: Sun Dec 15 16:26:34 2019
  write: io=3170.2MB, bw=54082KB/s, iops=13520, runt= 60040msec
    clat (usec): min=1, max=2811.6K, avg=487.71, stdev=17323.71
     lat (usec): min=1, max=2811.6K, avg=489.04, stdev=17329.86
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    4], 90.00th=[    4], 95.00th=[    6],
     | 99.00th=[   14], 99.50th=[   31], 99.90th=[126464], 99.95th=[189440],
     | 99.99th=[757760]
    bw (KB  /s): min=    0, max=40279, per=18.07%, avg=9773.52, stdev=7070.57
    lat (usec) : 2=0.05%, 4=78.16%, 10=19.65%, 20=1.52%, 50=0.15%
    lat (usec) : 100=0.05%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.02%, 10=0.03%, 20=0.04%, 50=0.09%
    lat (msec) : 100=0.07%, 250=0.11%, 500=0.02%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=0.28%, sys=0.87%, ctx=15613, majf=0, minf=85
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=811771/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=3170.2MB, aggrb=54082KB/s, minb=54082KB/s, maxb=54082KB/s, mint=60040msec, maxt=60040msec

Disk stats (read/write):
  vda: ios=170/84307, merge=0/6212, ticks=592/2584112, in_queue=2586072, util=99.40%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=20501: Sun Dec 15 16:25:34 2019
  read : io=1298.7MB, bw=22162KB/s, iops=5540, runt= 60004msec
    clat (usec): min=136, max=16560, avg=1438.74, stdev=1490.59
     lat (usec): min=137, max=16560, avg=1439.37, stdev=1490.64
    clat percentiles (usec):
     |  1.00th=[  171],  5.00th=[  189], 10.00th=[  203], 20.00th=[  225],
     | 30.00th=[  249], 40.00th=[  286], 50.00th=[  402], 60.00th=[ 1496],
     | 70.00th=[ 2448], 80.00th=[ 2992], 90.00th=[ 3568], 95.00th=[ 4048],
     | 99.00th=[ 5344], 99.50th=[ 6112], 99.90th=[ 7776], 99.95th=[ 8640],
     | 99.99th=[10176]
    bw (KB  /s): min= 1117, max= 8504, per=12.52%, avg=2775.28, stdev=1765.96
    lat (usec) : 250=30.63%, 500=21.52%, 750=3.15%, 1000=1.92%
    lat (msec) : 2=6.75%, 4=30.70%, 10=5.32%, 20=0.01%
  cpu          : usr=0.46%, sys=1.15%, ctx=333216, majf=0, minf=80
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=332448/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=1298.7MB, aggrb=22161KB/s, minb=22161KB/s, maxb=22161KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  vda: ios=332167/8, merge=0/5, ticks=464228/760, in_queue=464876, util=99.86%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=20527: Sun Dec 15 16:27:36 2019
  write: io=339804KB, bw=5661.7KB/s, iops=1415, runt= 60019msec
    clat (usec): min=66, max=1766.6K, avg=5646.40, stdev=40055.68
     lat (usec): min=66, max=1766.6K, avg=5647.17, stdev=40055.71
    clat percentiles (usec):
     |  1.00th=[   73],  5.00th=[   81], 10.00th=[   86], 20.00th=[   94],
     | 30.00th=[  101], 40.00th=[  107], 50.00th=[  115], 60.00th=[  125],
     | 70.00th=[  143], 80.00th=[  179], 90.00th=[ 1012], 95.00th=[ 6496],
     | 99.00th=[177152], 99.50th=[242688], 99.90th=[528384], 99.95th=[724992],
     | 99.99th=[1105920]
    bw (KB  /s): min=    2, max= 2640, per=13.35%, avg=755.86, stdev=514.57
    lat (usec) : 100=28.39%, 250=57.85%, 500=2.51%, 750=0.63%, 1000=0.59%
    lat (msec) : 2=1.96%, 4=2.08%, 10=1.67%, 20=0.56%, 50=1.20%
    lat (msec) : 100=0.65%, 250=1.47%, 500=0.33%, 750=0.07%, 1000=0.02%
    lat (msec) : 2000=0.02%
  cpu          : usr=0.13%, sys=0.62%, ctx=172753, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=84951/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=339804KB, aggrb=5661KB/s, minb=5661KB/s, maxb=5661KB/s, mint=60019msec, maxt=60019msec

Disk stats (read/write):
  vda: ios=172/84970, merge=0/2006, ticks=544/57484, in_queue=58024, util=94.67%


fio (США, Атланта)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=15476: Sun Dec 15 19:29:14 2019
  read : io=7837.8MB, bw=133749KB/s, iops=33437, runt= 60001msec
    clat (usec): min=1, max=7324, avg=234.11, stdev=141.99
     lat (usec): min=2, max=7325, avg=234.93, stdev=142.02
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[  167], 20.00th=[  193],
     | 30.00th=[  207], 40.00th=[  219], 50.00th=[  231], 60.00th=[  245],
     | 70.00th=[  262], 80.00th=[  286], 90.00th=[  322], 95.00th=[  358],
     | 99.00th=[  458], 99.50th=[  556], 99.90th=[ 2352], 99.95th=[ 2864],
     | 99.99th=[ 3888]
    bw (KB  /s): min=11944, max=23232, per=12.49%, avg=16709.43, stdev=1866.94
    lat (usec) : 2=0.01%, 4=6.74%, 10=1.06%, 20=0.17%, 50=0.03%
    lat (usec) : 100=0.01%, 250=55.01%, 500=36.31%, 750=0.36%, 1000=0.09%
    lat (msec) : 2=0.06%, 4=0.16%, 10=0.01%
  cpu          : usr=2.32%, sys=5.36%, ctx=1851110, majf=2, minf=94
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2006274/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7837.8MB, aggrb=133749KB/s, minb=133749KB/s, maxb=133749KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1845303/14, merge=0/7, ticks=397380/0, in_queue=397304, util=98.34%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=15511: Sun Dec 15 19:31:22 2019
  write: io=26235MB, bw=447631KB/s, iops=111907, runt= 60014msec
    clat (usec): min=1, max=117339, avg=44.70, stdev=861.50
     lat (usec): min=2, max=136215, avg=49.63, stdev=930.64
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    5],
     | 99.00th=[   19], 99.50th=[   40], 99.90th=[14272], 99.95th=[19328],
     | 99.99th=[31104]
    bw (KB  /s): min=    1, max=65736, per=12.63%, avg=56547.13, stdev=5586.94
    lat (usec) : 2=0.01%, 4=68.30%, 10=29.61%, 20=1.32%, 50=0.29%
    lat (usec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.02%, 1000=0.01%
    lat (msec) : 2=0.03%, 4=0.04%, 10=0.12%, 20=0.13%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=4.95%, sys=5.87%, ctx=77785, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=6716037/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=26235MB, aggrb=447631KB/s, minb=447631KB/s, maxb=447631KB/s, mint=60014msec, maxt=60014msec

Disk stats (read/write):
  vda: ios=192/1075724, merge=0/8974, ticks=768/6927672, in_queue=6933620, util=98.80%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=15496: Sun Dec 15 19:30:21 2019
  read : io=7593.5MB, bw=129593KB/s, iops=32398, runt= 60001msec
    clat (usec): min=135, max=6091, avg=241.58, stdev=71.45
     lat (usec): min=135, max=6091, avg=242.43, stdev=71.49
    clat percentiles (usec):
     |  1.00th=[  161],  5.00th=[  175], 10.00th=[  185], 20.00th=[  199],
     | 30.00th=[  209], 40.00th=[  219], 50.00th=[  231], 60.00th=[  243],
     | 70.00th=[  258], 80.00th=[  278], 90.00th=[  310], 95.00th=[  342],
     | 99.00th=[  422], 99.50th=[  462], 99.90th=[  708], 99.95th=[  980],
     | 99.99th=[ 2448]
    bw (KB  /s): min=13104, max=17648, per=12.50%, avg=16200.46, stdev=780.38
    lat (usec) : 250=65.19%, 500=34.51%, 750=0.22%, 1000=0.04%
    lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%
  cpu          : usr=2.51%, sys=4.83%, ctx=1947114, majf=0, minf=73
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1943932/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7593.5MB, aggrb=129593KB/s, minb=129593KB/s, maxb=129593KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1937903/10, merge=0/7, ticks=409816/0, in_queue=409684, util=99.82%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=15524: Sun Dec 15 19:32:22 2019
  write: io=2278.9MB, bw=38891KB/s, iops=9722, runt= 60003msec
    clat (usec): min=60, max=51553, avg=817.32, stdev=4886.56
     lat (usec): min=60, max=51554, avg=818.22, stdev=4886.59
    clat percentiles (usec):
     |  1.00th=[   66],  5.00th=[   68], 10.00th=[   70], 20.00th=[   76],
     | 30.00th=[   81], 40.00th=[   85], 50.00th=[   91], 60.00th=[   96],
     | 70.00th=[  102], 80.00th=[  112], 90.00th=[  139], 95.00th=[  181],
     | 99.00th=[34048], 99.50th=[36096], 99.90th=[38656], 99.95th=[39680],
     | 99.99th=[41728]
    bw (KB  /s): min= 3460, max= 5968, per=12.51%, avg=4863.46, stdev=404.00
    lat (usec) : 100=65.85%, 250=31.55%, 500=0.42%, 750=0.02%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=2.14%
    lat (msec) : 100=0.01%
  cpu          : usr=0.78%, sys=2.71%, ctx=1179671, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=583392/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2278.9MB, aggrb=38890KB/s, minb=38890KB/s, maxb=38890KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  vda: ios=612/582769, merge=0/1520, ticks=392/47444, in_queue=47752, util=78.96%


fio (США, Чикаго)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=14831: Mon Dec 16 01:33:33 2019
  read : io=8035.5MB, bw=137136KB/s, iops=34284, runt= 60001msec
    clat (usec): min=1, max=4262, avg=227.92, stdev=133.01
     lat (usec): min=2, max=4267, avg=228.79, stdev=133.05
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[  165], 20.00th=[  189],
     | 30.00th=[  201], 40.00th=[  213], 50.00th=[  225], 60.00th=[  239],
     | 70.00th=[  255], 80.00th=[  278], 90.00th=[  314], 95.00th=[  350],
     | 99.00th=[  458], 99.50th=[  548], 99.90th=[ 2256], 99.95th=[ 2768],
     | 99.99th=[ 3408]
    bw (KB  /s): min=13152, max=23400, per=12.49%, avg=17131.96, stdev=2002.66
    lat (usec) : 2=0.01%, 4=6.94%, 10=0.98%, 20=0.21%, 50=0.05%
    lat (usec) : 100=0.01%, 250=58.57%, 500=32.57%, 750=0.41%, 1000=0.06%
    lat (msec) : 2=0.05%, 4=0.15%, 10=0.01%
  cpu          : usr=2.58%, sys=5.57%, ctx=1895150, majf=2, minf=94
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2057075/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=8035.5MB, aggrb=137136KB/s, minb=137136KB/s, maxb=137136KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1884570/9, merge=0/4, ticks=377920/0, in_queue=377704, util=98.11%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=14858: Mon Dec 16 01:35:41 2019
  write: io=26627MB, bw=454312KB/s, iops=113578, runt= 60016msec
    clat (usec): min=1, max=140433, avg=44.25, stdev=841.09
     lat (usec): min=2, max=153527, avg=48.93, stdev=907.33
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    4], 90.00th=[    4], 95.00th=[    5],
     | 99.00th=[   19], 99.50th=[   38], 99.90th=[14144], 99.95th=[19584],
     | 99.99th=[30592]
    bw (KB  /s): min=    1, max=67693, per=12.64%, avg=57424.28, stdev=5407.98
    lat (usec) : 2=0.01%, 4=70.71%, 10=26.77%, 20=1.74%, 50=0.30%
    lat (usec) : 100=0.04%, 250=0.02%, 500=0.02%, 750=0.02%, 1000=0.01%
    lat (msec) : 2=0.03%, 4=0.04%, 10=0.12%, 20=0.13%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=4.94%, sys=5.93%, ctx=79995, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=6816502/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=26627MB, aggrb=454312KB/s, minb=454312KB/s, maxb=454312KB/s, mint=60016msec, maxt=60016msec

Disk stats (read/write):
  vda: ios=499/1074165, merge=0/9300, ticks=2744/6945264, in_queue=6948456, util=98.75%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=14845: Mon Dec 16 01:34:41 2019
  read : io=7746.9MB, bw=132208KB/s, iops=33052, runt= 60002msec
    clat (usec): min=132, max=31092, avg=236.79, stdev=90.81
     lat (usec): min=132, max=31092, avg=237.63, stdev=90.84
    clat percentiles (usec):
     |  1.00th=[  157],  5.00th=[  171], 10.00th=[  181], 20.00th=[  193],
     | 30.00th=[  205], 40.00th=[  215], 50.00th=[  225], 60.00th=[  237],
     | 70.00th=[  253], 80.00th=[  274], 90.00th=[  306], 95.00th=[  338],
     | 99.00th=[  414], 99.50th=[  462], 99.90th=[  708], 99.95th=[  924],
     | 99.99th=[ 3376]
    bw (KB  /s): min=13992, max=17600, per=12.50%, avg=16531.33, stdev=620.44
    lat (usec) : 250=68.62%, 500=31.05%, 750=0.25%, 1000=0.04%
    lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=2.35%, sys=4.85%, ctx=1987876, majf=0, minf=73
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1983189/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7746.9MB, aggrb=132208KB/s, minb=132208KB/s, maxb=132208KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  vda: ios=1977990/8, merge=0/5, ticks=408236/0, in_queue=407928, util=99.82%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=14873: Mon Dec 16 01:36:42 2019
  write: io=2261.6MB, bw=38594KB/s, iops=9648, runt= 60004msec
    clat (usec): min=61, max=44709, avg=823.69, stdev=4921.76
     lat (usec): min=61, max=44709, avg=824.57, stdev=4921.78
    clat percentiles (usec):
     |  1.00th=[   67],  5.00th=[   69], 10.00th=[   72], 20.00th=[   77],
     | 30.00th=[   82], 40.00th=[   86], 50.00th=[   91], 60.00th=[   96],
     | 70.00th=[  102], 80.00th=[  112], 90.00th=[  141], 95.00th=[  185],
     | 99.00th=[34560], 99.50th=[36096], 99.90th=[38144], 99.95th=[39168],
     | 99.99th=[40704]
    bw (KB  /s): min= 3547, max= 5952, per=12.50%, avg=4824.65, stdev=445.14
    lat (usec) : 100=65.60%, 250=31.76%, 500=0.46%, 750=0.03%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=2.14%
  cpu          : usr=0.85%, sys=2.61%, ctx=1170628, majf=0, minf=73
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=578946/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2261.6MB, aggrb=38593KB/s, minb=38593KB/s, maxb=38593KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  vda: ios=169/578268, merge=0/1640, ticks=164/47368, in_queue=47436, util=78.80%


fio (США, Даллас)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30640: Mon Dec 16 01:36:46 2019
  read : io=8101.4MB, bw=138255KB/s, iops=34563, runt= 60001msec
    clat (usec): min=1, max=6512, avg=227.73, stdev=127.95
     lat (usec): min=1, max=6513, avg=228.26, stdev=127.96
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[  171], 20.00th=[  197],
     | 30.00th=[  209], 40.00th=[  221], 50.00th=[  231], 60.00th=[  243],
     | 70.00th=[  258], 80.00th=[  274], 90.00th=[  298], 95.00th=[  326],
     | 99.00th=[  402], 99.50th=[  478], 99.90th=[ 2160], 99.95th=[ 2576],
     | 99.99th=[ 3280]
    bw (KB  /s): min=13720, max=24296, per=12.49%, avg=17270.28, stdev=1840.54
    lat (usec) : 2=0.19%, 4=7.28%, 10=0.68%, 20=0.10%, 50=0.01%
    lat (usec) : 100=0.01%, 250=57.33%, 500=33.96%, 750=0.17%, 1000=0.03%
    lat (msec) : 2=0.10%, 4=0.14%, 10=0.01%
  cpu          : usr=1.64%, sys=5.37%, ctx=1906830, majf=3, minf=93
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2073857/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=8101.4MB, aggrb=138254KB/s, minb=138254KB/s, maxb=138254KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1898815/11, merge=0/3, ticks=429144/0, in_queue=428844, util=98.21%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30667: Mon Dec 16 01:38:53 2019
  write: io=37120MB, bw=633470KB/s, iops=158367, runt= 60005msec
    clat (usec): min=1, max=268949, avg=35.49, stdev=825.71
     lat (usec): min=1, max=268950, avg=38.11, stdev=871.35
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    2], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    3], 90.00th=[    3], 95.00th=[    4],
     | 99.00th=[    7], 99.50th=[   14], 99.90th=[12992], 99.95th=[18304],
     | 99.99th=[30080]
    bw (KB  /s): min=    0, max=113093, per=12.76%, avg=80836.06, stdev=9692.35
    lat (usec) : 2=0.17%, 4=90.19%, 10=8.87%, 20=0.40%, 50=0.04%
    lat (usec) : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.02%, 10=0.08%, 20=0.10%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=2.99%, sys=7.69%, ctx=71565, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=9502839/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=37120MB, aggrb=633469KB/s, minb=633469KB/s, maxb=633469KB/s, mint=60005msec, maxt=60005msec

Disk stats (read/write):
  vda: ios=178/767585, merge=0/7960, ticks=1388/6914052, in_queue=6915356, util=98.99%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30654: Mon Dec 16 01:37:53 2019
  read : io=7669.1MB, bw=130896KB/s, iops=32724, runt= 60002msec
    clat (usec): min=138, max=4498, avg=240.61, stdev=71.44
     lat (usec): min=139, max=4499, avg=241.16, stdev=71.45
    clat percentiles (usec):
     |  1.00th=[  165],  5.00th=[  179], 10.00th=[  189], 20.00th=[  203],
     | 30.00th=[  213], 40.00th=[  223], 50.00th=[  233], 60.00th=[  243],
     | 70.00th=[  258], 80.00th=[  274], 90.00th=[  298], 95.00th=[  322],
     | 99.00th=[  386], 99.50th=[  422], 99.90th=[  668], 99.95th=[ 1240],
     | 99.99th=[ 3408]
    bw (KB  /s): min=14360, max=17592, per=12.51%, avg=16370.02, stdev=521.56
    lat (usec) : 250=65.23%, 500=34.55%, 750=0.13%, 1000=0.02%
    lat (msec) : 2=0.04%, 4=0.02%, 10=0.01%
  cpu          : usr=1.77%, sys=4.75%, ctx=1966567, majf=0, minf=72
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1963511/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7669.1MB, aggrb=130896KB/s, minb=130896KB/s, maxb=130896KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  vda: ios=1959684/10, merge=0/4, ticks=400752/0, in_queue=400488, util=99.80%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30680: Mon Dec 16 01:39:54 2019
  write: io=2169.8MB, bw=37028KB/s, iops=9256, runt= 60005msec
    clat (usec): min=64, max=65768, avg=859.93, stdev=5258.49
     lat (usec): min=64, max=65768, avg=860.55, stdev=5258.50
    clat percentiles (usec):
     |  1.00th=[   70],  5.00th=[   74], 10.00th=[   78], 20.00th=[   84],
     | 30.00th=[   89], 40.00th=[   94], 50.00th=[   99], 60.00th=[  104],
     | 70.00th=[  110], 80.00th=[  119], 90.00th=[  139], 95.00th=[  165],
     | 99.00th=[37632], 99.50th=[39680], 99.90th=[42240], 99.95th=[42752],
     | 99.99th=[44288]
    bw (KB  /s): min= 2715, max= 6448, per=12.51%, avg=4630.24, stdev=529.55
    lat (usec) : 100=51.63%, 250=45.89%, 500=0.36%, 750=0.03%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=2.04%
    lat (msec) : 100=0.01%
  cpu          : usr=0.56%, sys=2.64%, ctx=1122771, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=555463/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2169.8MB, aggrb=37027KB/s, minb=37027KB/s, maxb=37027KB/s, mint=60005msec, maxt=60005msec

Disk stats (read/write):
  vda: ios=170/554923, merge=0/1544, ticks=80/48648, in_queue=48628, util=80.92%


fio (США, Лос-Анджелес)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30302: Mon Dec 16 01:40:53 2019
  read : io=4527.3MB, bw=77216KB/s, iops=19304, runt= 60038msec
    clat (usec): min=2, max=1002.6K, avg=409.03, stdev=3749.04
     lat (usec): min=2, max=1002.6K, avg=409.87, stdev=3749.05
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    7], 10.00th=[  177], 20.00th=[  195],
     | 30.00th=[  209], 40.00th=[  219], 50.00th=[  233], 60.00th=[  247],
     | 70.00th=[  266], 80.00th=[  298], 90.00th=[  354], 95.00th=[  426],
     | 99.00th=[ 1736], 99.50th=[ 6176], 99.90th=[41728], 99.95th=[44288],
     | 99.99th=[55040]
    bw (KB  /s): min=    4, max=21176, per=12.78%, avg=9869.13, stdev=6950.72
    lat (usec) : 4=3.86%, 10=1.28%, 20=0.21%, 50=0.02%, 100=0.01%
    lat (usec) : 250=56.16%, 500=34.99%, 750=1.52%, 1000=0.64%
    lat (msec) : 2=0.37%, 4=0.23%, 10=0.38%, 20=0.04%, 50=0.27%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=1.49%, sys=3.24%, ctx=1100743, majf=0, minf=84
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1158981/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4527.3MB, aggrb=77216KB/s, minb=77216KB/s, maxb=77216KB/s, mint=60038msec, maxt=60038msec

Disk stats (read/write):
  vda: ios=1096289/9, merge=0/4, ticks=432752/56, in_queue=432560, util=99.17%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30328: Mon Dec 16 01:43:06 2019
  write: io=27193MB, bw=464019KB/s, iops=116004, runt= 60009msec
    clat (usec): min=1, max=176038, avg=43.66, stdev=914.60
     lat (usec): min=2, max=200012, avg=48.16, stdev=982.18
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    4], 90.00th=[    4], 95.00th=[    5],
     | 99.00th=[   15], 99.50th=[   23], 99.90th=[14912], 99.95th=[20096],
     | 99.99th=[32128]
    bw (KB  /s): min=    1, max=72232, per=12.65%, avg=58689.60, stdev=6954.65
    lat (usec) : 2=0.01%, 4=71.10%, 10=27.12%, 20=1.20%, 50=0.15%
    lat (usec) : 100=0.04%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.03%, 10=0.10%, 20=0.12%, 50=0.05%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=4.91%, sys=5.92%, ctx=71147, majf=0, minf=60
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=6961325/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=27193MB, aggrb=464018KB/s, minb=464018KB/s, maxb=464018KB/s, mint=60009msec, maxt=60009msec

Disk stats (read/write):
  vda: ios=174/804965, merge=0/9109, ticks=844/6904508, in_queue=6906288, util=98.78%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30315: Mon Dec 16 01:42:06 2019
  read : io=4759.4MB, bw=81225KB/s, iops=20306, runt= 60001msec
    clat (usec): min=134, max=83824, avg=388.59, stdev=2280.97
     lat (usec): min=134, max=83825, avg=389.43, stdev=2280.98
    clat percentiles (usec):
     |  1.00th=[  161],  5.00th=[  175], 10.00th=[  185], 20.00th=[  199],
     | 30.00th=[  211], 40.00th=[  221], 50.00th=[  233], 60.00th=[  247],
     | 70.00th=[  266], 80.00th=[  294], 90.00th=[  342], 95.00th=[  394],
     | 99.00th=[  708], 99.50th=[ 3728], 99.90th=[43776], 99.95th=[46848],
     | 99.99th=[55552]
    bw (KB  /s): min=   77, max=16936, per=12.65%, avg=10274.83, stdev=6866.05
    lat (usec) : 250=61.27%, 500=36.86%, 750=0.93%, 1000=0.16%
    lat (msec) : 2=0.20%, 4=0.10%, 10=0.17%, 20=0.03%, 50=0.27%
    lat (msec) : 100=0.02%
  cpu          : usr=1.63%, sys=3.04%, ctx=1220916, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1218402/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4759.4MB, aggrb=81225KB/s, minb=81225KB/s, maxb=81225KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1216358/8, merge=0/5, ticks=429456/96, in_queue=429404, util=99.85%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30341: Mon Dec 16 01:44:07 2019
  write: io=1957.9MB, bw=33412KB/s, iops=8352, runt= 60003msec
    clat (usec): min=60, max=282347, avg=952.06, stdev=5702.32
     lat (usec): min=61, max=282348, avg=952.98, stdev=5702.34
    clat percentiles (usec):
     |  1.00th=[   68],  5.00th=[   72], 10.00th=[   76], 20.00th=[   83],
     | 30.00th=[   88], 40.00th=[   93], 50.00th=[   98], 60.00th=[  104],
     | 70.00th=[  111], 80.00th=[  125], 90.00th=[  177], 95.00th=[  243],
     | 99.00th=[36096], 99.50th=[38144], 99.90th=[43264], 99.95th=[46848],
     | 99.99th=[112128]
    bw (KB  /s): min= 1125, max= 5928, per=12.51%, avg=4180.03, stdev=644.04
    lat (usec) : 100=52.16%, 250=43.18%, 500=2.04%, 750=0.12%, 1000=0.05%
    lat (msec) : 2=0.07%, 4=0.04%, 10=0.02%, 20=0.01%, 50=2.29%
    lat (msec) : 100=0.03%, 250=0.01%, 500=0.01%
  cpu          : usr=0.74%, sys=2.51%, ctx=1014281, majf=0, minf=68
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=501203/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=1957.9MB, aggrb=33411KB/s, minb=33411KB/s, maxb=33411KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  vda: ios=173/500597, merge=0/1694, ticks=236/48308, in_queue=48456, util=80.38%


fio (США, Майами)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29120: Mon Dec 16 01:44:27 2019
  read : io=9084.3MB, bw=155035KB/s, iops=38758, runt= 60001msec
    clat (usec): min=1, max=4963, avg=202.89, stdev=119.18
     lat (usec): min=1, max=4964, avg=203.40, stdev=119.20
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[  153], 20.00th=[  177],
     | 30.00th=[  189], 40.00th=[  199], 50.00th=[  209], 60.00th=[  217],
     | 70.00th=[  229], 80.00th=[  243], 90.00th=[  266], 95.00th=[  290],
     | 99.00th=[  350], 99.50th=[  394], 99.90th=[ 2064], 99.95th=[ 2416],
     | 99.99th=[ 3216]
    bw (KB  /s): min=15200, max=26664, per=12.50%, avg=19375.82, stdev=2121.46
    lat (usec) : 2=0.37%, 4=8.07%, 10=0.62%, 20=0.14%, 50=0.03%
    lat (usec) : 100=0.01%, 250=74.10%, 500=16.39%, 750=0.05%, 1000=0.02%
    lat (msec) : 2=0.10%, 4=0.12%, 10=0.01%
  cpu          : usr=1.75%, sys=5.59%, ctx=2120348, majf=2, minf=85
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2325570/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=9084.3MB, aggrb=155035KB/s, minb=155035KB/s, maxb=155035KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=2106654/9, merge=0/4, ticks=407152/0, in_queue=406904, util=98.14%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29147: Mon Dec 16 01:46:35 2019
  write: io=36107MB, bw=616162KB/s, iops=154040, runt= 60007msec
    clat (usec): min=1, max=249081, avg=36.39, stdev=848.32
     lat (usec): min=1, max=249082, avg=39.14, stdev=895.29
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    3], 90.00th=[    3], 95.00th=[    4],
     | 99.00th=[   17], 99.50th=[   19], 99.90th=[13248], 99.95th=[19072],
     | 99.99th=[31872]
    bw (KB  /s): min=    1, max=116880, per=12.62%, avg=77787.91, stdev=9285.64
    lat (usec) : 2=0.15%, 4=92.02%, 10=6.23%, 20=1.17%, 50=0.11%
    lat (usec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.02%, 10=0.08%, 20=0.10%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=3.13%, sys=7.64%, ctx=72362, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=9243503/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=36107MB, aggrb=616161KB/s, minb=616161KB/s, maxb=616161KB/s, mint=60007msec, maxt=60007msec

Disk stats (read/write):
  vda: ios=175/658814, merge=0/8609, ticks=1444/6946772, in_queue=6949828, util=99.24%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29134: Mon Dec 16 01:45:35 2019
  read : io=8421.3MB, bw=143720KB/s, iops=35929, runt= 60001msec
    clat (usec): min=131, max=5964, avg=218.95, stdev=56.93
     lat (usec): min=132, max=5964, avg=219.50, stdev=56.95
    clat percentiles (usec):
     |  1.00th=[  155],  5.00th=[  165], 10.00th=[  173], 20.00th=[  185],
     | 30.00th=[  195], 40.00th=[  203], 50.00th=[  211], 60.00th=[  221],
     | 70.00th=[  233], 80.00th=[  247], 90.00th=[  270], 95.00th=[  294],
     | 99.00th=[  350], 99.50th=[  378], 99.90th=[  596], 99.95th=[  948],
     | 99.99th=[ 2128]
    bw (KB  /s): min=14576, max=19592, per=12.50%, avg=17966.93, stdev=957.49
    lat (usec) : 250=81.46%, 500=18.40%, 750=0.07%, 1000=0.02%
    lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%
  cpu          : usr=1.80%, sys=5.00%, ctx=2158860, majf=0, minf=80
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2155832/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=8421.3MB, aggrb=143719KB/s, minb=143719KB/s, maxb=143719KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=2154518/8, merge=0/5, ticks=426160/0, in_queue=425888, util=99.86%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29160: Mon Dec 16 01:47:35 2019
  write: io=2524.6MB, bw=43084KB/s, iops=10770, runt= 60003msec
    clat (usec): min=59, max=66717, avg=738.95, stdev=4867.38
     lat (usec): min=59, max=66717, avg=739.53, stdev=4867.39
    clat percentiles (usec):
     |  1.00th=[   65],  5.00th=[   67], 10.00th=[   69], 20.00th=[   72],
     | 30.00th=[   76], 40.00th=[   80], 50.00th=[   84], 60.00th=[   89],
     | 70.00th=[   94], 80.00th=[  102], 90.00th=[  117], 95.00th=[  137],
     | 99.00th=[37120], 99.50th=[39168], 99.90th=[41728], 99.95th=[42752],
     | 99.99th=[44288]
    bw (KB  /s): min= 3867, max= 6800, per=12.50%, avg=5386.89, stdev=586.21
    lat (usec) : 100=77.20%, 250=20.84%, 500=0.17%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=1.77%
    lat (msec) : 100=0.01%
  cpu          : usr=0.60%, sys=2.64%, ctx=1304520, majf=0, minf=58
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=646286/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2524.6MB, aggrb=43083KB/s, minb=43083KB/s, maxb=43083KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  vda: ios=169/645485, merge=0/1412, ticks=124/48104, in_queue=48064, util=79.91%


fio (США, Нью-Йорк)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=31460: Mon Dec 16 01:24:50 2019
  read : io=7908.6MB, bw=134971KB/s, iops=33742, runt= 60001msec
    clat (usec): min=2, max=12146, avg=231.04, stdev=161.17
     lat (usec): min=2, max=12147, avg=231.97, stdev=161.24
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    8], 10.00th=[  157], 20.00th=[  179],
     | 30.00th=[  193], 40.00th=[  207], 50.00th=[  221], 60.00th=[  237],
     | 70.00th=[  258], 80.00th=[  286], 90.00th=[  334], 95.00th=[  382],
     | 99.00th=[  524], 99.50th=[  620], 99.90th=[ 2384], 99.95th=[ 2960],
     | 99.99th=[ 4448]
    bw (KB  /s): min=13384, max=22712, per=12.51%, avg=16881.32, stdev=1712.99
    lat (usec) : 4=3.51%, 10=2.33%, 20=1.79%, 50=0.38%, 100=0.01%
    lat (usec) : 250=58.30%, 500=32.43%, 750=0.94%, 1000=0.08%
    lat (msec) : 2=0.05%, 4=0.16%, 10=0.01%, 20=0.01%
  cpu          : usr=2.97%, sys=7.95%, ctx=1868322, majf=2, minf=102
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2024594/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7908.6MB, aggrb=134970KB/s, minb=134970KB/s, maxb=134970KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1855108/9, merge=0/4, ticks=324384/0, in_queue=323904, util=98.33%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=31487: Mon Dec 16 01:26:56 2019
  write: io=23284MB, bw=397335KB/s, iops=99333, runt= 60007msec
    clat (usec): min=1, max=128077, avg=54.48, stdev=877.46
     lat (usec): min=2, max=128078, avg=59.22, stdev=936.71
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    7], 95.00th=[   10],
     | 99.00th=[   25], 99.50th=[  716], 99.90th=[15680], 99.95th=[19840],
     | 99.99th=[28288]
    bw (KB  /s): min=    1, max=63112, per=12.66%, avg=50293.42, stdev=7068.36
    lat (usec) : 2=0.01%, 4=61.17%, 10=33.34%, 20=4.16%, 50=0.63%
    lat (usec) : 100=0.10%, 250=0.04%, 500=0.03%, 750=0.03%, 1000=0.01%
    lat (msec) : 2=0.04%, 4=0.06%, 10=0.17%, 20=0.16%, 50=0.05%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=4.45%, sys=6.10%, ctx=95599, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5960722/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=23284MB, aggrb=397335KB/s, minb=397335KB/s, maxb=397335KB/s, mint=60007msec, maxt=60007msec

Disk stats (read/write):
  vda: ios=174/1557066, merge=0/10682, ticks=160/6476912, in_queue=6477100, util=94.01%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=31474: Mon Dec 16 01:25:56 2019
  read : io=8180.5MB, bw=139606KB/s, iops=34901, runt= 60003msec
    clat (usec): min=118, max=10289, avg=223.40, stdev=102.27
     lat (usec): min=119, max=10289, avg=224.29, stdev=102.35
    clat percentiles (usec):
     |  1.00th=[  145],  5.00th=[  157], 10.00th=[  165], 20.00th=[  177],
     | 30.00th=[  187], 40.00th=[  197], 50.00th=[  209], 60.00th=[  221],
     | 70.00th=[  237], 80.00th=[  258], 90.00th=[  294], 95.00th=[  330],
     | 99.00th=[  422], 99.50th=[  478], 99.90th=[  748], 99.95th=[ 1176],
     | 99.99th=[ 5344]
    bw (KB  /s): min=14192, max=19104, per=12.50%, avg=17452.85, stdev=775.78
    lat (usec) : 250=76.22%, 500=23.39%, 750=0.30%, 1000=0.04%
    lat (msec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.01%
  cpu          : usr=3.12%, sys=6.80%, ctx=2099536, majf=0, minf=65
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2094191/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=8180.5MB, aggrb=139605KB/s, minb=139605KB/s, maxb=139605KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  vda: ios=2090729/8, merge=0/5, ticks=366884/0, in_queue=366540, util=99.80%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=31500: Mon Dec 16 01:27:57 2019
  write: io=2163.8MB, bw=36926KB/s, iops=9231, runt= 60004msec
    clat (usec): min=66, max=48157, avg=860.52, stdev=4642.86
     lat (usec): min=66, max=48158, avg=861.51, stdev=4642.89
    clat percentiles (usec):
     |  1.00th=[   72],  5.00th=[   75], 10.00th=[   77], 20.00th=[   82],
     | 30.00th=[   87], 40.00th=[   92], 50.00th=[   97], 60.00th=[  103],
     | 70.00th=[  110], 80.00th=[  119], 90.00th=[  141], 95.00th=[  173],
     | 99.00th=[30080], 99.50th=[31104], 99.90th=[33024], 99.95th=[34048],
     | 99.99th=[37120]
    bw (KB  /s): min= 3373, max= 5824, per=12.51%, avg=4618.34, stdev=393.71
    lat (usec) : 100=53.98%, 250=43.09%, 500=0.28%, 750=0.02%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=2.61%
  cpu          : usr=0.93%, sys=3.20%, ctx=1122973, majf=0, minf=73
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=553923/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2163.8MB, aggrb=36925KB/s, minb=36925KB/s, maxb=36925KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  vda: ios=171/553407, merge=0/1598, ticks=188/44808, in_queue=44824, util=74.39%


fio (США, Сиэтл)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30897: Mon Dec 16 01:48:04 2019
  read : io=8540.3MB, bw=145752KB/s, iops=36437, runt= 60001msec
    clat (usec): min=1, max=15745, avg=214.05, stdev=185.39
     lat (usec): min=2, max=15745, avg=214.92, stdev=185.76
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    8], 10.00th=[  153], 20.00th=[  175],
     | 30.00th=[  187], 40.00th=[  195], 50.00th=[  205], 60.00th=[  217],
     | 70.00th=[  231], 80.00th=[  249], 90.00th=[  286], 95.00th=[  334],
     | 99.00th=[  510], 99.50th=[ 1004], 99.90th=[ 2544], 99.95th=[ 2832],
     | 99.99th=[ 4256]
    bw (KB  /s): min=13048, max=24792, per=12.49%, avg=18199.83, stdev=1860.18
    lat (usec) : 2=0.01%, 4=2.96%, 10=3.83%, 20=2.03%, 50=0.28%
    lat (usec) : 100=0.01%, 250=71.01%, 500=18.85%, 750=0.43%, 1000=0.11%
    lat (msec) : 2=0.23%, 4=0.26%, 10=0.01%, 20=0.01%
  cpu          : usr=3.03%, sys=7.68%, ctx=1995422, majf=2, minf=87
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2186315/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=8540.3MB, aggrb=145751KB/s, minb=145751KB/s, maxb=145751KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1985256/10, merge=0/3, ticks=350132/0, in_queue=350100, util=98.15%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30924: Mon Dec 16 01:50:10 2019
  write: io=25968MB, bw=443099KB/s, iops=110774, runt= 60012msec
    clat (usec): min=1, max=99789, avg=47.62, stdev=816.45
     lat (usec): min=2, max=108292, avg=52.13, stdev=875.35
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    4], 90.00th=[    5], 95.00th=[    7],
     | 99.00th=[   16], 99.50th=[  159], 99.90th=[14528], 99.95th=[18816],
     | 99.99th=[27520]
    bw (KB  /s): min=    1, max=66464, per=12.65%, avg=56037.47, stdev=5979.41
    lat (usec) : 2=0.01%, 4=71.13%, 10=25.78%, 20=2.32%, 50=0.18%
    lat (usec) : 100=0.07%, 250=0.03%, 500=0.02%, 750=0.03%, 1000=0.01%
    lat (msec) : 2=0.03%, 4=0.06%, 10=0.15%, 20=0.15%, 50=0.04%
    lat (msec) : 100=0.01%
  cpu          : usr=4.66%, sys=5.89%, ctx=94569, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=6647820/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=25968MB, aggrb=443099KB/s, minb=443099KB/s, maxb=443099KB/s, mint=60012msec, maxt=60012msec

Disk stats (read/write):
  vda: ios=169/1692956, merge=0/10324, ticks=260/6620580, in_queue=6621932, util=96.39%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30911: Mon Dec 16 01:49:09 2019
  read : io=8739.6MB, bw=149150KB/s, iops=37287, runt= 60002msec
    clat (usec): min=121, max=13671, avg=209.06, stdev=70.91
     lat (usec): min=122, max=13672, avg=209.89, stdev=70.97
    clat percentiles (usec):
     |  1.00th=[  147],  5.00th=[  159], 10.00th=[  167], 20.00th=[  177],
     | 30.00th=[  185], 40.00th=[  191], 50.00th=[  199], 60.00th=[  209],
     | 70.00th=[  219], 80.00th=[  235], 90.00th=[  262], 95.00th=[  290],
     | 99.00th=[  362], 99.50th=[  398], 99.90th=[  524], 99.95th=[  700],
     | 99.99th=[ 2192]
    bw (KB  /s): min=16304, max=19560, per=12.50%, avg=18649.24, stdev=461.35
    lat (usec) : 250=86.74%, 500=13.13%, 750=0.08%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=3.19%, sys=6.68%, ctx=2243199, majf=0, minf=80
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2237318/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=8739.6MB, aggrb=149149KB/s, minb=149149KB/s, maxb=149149KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  vda: ios=2229705/8, merge=0/5, ticks=387068/0, in_queue=387048, util=99.89%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30937: Mon Dec 16 01:51:10 2019
  write: io=2329.5MB, bw=39753KB/s, iops=9938, runt= 60006msec
    clat (usec): min=67, max=47499, avg=799.36, stdev=4594.63
     lat (usec): min=68, max=47500, avg=800.27, stdev=4594.65
    clat percentiles (usec):
     |  1.00th=[   74],  5.00th=[   76], 10.00th=[   77], 20.00th=[   80],
     | 30.00th=[   84], 40.00th=[   88], 50.00th=[   92], 60.00th=[   96],
     | 70.00th=[  101], 80.00th=[  109], 90.00th=[  121], 95.00th=[  137],
     | 99.00th=[31360], 99.50th=[32640], 99.90th=[34560], 99.95th=[35584],
     | 99.99th=[38144]
    bw (KB  /s): min= 3671, max= 5968, per=12.51%, avg=4972.47, stdev=355.85
    lat (usec) : 100=66.85%, 250=30.72%, 500=0.10%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=2.31%
  cpu          : usr=1.01%, sys=2.93%, ctx=1207149, majf=0, minf=73
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=596349/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2329.5MB, aggrb=39752KB/s, minb=39752KB/s, maxb=39752KB/s, mint=60006msec, maxt=60006msec

Disk stats (read/write):
  vda: ios=173/595676, merge=0/1579, ticks=128/45208, in_queue=45220, util=75.16%


fio (США, Сан-Хосе)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=22097: Mon Dec 16 03:10:03 2019
  read : io=8602.3MB, bw=146810KB/s, iops=36702, runt= 60001msec
    clat (usec): min=1, max=37853, avg=214.28, stdev=166.22
     lat (usec): min=1, max=37853, avg=214.82, stdev=166.22
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[  157], 20.00th=[  181],
     | 30.00th=[  193], 40.00th=[  203], 50.00th=[  213], 60.00th=[  223],
     | 70.00th=[  237], 80.00th=[  255], 90.00th=[  286], 95.00th=[  318],
     | 99.00th=[  442], 99.50th=[  676], 99.90th=[ 2352], 99.95th=[ 2768],
     | 99.99th=[ 4048]
    bw (KB  /s): min=12072, max=25480, per=12.50%, avg=18347.99, stdev=2080.92
    lat (usec) : 2=0.21%, 4=7.02%, 10=1.23%, 20=0.58%, 50=0.04%
    lat (usec) : 100=0.01%, 250=68.73%, 500=21.43%, 750=0.29%, 1000=0.11%
    lat (msec) : 2=0.17%, 4=0.17%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.79%, sys=5.69%, ctx=2009851, majf=2, minf=95
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2202186/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=8602.3MB, aggrb=146809KB/s, minb=146809KB/s, maxb=146809KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  vda: ios=1997992/9, merge=0/4, ticks=411600/0, in_queue=411320, util=98.32%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=22123: Mon Dec 16 03:12:12 2019
  write: io=35829MB, bw=610865KB/s, iops=152716, runt= 60061msec
    clat (usec): min=1, max=192241, avg=38.25, stdev=985.47
     lat (usec): min=1, max=270211, avg=40.80, stdev=1029.68
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    3], 90.00th=[    3], 95.00th=[    4],
     | 99.00th=[   10], 99.50th=[   18], 99.90th=[13376], 99.95th=[19584],
     | 99.99th=[35584]
    bw (KB  /s): min=    1, max=97767, per=12.65%, avg=77295.31, stdev=10675.70
    lat (usec) : 2=0.22%, 4=92.18%, 10=6.54%, 20=0.68%, 50=0.08%
    lat (usec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.02%, 10=0.07%, 20=0.10%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=2.80%, sys=7.39%, ctx=68251, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=9172295/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=35829MB, aggrb=610865KB/s, minb=610865KB/s, maxb=610865KB/s, mint=60061msec, maxt=60061msec

Disk stats (read/write):
  vda: ios=172/642212, merge=0/8314, ticks=1048/6914268, in_queue=6915840, util=99.12%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=22110: Mon Dec 16 03:11:12 2019
  read : io=8145.5MB, bw=139011KB/s, iops=34752, runt= 60002msec
    clat (usec): min=131, max=6165, avg=226.49, stdev=78.22
     lat (usec): min=132, max=6166, avg=227.00, stdev=78.23
    clat percentiles (usec):
     |  1.00th=[  155],  5.00th=[  167], 10.00th=[  177], 20.00th=[  189],
     | 30.00th=[  197], 40.00th=[  207], 50.00th=[  215], 60.00th=[  227],
     | 70.00th=[  239], 80.00th=[  255], 90.00th=[  282], 95.00th=[  310],
     | 99.00th=[  394], 99.50th=[  454], 99.90th=[ 1160], 99.95th=[ 1624],
     | 99.99th=[ 3152]
    bw (KB  /s): min=14064, max=18608, per=12.50%, avg=17378.33, stdev=844.92
    lat (usec) : 250=77.18%, 500=22.45%, 750=0.20%, 1000=0.05%
    lat (msec) : 2=0.10%, 4=0.02%, 10=0.01%
  cpu          : usr=1.73%, sys=4.97%, ctx=2088154, majf=0, minf=72
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2085239/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=8145.5MB, aggrb=139011KB/s, minb=139011KB/s, maxb=139011KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  vda: ios=2080813/8, merge=0/5, ticks=428548/0, in_queue=428396, util=99.87%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=22136: Mon Dec 16 03:13:12 2019
  write: io=2439.5MB, bw=41632KB/s, iops=10407, runt= 60003msec
    clat (usec): min=60, max=52245, avg=764.84, stdev=4993.14
     lat (usec): min=61, max=52245, avg=765.41, stdev=4993.15
    clat percentiles (usec):
     |  1.00th=[   66],  5.00th=[   68], 10.00th=[   70], 20.00th=[   74],
     | 30.00th=[   78], 40.00th=[   82], 50.00th=[   87], 60.00th=[   91],
     | 70.00th=[   97], 80.00th=[  105], 90.00th=[  123], 95.00th=[  149],
     | 99.00th=[37632], 99.50th=[40192], 99.90th=[42752], 99.95th=[43264],
     | 99.99th=[45312]
    bw (KB  /s): min= 3512, max= 6645, per=12.50%, avg=5204.98, stdev=587.61
    lat (usec) : 100=74.30%, 250=23.54%, 500=0.30%, 750=0.02%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=1.80%
    lat (msec) : 100=0.01%
  cpu          : usr=0.57%, sys=2.59%, ctx=1260736, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=624507/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2439.5MB, aggrb=41631KB/s, minb=41631KB/s, maxb=41631KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  vda: ios=174/623749, merge=0/1473, ticks=232/48604, in_queue=48728, util=80.83%


Speedtest (Австралия, Сидней)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (139.180.167.158)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Exetel (Brisbane) [0.59 km]: 15.529 ms
Testing download speed................................................................................
Download: 563.62 Mbit/s
Testing upload speed................................................................................................
Upload: 443.93 Mbit/s


Speedtest (Канада, Торонто)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (155.138.134.23)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by FibreStream (Toronto, ON) [3.45 km]: 1.132 ms
Testing download speed................................................................................
Download: 5165.30 Mbit/s
Testing upload speed................................................................................................
Upload: 3611.44 Mbit/s


Speedtest (Германия, Франкфурт)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (136.244.81.190)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Vodafone DE (Frankfurt) [2.33 km]: 2.37 ms
Testing download speed................................................................................
Download: 612.19 Mbit/s
Testing upload speed................................................................................................
Upload: 811.55 Mbit/s


Speedtest (Франция, Париж)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (199.247.12.8)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Naitways (Paris) [2.04 km]: 3.818 ms
Testing download speed................................................................................
Download: 1079.52 Mbit/s
Testing upload speed................................................................................................
Upload: 897.98 Mbit/s


Speedtest (Япония, Токио)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (45.77.22.120)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by GLBB Japan (Tokyo) [12.49 km]: 1.346 ms
Testing download speed................................................................................
Download: 3699.98 Mbit/s
Testing upload speed................................................................................................
Upload: 1897.78 Mbit/s


Speedtest (Нидерланды, Амстердам)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (209.250.247.182)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by YISP B.V. (Haarlem) [1.45 km]: 1.08 ms
Testing download speed................................................................................
Download: 2836.05 Mbit/s
Testing upload speed................................................................................................
Upload: 2417.74 Mbit/s


Speedtest (Сингапур)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (66.42.49.136)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NewMedia Express (Singapore) [6.13 km]: 15.508 ms
Testing download speed................................................................................
Download: 236.03 Mbit/s
Testing upload speed................................................................................................
Upload: 161.70 Mbit/s


Speedtest (Великобритания, Лондон)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (95.179.238.22)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Redraw Internet (London) [2.97 km]: 1.057 ms
Testing download speed................................................................................
Download: 714.62 Mbit/s
Testing upload speed................................................................................................
Upload: 759.21 Mbit/s


Speedtest (США, Атланта)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (155.138.200.236)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Whitesky Communications LLC (Atlanta, GA) [6.39 km]: 1.689 ms
Testing download speed................................................................................
Download: 4354.27 Mbit/s
Testing upload speed................................................................................................
Upload: 3473.80 Mbit/s


Speedtest (США, Чикаго)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (207.148.12.189)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Windstream (Chicago, IL) [31.77 km]: 8.795 ms
Testing download speed................................................................................
Download: 1454.70 Mbit/s
Testing upload speed................................................................................................
Upload: 1008.58 Mbit/s


Speedtest (США, Даллас)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (144.202.72.219)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Nitel (Dallas, TX) [1.45 km]: 1.099 ms
Testing download speed................................................................................
Download: 460.90 Mbit/s
Testing upload speed................................................................................................
Upload: 302.76 Mbit/s


Speedtest (США, Лос-Анджелес)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (45.76.68.204)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Bel Air Internet LLC (Los Angeles, CA) [2.73 km]: 1.023 ms
Testing download speed................................................................................
Download: 3429.38 Mbit/s
Testing upload speed................................................................................................
Upload: 2450.90 Mbit/s


Speedtest (США, Майами)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (140.82.28.193)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by TIGO (Miami, FL) [2.79 km]: 0.949 ms
Testing download speed................................................................................
Download: 938.03 Mbit/s
Testing upload speed................................................................................................
Upload: 835.08 Mbit/s


Speedtest (США, Нью-Йорк)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (45.77.97.253)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Vultr (Matawan, NJ) [24.71 km]: 1.114 ms
Testing download speed................................................................................
Download: 3753.04 Mbit/s
Testing upload speed................................................................................................
Upload: 2846.02 Mbit/s


Speedtest (США, Сиэтл)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (149.28.15.136)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Wowrack (Seattle, WA) [1.31 km]: 10.518 ms
Testing download speed................................................................................
Download: 4249.55 Mbit/s
Testing upload speed................................................................................................
Upload: 2548.90 Mbit/s


Speedtest (США, Сан-Хосе)
Retrieving speedtest.net configuration...
Testing from Choopa, LLC (45.77.2.227)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Razzolink Inc (San Jose, CA) [0.45 km]: 1.203 ms
Testing download speed................................................................................
Download: 4801.60 Mbit/s
Testing upload speed................................................................................................
Upload: 3428.94 Mbit/s



Виртуальный сервер от Linode:




  • тариф Nanode (25 GB SSD / 1x vCPU / 1 GB RAM / 1000 Mbit/s за $5 в месяц);
  • дата-центры в Австралии, Великобритании, Германии, Индии, Канаде, Сингапуре, США, Японии.

Тесты:


Unixbench (Австралия, Сидней)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: localhost: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3 (2019-09-02)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: AMD EPYC 7601 32-Core Processor (4400.0 bogomips)
          x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   02:55:11 up 2 min,  1 user,  load average: 0.66, 0.22, 0.08; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 02:55:11 - 03:23:14
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       30133063.3 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4446.8 MWIPS (9.7 s, 7 samples)
Execl Throughput                               5037.1 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        840751.9 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          232585.4 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       2355072.1 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1567758.3 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 218719.7 lps   (10.0 s, 7 samples)
Process Creation                              15426.4 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   9564.1 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   1254.2 lpm   (60.0 s, 2 samples)
System Call Overhead                        2412004.0 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   30133063.3   2582.1
Double-Precision Whetstone                       55.0       4446.8    808.5
Execl Throughput                                 43.0       5037.1   1171.4
File Copy 1024 bufsize 2000 maxblocks          3960.0     840751.9   2123.1
File Copy 256 bufsize 500 maxblocks            1655.0     232585.4   1405.4
File Copy 4096 bufsize 8000 maxblocks          5800.0    2355072.1   4060.5
Pipe Throughput                               12440.0    1567758.3   1260.3
Pipe-based Context Switching                   4000.0     218719.7    546.8
Process Creation                                126.0      15426.4   1224.3
Shell Scripts (1 concurrent)                     42.4       9564.1   2255.7
Shell Scripts (8 concurrent)                      6.0       1254.2   2090.3
System Call Overhead                          15000.0    2412004.0   1608.0
                                                                   ========
System Benchmarks Index Score                                        1548.1


Unixbench (Канада, Торонто)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: localhost: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3 (2019-09-02)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: AMD EPYC 7501 32-Core Processor (4000.0 bogomips)
          x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   23:43:33 up 2 min,  1 user,  load average: 0.67, 0.23, 0.08; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Fri Dec 13 2019 23:43:33 - 00:11:37
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       28218808.4 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4106.8 MWIPS (9.8 s, 7 samples)
Execl Throughput                               4001.1 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        776704.7 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          211047.3 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1886947.8 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1465079.0 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 204475.4 lps   (10.0 s, 7 samples)
Process Creation                              13209.9 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7985.9 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   1067.9 lpm   (60.0 s, 2 samples)
System Call Overhead                        2245138.6 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   28218808.4   2418.1
Double-Precision Whetstone                       55.0       4106.8    746.7
Execl Throughput                                 43.0       4001.1    930.5
File Copy 1024 bufsize 2000 maxblocks          3960.0     776704.7   1961.4
File Copy 256 bufsize 500 maxblocks            1655.0     211047.3   1275.2
File Copy 4096 bufsize 8000 maxblocks          5800.0    1886947.8   3253.4
Pipe Throughput                               12440.0    1465079.0   1177.7
Pipe-based Context Switching                   4000.0     204475.4    511.2
Process Creation                                126.0      13209.9   1048.4
Shell Scripts (1 concurrent)                     42.4       7985.9   1883.5
Shell Scripts (8 concurrent)                      6.0       1067.9   1779.8
System Call Overhead                          15000.0    2245138.6   1496.8
                                                                   ========
System Benchmarks Index Score                                        1368.9


Unixbench (Германия, Франкфурт)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: localhost: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3 (2019-09-02)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: AMD EPYC 7501 32-Core Processor (4000.0 bogomips)
          x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   19:05:00 up 4 min,  1 user,  load average: 0.56, 0.27, 0.12; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Sun Dec 15 2019 19:05:00 - 19:33:04
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       28555684.8 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4214.4 MWIPS (9.7 s, 7 samples)
Execl Throughput                               4731.7 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        799611.6 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          218261.4 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1772488.0 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1458871.8 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 205092.6 lps   (10.0 s, 7 samples)
Process Creation                              14474.3 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   8362.5 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   1126.3 lpm   (60.0 s, 2 samples)
System Call Overhead                        2170470.6 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   28555684.8   2446.9
Double-Precision Whetstone                       55.0       4214.4    766.3
Execl Throughput                                 43.0       4731.7   1100.4
File Copy 1024 bufsize 2000 maxblocks          3960.0     799611.6   2019.2
File Copy 256 bufsize 500 maxblocks            1655.0     218261.4   1318.8
File Copy 4096 bufsize 8000 maxblocks          5800.0    1772488.0   3056.0
Pipe Throughput                               12440.0    1458871.8   1172.7
Pipe-based Context Switching                   4000.0     205092.6    512.7
Process Creation                                126.0      14474.3   1148.8
Shell Scripts (1 concurrent)                     42.4       8362.5   1972.3
Shell Scripts (8 concurrent)                      6.0       1126.3   1877.2
System Call Overhead                          15000.0    2170470.6   1447.0
                                                                   ========
System Benchmarks Index Score                                        1410.7


Unixbench (Индия, Мумбаи)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: localhost: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3 (2019-09-02)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: AMD EPYC 7501 32-Core Processor (4000.0 bogomips)
          x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   02:53:15 up 2 min,  1 user,  load average: 0.59, 0.23, 0.09; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 02:53:15 - 03:21:18
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       28681152.2 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4207.2 MWIPS (9.8 s, 7 samples)
Execl Throughput                               4827.1 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        798915.0 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          218718.5 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       2211908.2 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1480545.2 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 205294.0 lps   (10.0 s, 7 samples)
Process Creation                              14957.8 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   8948.3 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   1173.7 lpm   (60.0 s, 2 samples)
System Call Overhead                        2297535.6 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   28681152.2   2457.7
Double-Precision Whetstone                       55.0       4207.2    764.9
Execl Throughput                                 43.0       4827.1   1122.6
File Copy 1024 bufsize 2000 maxblocks          3960.0     798915.0   2017.5
File Copy 256 bufsize 500 maxblocks            1655.0     218718.5   1321.6
File Copy 4096 bufsize 8000 maxblocks          5800.0    2211908.2   3813.6
Pipe Throughput                               12440.0    1480545.2   1190.1
Pipe-based Context Switching                   4000.0     205294.0    513.2
Process Creation                                126.0      14957.8   1187.1
Shell Scripts (1 concurrent)                     42.4       8948.3   2110.5
Shell Scripts (8 concurrent)                      6.0       1173.7   1956.1
System Call Overhead                          15000.0    2297535.6   1531.7
                                                                   ========
System Benchmarks Index Score                                        1465.8


Unixbench (Япония, Токио)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: localhost: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3 (2019-09-02)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz (4600.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   02:50:20 up 2 min,  1 user,  load average: 0.80, 0.28, 0.10; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 02:50:20 - 03:18:26
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       25781663.0 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4162.8 MWIPS (9.9 s, 7 samples)
Execl Throughput                               3905.5 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        436438.9 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          117964.6 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1190983.5 KBps  (30.0 s, 2 samples)
Pipe Throughput                              600292.7 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 111777.7 lps   (10.0 s, 7 samples)
Process Creation                               9587.3 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   6991.2 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    905.6 lpm   (60.0 s, 2 samples)
System Call Overhead                         341762.2 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   25781663.0   2209.2
Double-Precision Whetstone                       55.0       4162.8    756.9
Execl Throughput                                 43.0       3905.5    908.3
File Copy 1024 bufsize 2000 maxblocks          3960.0     436438.9   1102.1
File Copy 256 bufsize 500 maxblocks            1655.0     117964.6    712.8
File Copy 4096 bufsize 8000 maxblocks          5800.0    1190983.5   2053.4
Pipe Throughput                               12440.0     600292.7    482.6
Pipe-based Context Switching                   4000.0     111777.7    279.4
Process Creation                                126.0       9587.3    760.9
Shell Scripts (1 concurrent)                     42.4       6991.2   1648.9
Shell Scripts (8 concurrent)                      6.0        905.6   1509.3
System Call Overhead                          15000.0     341762.2    227.8
                                                                   ========
System Benchmarks Index Score                                         850.1


Unixbench (Сингапур)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: localhost: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3 (2019-09-02)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz (4600.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   00:59:58 up 3 min,  1 user,  load average: 0.68, 0.25, 0.09; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Mon Dec 16 2019 00:59:58 - 01:28:03
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       25946607.9 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4166.4 MWIPS (9.9 s, 7 samples)
Execl Throughput                               3876.2 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        449371.9 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          118226.6 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1265344.6 KBps  (30.0 s, 2 samples)
Pipe Throughput                              603138.5 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 112800.7 lps   (10.0 s, 7 samples)
Process Creation                               9763.9 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   6993.3 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    905.4 lpm   (60.0 s, 2 samples)
System Call Overhead                         341957.3 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   25946607.9   2223.4
Double-Precision Whetstone                       55.0       4166.4    757.5
Execl Throughput                                 43.0       3876.2    901.5
File Copy 1024 bufsize 2000 maxblocks          3960.0     449371.9   1134.8
File Copy 256 bufsize 500 maxblocks            1655.0     118226.6    714.4
File Copy 4096 bufsize 8000 maxblocks          5800.0    1265344.6   2181.6
Pipe Throughput                               12440.0     603138.5    484.8
Pipe-based Context Switching                   4000.0     112800.7    282.0
Process Creation                                126.0       9763.9    774.9
Shell Scripts (1 concurrent)                     42.4       6993.3   1649.4
Shell Scripts (8 concurrent)                      6.0        905.4   1508.9
System Call Overhead                          15000.0     341957.3    228.0
                                                                   ========
System Benchmarks Index Score                                         859.0


Unixbench (Великобритания, Лондон)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: localhost: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3 (2019-09-02)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (5000.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   15:58:44 up 2 min,  1 user,  load average: 0.81, 0.32, 0.12; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Sun Dec 15 2019 15:58:44 - 16:26:52
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       27229246.0 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3624.0 MWIPS (9.7 s, 7 samples)
Execl Throughput                               3869.2 lps   (29.6 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        463380.7 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          125234.3 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1075247.3 KBps  (30.0 s, 2 samples)
Pipe Throughput                              650363.8 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 119004.4 lps   (10.0 s, 7 samples)
Process Creation                               9502.3 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7004.4 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    881.0 lpm   (60.0 s, 2 samples)
System Call Overhead                         356503.7 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   27229246.0   2333.3
Double-Precision Whetstone                       55.0       3624.0    658.9
Execl Throughput                                 43.0       3869.2    899.8
File Copy 1024 bufsize 2000 maxblocks          3960.0     463380.7   1170.2
File Copy 256 bufsize 500 maxblocks            1655.0     125234.3    756.7
File Copy 4096 bufsize 8000 maxblocks          5800.0    1075247.3   1853.9
Pipe Throughput                               12440.0     650363.8    522.8
Pipe-based Context Switching                   4000.0     119004.4    297.5
Process Creation                                126.0       9502.3    754.2
Shell Scripts (1 concurrent)                     42.4       7004.4   1652.0
Shell Scripts (8 concurrent)                      6.0        881.0   1468.3
System Call Overhead                          15000.0     356503.7    237.7
                                                                   ========
System Benchmarks Index Score                                         855.4


Unixbench (США, Атланта)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: localhost: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3 (2019-09-02)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: AMD EPYC 7501 32-Core Processor (4000.0 bogomips)
          x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   07:44:55 up 3 min,  1 user,  load average: 0.86, 0.39, 0.15; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Thu Dec 05 2019 07:44:55 - 08:13:01
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       27076602.3 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4059.3 MWIPS (9.8 s, 7 samples)
Execl Throughput                               3779.0 lps   (29.9 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        648772.3 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          182118.5 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1695046.2 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1411027.5 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 186657.9 lps   (10.0 s, 7 samples)
Process Creation                              10081.5 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7214.0 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    909.5 lpm   (60.0 s, 2 samples)
System Call Overhead                        2183324.7 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   27076602.3   2320.2
Double-Precision Whetstone                       55.0       4059.3    738.1
Execl Throughput                                 43.0       3779.0    878.8
File Copy 1024 bufsize 2000 maxblocks          3960.0     648772.3   1638.3
File Copy 256 bufsize 500 maxblocks            1655.0     182118.5   1100.4
File Copy 4096 bufsize 8000 maxblocks          5800.0    1695046.2   2922.5
Pipe Throughput                               12440.0    1411027.5   1134.3
Pipe-based Context Switching                   4000.0     186657.9    466.6
Process Creation                                126.0      10081.5    800.1
Shell Scripts (1 concurrent)                     42.4       7214.0   1701.4
Shell Scripts (8 concurrent)                      6.0        909.5   1515.9
System Call Overhead                          15000.0    2183324.7   1455.5
                                                                   ========
System Benchmarks Index Score                                        1235.1


Unixbench (США, Даллас)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: localhost: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3 (2019-09-02)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz (4600.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   14:18:56 up 4 min,  1 user,  load average: 0.72, 0.27, 0.10; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Wed Dec 11 2019 14:18:56 - 14:47:01
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       25639903.4 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4159.3 MWIPS (9.8 s, 7 samples)
Execl Throughput                               3906.9 lps   (29.9 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        450551.9 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          120930.2 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1241998.9 KBps  (30.0 s, 2 samples)
Pipe Throughput                              599891.8 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 117628.7 lps   (10.0 s, 7 samples)
Process Creation                               9526.7 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7065.9 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    909.3 lpm   (60.0 s, 2 samples)
System Call Overhead                         342067.6 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   25639903.4   2197.1
Double-Precision Whetstone                       55.0       4159.3    756.2
Execl Throughput                                 43.0       3906.9    908.6
File Copy 1024 bufsize 2000 maxblocks          3960.0     450551.9   1137.8
File Copy 256 bufsize 500 maxblocks            1655.0     120930.2    730.7
File Copy 4096 bufsize 8000 maxblocks          5800.0    1241998.9   2141.4
Pipe Throughput                               12440.0     599891.8    482.2
Pipe-based Context Switching                   4000.0     117628.7    294.1
Process Creation                                126.0       9526.7    756.1
Shell Scripts (1 concurrent)                     42.4       7065.9   1666.5
Shell Scripts (8 concurrent)                      6.0        909.3   1515.4
System Call Overhead                          15000.0     342067.6    228.0
                                                                   ========
System Benchmarks Index Score                                         861.0


Unixbench (США, Фримонт)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: localhost: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3 (2019-09-02)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz (4600.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   18:18:58 up 3 min,  1 user,  load average: 0.64, 0.23, 0.09; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Wed Dec 11 2019 18:18:58 - 18:47:02
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       25968547.5 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4168.6 MWIPS (9.8 s, 7 samples)
Execl Throughput                               3912.3 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        467206.8 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          121416.0 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1441297.8 KBps  (30.0 s, 2 samples)
Pipe Throughput                              604580.9 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 117233.5 lps   (10.0 s, 7 samples)
Process Creation                               9613.5 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7118.7 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    936.4 lpm   (60.0 s, 2 samples)
System Call Overhead                         343482.8 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   25968547.5   2225.2
Double-Precision Whetstone                       55.0       4168.6    757.9
Execl Throughput                                 43.0       3912.3    909.8
File Copy 1024 bufsize 2000 maxblocks          3960.0     467206.8   1179.8
File Copy 256 bufsize 500 maxblocks            1655.0     121416.0    733.6
File Copy 4096 bufsize 8000 maxblocks          5800.0    1441297.8   2485.0
Pipe Throughput                               12440.0     604580.9    486.0
Pipe-based Context Switching                   4000.0     117233.5    293.1
Process Creation                                126.0       9613.5    763.0
Shell Scripts (1 concurrent)                     42.4       7118.7   1678.9
Shell Scripts (8 concurrent)                      6.0        936.4   1560.7
System Call Overhead                          15000.0     343482.8    229.0
                                                                   ========
System Benchmarks Index Score                                         879.8


Unixbench (США, Ньюарк)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: localhost: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3 (2019-09-02)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: AMD EPYC 7601 32-Core Processor (4400.0 bogomips)
          x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   12:37:25 up 4 min,  1 user,  load average: 0.59, 0.26, 0.10; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Thu Dec 12 2019 12:37:25 - 13:05:28
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       28849705.0 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4265.4 MWIPS (9.7 s, 7 samples)
Execl Throughput                               3989.7 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        764588.9 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          214230.9 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1826179.5 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1533597.7 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 202033.0 lps   (10.0 s, 7 samples)
Process Creation                              11268.5 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7998.9 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   1012.8 lpm   (60.0 s, 2 samples)
System Call Overhead                        2192009.9 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   28849705.0   2472.1
Double-Precision Whetstone                       55.0       4265.4    775.5
Execl Throughput                                 43.0       3989.7    927.8
File Copy 1024 bufsize 2000 maxblocks          3960.0     764588.9   1930.8
File Copy 256 bufsize 500 maxblocks            1655.0     214230.9   1294.4
File Copy 4096 bufsize 8000 maxblocks          5800.0    1826179.5   3148.6
Pipe Throughput                               12440.0    1533597.7   1232.8
Pipe-based Context Switching                   4000.0     202033.0    505.1
Process Creation                                126.0      11268.5    894.3
Shell Scripts (1 concurrent)                     42.4       7998.9   1886.5
Shell Scripts (8 concurrent)                      6.0       1012.8   1688.1
System Call Overhead                          15000.0    2192009.9   1461.3
                                                                   ========
System Benchmarks Index Score                                        1348.9


dd (Австралия, Сидней)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 1.84047 s, 1.2 GB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 1.70147 s, 1.3 GB/s


dd (Канада, Торонто)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.06427 s, 1.0 GB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.03386 s, 1.1 GB/s


dd (Германия, Франкфурт)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.23042 s, 963 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.02314 s, 1.1 GB/s


dd (Индия, Мумбаи)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 1.87866 s, 1.1 GB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 1.79076 s, 1.2 GB/s


dd (Япония, Токио)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.3769 s, 903 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.4095 s, 891 MB/s


dd (Сингапур)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.31406 s, 928 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.32339 s, 924 MB/s


dd (Великобритания, Лондон)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.40905 s, 891 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.72641 s, 788 MB/s


dd (США, Атланта)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.98208 s, 720 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 24.2335 s, 88.6 MB/s


dd (США, Даллас)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.42872 s, 884 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.36519 s, 638 MB/s


dd (США, Фримонт)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.50491 s, 857 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.53728 s, 846 MB/s


dd (США, Ньюарк)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.13085 s, 1.0 GB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.08892 s, 695 MB/s


fio (Австралия, Сидней)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=8604: Mon Dec 16 03:24:20 2019
  read : io=6755.1MB, bw=115299KB/s, iops=28824, runt= 60001msec
    clat (usec): min=0, max=100545, avg=274.82, stdev=569.86
     lat (usec): min=1, max=100545, avg=275.21, stdev=569.86
    clat percentiles (usec):
     |  1.00th=[    1],  5.00th=[    2], 10.00th=[  169], 20.00th=[  197],
     | 30.00th=[  211], 40.00th=[  223], 50.00th=[  237], 60.00th=[  255],
     | 70.00th=[  294], 80.00th=[  354], 90.00th=[  446], 95.00th=[  540],
     | 99.00th=[  740], 99.50th=[  804], 99.90th=[ 1944], 99.95th=[ 2352],
     | 99.99th=[13760]
    bw (KB  /s): min= 6472, max=21216, per=12.51%, avg=14418.69, stdev=2131.98
    lat (usec) : 2=2.23%, 4=5.07%, 10=0.30%, 20=0.16%, 50=0.03%
    lat (usec) : 100=0.02%, 250=49.90%, 500=35.60%, 750=5.76%, 1000=0.73%
    lat (msec) : 2=0.11%, 4=0.08%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.90%, sys=6.29%, ctx=1609714, majf=2, minf=104
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1729514/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=6755.1MB, aggrb=115299KB/s, minb=115299KB/s, maxb=115299KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1593629/10, merge=136/5, ticks=403736/0, in_queue=403412, util=99.03%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=8633: Mon Dec 16 03:26:23 2019
  write: io=37600MB, bw=641604KB/s, iops=160400, runt= 60009msec
    clat (usec): min=1, max=138933, avg=36.00, stdev=698.17
     lat (usec): min=1, max=138933, avg=38.40, stdev=740.72
    clat percentiles (usec):
     |  1.00th=[    1],  5.00th=[    1], 10.00th=[    1], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    2], 50.00th=[    2], 60.00th=[    2],
     | 70.00th=[    2], 80.00th=[    2], 90.00th=[    3], 95.00th=[    3],
     | 99.00th=[    4], 99.50th=[  113], 99.90th=[11712], 99.95th=[16064],
     | 99.99th=[27008]
    bw (KB  /s): min=    1, max=103354, per=12.62%, avg=80945.26, stdev=7323.10
    lat (usec) : 2=10.82%, 4=88.07%, 10=0.29%, 20=0.13%, 50=0.13%
    lat (usec) : 100=0.05%, 250=0.03%, 500=0.04%, 750=0.04%, 1000=0.03%
    lat (msec) : 2=0.04%, 4=0.05%, 10=0.15%, 20=0.10%, 50=0.03%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.46%, sys=7.39%, ctx=100753, majf=0, minf=67
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=9625502/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=37600MB, aggrb=641603KB/s, minb=641603KB/s, maxb=641603KB/s, mint=60009msec, maxt=60009msec

Disk stats (read/write):
  sda: ios=172/3092395, merge=31/1341, ticks=4/5409372, in_queue=5404936, util=70.18%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=8617: Mon Dec 16 03:25:23 2019
  read : io=6485.6MB, bw=110683KB/s, iops=27670, runt= 60002msec
    clat (usec): min=61, max=579406, avg=286.37, stdev=809.14
     lat (usec): min=61, max=579406, avg=286.76, stdev=809.14
    clat percentiles (usec):
     |  1.00th=[  155],  5.00th=[  175], 10.00th=[  185], 20.00th=[  201],
     | 30.00th=[  213], 40.00th=[  223], 50.00th=[  237], 60.00th=[  255],
     | 70.00th=[  294], 80.00th=[  350], 90.00th=[  438], 95.00th=[  524],
     | 99.00th=[  708], 99.50th=[  772], 99.90th=[  900], 99.95th=[  964],
     | 99.99th=[22144]
    bw (KB  /s): min= 2889, max=15576, per=12.51%, avg=13846.49, stdev=1486.54
    lat (usec) : 100=0.03%, 250=57.83%, 500=36.15%, 750=5.33%, 1000=0.63%
    lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=0.93%, sys=5.87%, ctx=1664165, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1660305/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=6485.6MB, aggrb=110683KB/s, minb=110683KB/s, maxb=110683KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  sda: ios=1659424/20, merge=317/15, ticks=421880/0, in_queue=421648, util=99.87%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=8646: Mon Dec 16 03:27:23 2019
  write: io=1930.2MB, bw=32955KB/s, iops=8238, runt= 60001msec
    clat (usec): min=66, max=66656, avg=968.12, stdev=6325.23
     lat (usec): min=67, max=66656, avg=968.51, stdev=6325.24
    clat percentiles (usec):
     |  1.00th=[   77],  5.00th=[   79], 10.00th=[   80], 20.00th=[   83],
     | 30.00th=[   87], 40.00th=[   90], 50.00th=[   92], 60.00th=[   94],
     | 70.00th=[   96], 80.00th=[  103], 90.00th=[  306], 95.00th=[  346],
     | 99.00th=[48384], 99.50th=[49920], 99.90th=[50944], 99.95th=[50944],
     | 99.99th=[53504]
    bw (KB  /s): min= 2736, max= 5424, per=12.51%, avg=4121.09, stdev=523.74
    lat (usec) : 100=76.13%, 250=10.66%, 500=11.42%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=1.36%
    lat (msec) : 100=0.42%
  cpu          : usr=0.30%, sys=2.13%, ctx=997805, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=494335/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=1930.2MB, aggrb=32955KB/s, minb=32955KB/s, maxb=32955KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=50/493789, merge=157/1755, ticks=40/49980, in_queue=49936, util=83.16%


fio (Канада, Торонто)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=15535: Sat Dec 14 00:12:44 2019
  read : io=7105.6MB, bw=121265KB/s, iops=30316, runt= 60001msec
    clat (usec): min=0, max=1003.9K, avg=260.89, stdev=940.50
     lat (usec): min=1, max=1003.9K, avg=261.29, stdev=940.50
    clat percentiles (usec):
     |  1.00th=[    1],  5.00th=[    2], 10.00th=[  161], 20.00th=[  189],
     | 30.00th=[  203], 40.00th=[  215], 50.00th=[  227], 60.00th=[  245],
     | 70.00th=[  278], 80.00th=[  326], 90.00th=[  406], 95.00th=[  482],
     | 99.00th=[  708], 99.50th=[  828], 99.90th=[ 2288], 99.95th=[ 2704],
     | 99.99th=[14400]
    bw (KB  /s): min= 3942, max=21760, per=12.53%, avg=15190.66, stdev=1907.96
    lat (usec) : 2=3.64%, 4=3.76%, 10=0.34%, 20=0.12%, 50=0.01%
    lat (usec) : 100=0.05%, 250=53.79%, 500=34.02%, 750=3.49%, 1000=0.50%
    lat (msec) : 2=0.12%, 4=0.14%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 2000=0.01%
  cpu          : usr=1.05%, sys=7.07%, ctx=1681777, majf=2, minf=98
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1819009/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7105.6MB, aggrb=121265KB/s, minb=121265KB/s, maxb=121265KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1671621/10, merge=144/5, ticks=385432/0, in_queue=385092, util=98.64%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=15561: Sat Dec 14 00:14:47 2019
  write: io=34626MB, bw=590888KB/s, iops=147722, runt= 60007msec
    clat (usec): min=1, max=110371, avg=38.56, stdev=740.93
     lat (usec): min=1, max=110371, avg=41.20, stdev=786.70
    clat percentiles (usec):
     |  1.00th=[    1],  5.00th=[    1], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    2], 50.00th=[    2], 60.00th=[    2],
     | 70.00th=[    2], 80.00th=[    2], 90.00th=[    2], 95.00th=[    3],
     | 99.00th=[    4], 99.50th=[  155], 99.90th=[12608], 99.95th=[17280],
     | 99.99th=[28032]
    bw (KB  /s): min=    1, max=93867, per=12.62%, avg=74551.39, stdev=7277.00
    lat (usec) : 2=7.80%, 4=91.03%, 10=0.34%, 20=0.07%, 50=0.08%
    lat (usec) : 100=0.12%, 250=0.08%, 500=0.03%, 750=0.03%, 1000=0.03%
    lat (msec) : 2=0.04%, 4=0.05%, 10=0.14%, 20=0.11%, 50=0.03%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.88%, sys=6.88%, ctx=91744, majf=0, minf=76
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=8864358/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=34626MB, aggrb=590888KB/s, minb=590888KB/s, maxb=590888KB/s, mint=60007msec, maxt=60007msec

Disk stats (read/write):
  sda: ios=172/2770808, merge=31/1461, ticks=4/5007316, in_queue=5002336, util=67.08%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=15548: Sat Dec 14 00:13:47 2019
  read : io=6997.4MB, bw=119417KB/s, iops=29854, runt= 60002msec
    clat (usec): min=51, max=1008.4K, avg=264.70, stdev=1211.12
     lat (usec): min=52, max=1008.4K, avg=265.16, stdev=1211.12
    clat percentiles (usec):
     |  1.00th=[   79],  5.00th=[  139], 10.00th=[  165], 20.00th=[  185],
     | 30.00th=[  199], 40.00th=[  211], 50.00th=[  223], 60.00th=[  241],
     | 70.00th=[  270], 80.00th=[  322], 90.00th=[  398], 95.00th=[  478],
     | 99.00th=[  716], 99.50th=[  788], 99.90th=[ 1032], 99.95th=[ 1320],
     | 99.99th=[ 9536]
    bw (KB  /s): min=  391, max=23312, per=12.54%, avg=14972.94, stdev=1899.15
    lat (usec) : 100=2.50%, 250=61.34%, 500=31.96%, 750=3.47%, 1000=0.61%
    lat (msec) : 2=0.09%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 2000=0.01%
  cpu          : usr=1.09%, sys=6.67%, ctx=1822378, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1791317/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=6997.4MB, aggrb=119417KB/s, minb=119417KB/s, maxb=119417KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  sda: ios=1787341/10, merge=136/5, ticks=397456/0, in_queue=397140, util=99.63%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=15574: Sat Dec 14 00:15:48 2019
  write: io=2134.7MB, bw=36430KB/s, iops=9107, runt= 60003msec
    clat (usec): min=64, max=143251, avg=875.05, stdev=5614.52
     lat (usec): min=65, max=143252, avg=875.49, stdev=5614.53
    clat percentiles (usec):
     |  1.00th=[   70],  5.00th=[   71], 10.00th=[   73], 20.00th=[   75],
     | 30.00th=[   77], 40.00th=[   80], 50.00th=[   82], 60.00th=[   86],
     | 70.00th=[   91], 80.00th=[  104], 90.00th=[  233], 95.00th=[  306],
     | 99.00th=[40192], 99.50th=[44288], 99.90th=[46848], 99.95th=[48384],
     | 99.99th=[61184]
    bw (KB  /s): min= 2576, max= 6080, per=12.51%, avg=4556.05, stdev=585.26
    lat (usec) : 100=78.06%, 250=13.06%, 500=6.91%, 750=0.05%, 1000=0.03%
    lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=1.84%
    lat (msec) : 100=0.03%, 250=0.01%
  cpu          : usr=0.40%, sys=2.49%, ctx=1103528, majf=0, minf=66
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=546471/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2134.7MB, aggrb=36429KB/s, minb=36429KB/s, maxb=36429KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  sda: ios=179/545918, merge=322/1528, ticks=408/47584, in_queue=47876, util=79.18%


fio (Германия, Франкфурт)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=5708: Sun Dec 15 19:34:11 2019
  read : io=7256.2MB, bw=123836KB/s, iops=30959, runt= 60001msec
    clat (usec): min=1, max=928516, avg=255.21, stdev=866.57
     lat (usec): min=1, max=928516, avg=255.62, stdev=866.58
    clat percentiles (usec):
     |  1.00th=[    1],  5.00th=[    2], 10.00th=[  145], 20.00th=[  191],
     | 30.00th=[  209], 40.00th=[  221], 50.00th=[  235], 60.00th=[  251],
     | 70.00th=[  274], 80.00th=[  310], 90.00th=[  366], 95.00th=[  426],
     | 99.00th=[  692], 99.50th=[  804], 99.90th=[ 2736], 99.95th=[ 3376],
     | 99.99th=[13376]
    bw (KB  /s): min= 3947, max=28312, per=12.53%, avg=15518.60, stdev=2109.24
    lat (usec) : 2=2.34%, 4=5.09%, 10=0.35%, 20=0.17%, 50=0.03%
    lat (usec) : 100=0.48%, 250=51.00%, 500=37.75%, 750=2.13%, 1000=0.36%
    lat (msec) : 2=0.09%, 4=0.19%, 10=0.02%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 1000=0.01%
  cpu          : usr=1.06%, sys=7.27%, ctx=1732810, majf=2, minf=96
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1857571/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7256.2MB, aggrb=123836KB/s, minb=123836KB/s, maxb=123836KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1707687/10, merge=136/5, ticks=380524/0, in_queue=380068, util=98.56%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=5737: Sun Dec 15 19:36:15 2019
  write: io=32623MB, bw=556695KB/s, iops=139173, runt= 60007msec
    clat (usec): min=1, max=84101, avg=41.66, stdev=765.39
     lat (usec): min=1, max=84101, avg=44.37, stdev=811.18
    clat percentiles (usec):
     |  1.00th=[    1],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    2], 50.00th=[    2], 60.00th=[    2],
     | 70.00th=[    2], 80.00th=[    2], 90.00th=[    3], 95.00th=[    3],
     | 99.00th=[    7], 99.50th=[  306], 99.90th=[12992], 99.95th=[17280],
     | 99.99th=[28288]
    bw (KB  /s): min=    1, max=95040, per=12.63%, avg=70283.71, stdev=7683.83
    lat (usec) : 2=4.00%, 4=94.50%, 10=0.53%, 20=0.11%, 50=0.19%
    lat (usec) : 100=0.10%, 250=0.05%, 500=0.03%, 750=0.04%, 1000=0.03%
    lat (msec) : 2=0.04%, 4=0.05%, 10=0.15%, 20=0.12%, 50=0.03%
    lat (msec) : 100=0.01%
  cpu          : usr=1.62%, sys=6.95%, ctx=93483, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=8351405/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=32623MB, aggrb=556695KB/s, minb=556695KB/s, maxb=556695KB/s, mint=60007msec, maxt=60007msec

Disk stats (read/write):
  sda: ios=172/2732266, merge=31/1355, ticks=144/5451696, in_queue=5446676, util=71.80%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=5721: Sun Dec 15 19:35:14 2019
  read : io=6805.7MB, bw=116146KB/s, iops=29036, runt= 60002msec
    clat (usec): min=56, max=1006.1K, avg=272.36, stdev=1070.90
     lat (usec): min=56, max=1006.1K, avg=272.80, stdev=1070.90
    clat percentiles (usec):
     |  1.00th=[  131],  5.00th=[  169], 10.00th=[  181], 20.00th=[  199],
     | 30.00th=[  213], 40.00th=[  225], 50.00th=[  237], 60.00th=[  253],
     | 70.00th=[  278], 80.00th=[  310], 90.00th=[  362], 95.00th=[  418],
     | 99.00th=[  612], 99.50th=[  812], 99.90th=[ 3152], 99.95th=[ 4960],
     | 99.99th=[22400]
    bw (KB  /s): min=  315, max=22760, per=12.53%, avg=14554.98, stdev=2178.59
    lat (usec) : 100=0.40%, 250=57.47%, 500=40.06%, 750=1.49%, 1000=0.25%
    lat (msec) : 2=0.16%, 4=0.11%, 10=0.05%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 750=0.01%, 2000=0.01%
  cpu          : usr=1.02%, sys=6.78%, ctx=1754222, majf=0, minf=66
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1742254/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=6805.7MB, aggrb=116146KB/s, minb=116146KB/s, maxb=116146KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  sda: ios=1739426/12, merge=316/13, ticks=389336/0, in_queue=389096, util=99.74%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=5750: Sun Dec 15 19:37:15 2019
  write: io=2106.5MB, bw=35947KB/s, iops=8986, runt= 60003msec
    clat (usec): min=66, max=79631, avg=887.06, stdev=5832.79
     lat (usec): min=66, max=79632, avg=887.50, stdev=5832.80
    clat percentiles (usec):
     |  1.00th=[   70],  5.00th=[   73], 10.00th=[   74], 20.00th=[   76],
     | 30.00th=[   80], 40.00th=[   84], 50.00th=[   88], 60.00th=[   93],
     | 70.00th=[   99], 80.00th=[  113], 90.00th=[  209], 95.00th=[  270],
     | 99.00th=[43776], 99.50th=[46336], 99.90th=[49920], 99.95th=[52992],
     | 99.99th=[61184]
    bw (KB  /s): min= 2848, max= 5945, per=12.52%, avg=4499.10, stdev=664.94
    lat (usec) : 100=70.00%, 250=23.23%, 500=4.81%, 750=0.09%, 1000=0.04%
    lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.01%, 50=1.67%
    lat (msec) : 100=0.10%
  cpu          : usr=0.38%, sys=2.28%, ctx=1088372, majf=0, minf=66
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=539238/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2106.5MB, aggrb=35947KB/s, minb=35947KB/s, maxb=35947KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  sda: ios=57/538803, merge=144/1594, ticks=28/47844, in_queue=47732, util=79.49%


fio (Индия, Мумбаи)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29999: Mon Dec 16 03:22:24 2019
  read : io=7680.4MB, bw=131076KB/s, iops=32769, runt= 60001msec
    clat (usec): min=1, max=1005.9K, avg=241.25, stdev=958.77
     lat (usec): min=1, max=1005.9K, avg=241.61, stdev=958.78
    clat percentiles (usec):
     |  1.00th=[    1],  5.00th=[    2], 10.00th=[  161], 20.00th=[  185],
     | 30.00th=[  199], 40.00th=[  209], 50.00th=[  219], 60.00th=[  233],
     | 70.00th=[  255], 80.00th=[  294], 90.00th=[  342], 95.00th=[  398],
     | 99.00th=[  636], 99.50th=[  716], 99.90th=[ 2352], 99.95th=[ 2896],
     | 99.99th=[12736]
    bw (KB  /s): min= 3690, max=22984, per=12.52%, avg=16407.99, stdev=1870.64
    lat (usec) : 2=3.82%, 4=3.66%, 10=0.35%, 20=0.09%, 50=0.01%
    lat (usec) : 100=0.02%, 250=59.86%, 500=30.20%, 750=1.58%, 1000=0.16%
    lat (msec) : 2=0.09%, 4=0.12%, 10=0.02%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 500=0.01%, 2000=0.01%
  cpu          : usr=1.05%, sys=6.95%, ctx=1815281, majf=2, minf=89
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1966177/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7680.4MB, aggrb=131076KB/s, minb=131076KB/s, maxb=131076KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1805586/33, merge=187/30, ticks=392240/4, in_queue=392088, util=98.48%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30118: Mon Dec 16 03:24:27 2019
  write: io=36526MB, bw=623330KB/s, iops=155832, runt= 60004msec
    clat (usec): min=1, max=119319, avg=36.80, stdev=722.41
     lat (usec): min=1, max=119319, avg=39.29, stdev=765.26
    clat percentiles (usec):
     |  1.00th=[    1],  5.00th=[    1], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    2], 50.00th=[    2], 60.00th=[    2],
     | 70.00th=[    2], 80.00th=[    2], 90.00th=[    3], 95.00th=[    3],
     | 99.00th=[    4], 99.50th=[   97], 99.90th=[11968], 99.95th=[16768],
     | 99.99th=[27776]
    bw (KB  /s): min=    1, max=97309, per=12.62%, avg=78634.24, stdev=7494.43
    lat (usec) : 2=6.84%, 4=92.00%, 10=0.33%, 20=0.07%, 50=0.15%
    lat (usec) : 100=0.12%, 250=0.04%, 500=0.03%, 750=0.04%, 1000=0.03%
    lat (msec) : 2=0.04%, 4=0.05%, 10=0.14%, 20=0.10%, 50=0.03%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.89%, sys=6.90%, ctx=94592, majf=0, minf=67
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=9350572/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=36526MB, aggrb=623329KB/s, minb=623329KB/s, maxb=623329KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  sda: ios=276/2956177, merge=354/1423, ticks=184/5082644, in_queue=5078200, util=67.28%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30060: Mon Dec 16 03:23:27 2019
  read : io=7109.1MB, bw=121340KB/s, iops=30335, runt= 60001msec
    clat (usec): min=54, max=1005.3K, avg=260.81, stdev=980.86
     lat (usec): min=54, max=1005.3K, avg=261.23, stdev=980.88
    clat percentiles (usec):
     |  1.00th=[  149],  5.00th=[  167], 10.00th=[  177], 20.00th=[  193],
     | 30.00th=[  203], 40.00th=[  215], 50.00th=[  225], 60.00th=[  241],
     | 70.00th=[  266], 80.00th=[  302], 90.00th=[  354], 95.00th=[  414],
     | 99.00th=[  644], 99.50th=[  716], 99.90th=[  948], 99.95th=[ 3376],
     | 99.99th=[12352]
    bw (KB  /s): min= 3389, max=23968, per=12.52%, avg=15196.38, stdev=1379.78
    lat (usec) : 100=0.09%, 250=64.27%, 500=33.03%, 750=2.27%, 1000=0.24%
    lat (msec) : 2=0.04%, 4=0.01%, 10=0.03%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 2000=0.01%
  cpu          : usr=1.10%, sys=6.31%, ctx=1824541, majf=0, minf=84
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1820134/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7109.1MB, aggrb=121340KB/s, minb=121340KB/s, maxb=121340KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1814323/36, merge=54/29, ticks=399948/4, in_queue=399628, util=99.33%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30175: Mon Dec 16 03:25:28 2019
  write: io=2144.3MB, bw=36593KB/s, iops=9148, runt= 60003msec
    clat (usec): min=66, max=76722, avg=871.26, stdev=5688.89
     lat (usec): min=66, max=76722, avg=871.72, stdev=5688.97
    clat percentiles (usec):
     |  1.00th=[   70],  5.00th=[   73], 10.00th=[   75], 20.00th=[   78],
     | 30.00th=[   80], 40.00th=[   82], 50.00th=[   85], 60.00th=[   88],
     | 70.00th=[   92], 80.00th=[  102], 90.00th=[  225], 95.00th=[  290],
     | 99.00th=[42752], 99.50th=[44800], 99.90th=[46848], 99.95th=[48896],
     | 99.99th=[61184]
    bw (KB  /s): min= 2913, max= 6144, per=12.50%, avg=4575.83, stdev=573.66
    lat (usec) : 100=78.69%, 250=13.10%, 500=6.35%, 750=0.02%, 1000=0.02%
    lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=1.76%
    lat (msec) : 100=0.03%
  cpu          : usr=0.37%, sys=2.38%, ctx=1108100, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=548927/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2144.3MB, aggrb=36593KB/s, minb=36593KB/s, maxb=36593KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  sda: ios=351/548346, merge=381/1573, ticks=220/47296, in_queue=47364, util=78.53%


fio (Япония, Токио)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=8994: Mon Dec 16 03:19:35 2019
  read : io=6104.3MB, bw=104172KB/s, iops=26042, runt= 60004msec
    clat (usec): min=2, max=175502, avg=301.62, stdev=709.96
     lat (usec): min=2, max=175503, avg=302.53, stdev=709.98
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[  199], 20.00th=[  215],
     | 30.00th=[  227], 40.00th=[  237], 50.00th=[  247], 60.00th=[  258],
     | 70.00th=[  278], 80.00th=[  310], 90.00th=[  382], 95.00th=[  516],
     | 99.00th=[ 1464], 99.50th=[ 2608], 99.90th=[ 5088], 99.95th=[ 7840],
     | 99.99th=[27776]
    bw (KB  /s): min= 4801, max=18768, per=12.52%, avg=13047.25, stdev=1979.51
    lat (usec) : 4=4.96%, 10=1.05%, 20=0.22%, 50=0.02%, 100=0.28%
    lat (usec) : 250=46.83%, 500=41.33%, 750=2.46%, 1000=1.19%
    lat (msec) : 2=0.93%, 4=0.54%, 10=0.16%, 20=0.02%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.92%, sys=7.58%, ctx=1471253, majf=0, minf=67
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1562682/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=6104.3MB, aggrb=104171KB/s, minb=104171KB/s, maxb=104171KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  sda: ios=1464787/22, merge=242/13, ticks=373888/12, in_queue=373712, util=98.37%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=9020: Mon Dec 16 03:21:39 2019
  write: io=21704MB, bw=370364KB/s, iops=92590, runt= 60007msec
    clat (usec): min=2, max=119437, avg=53.10, stdev=808.22
     lat (usec): min=3, max=119438, avg=59.46, stdev=877.89
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[  104], 99.50th=[ 2672], 99.90th=[13504], 99.95th=[18816],
     | 99.99th=[28032]
    bw (KB  /s): min=    1, max=59442, per=12.66%, avg=46888.23, stdev=3647.44
    lat (usec) : 4=46.23%, 10=52.01%, 20=0.43%, 50=0.10%, 100=0.18%
    lat (usec) : 250=0.22%, 500=0.11%, 750=0.04%, 1000=0.03%
    lat (msec) : 2=0.09%, 4=0.15%, 10=0.24%, 20=0.12%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=4.07%, sys=6.16%, ctx=115702, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5556102/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=21704MB, aggrb=370363KB/s, minb=370363KB/s, maxb=370363KB/s, mint=60007msec, maxt=60007msec

Disk stats (read/write):
  sda: ios=146/1732737, merge=57/1467, ticks=140/5125380, in_queue=5123836, util=67.83%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=9007: Mon Dec 16 03:20:39 2019
  read : io=5844.9MB, bw=99749KB/s, iops=24937, runt= 60002msec
    clat (usec): min=70, max=100509, avg=314.94, stdev=758.21
     lat (usec): min=70, max=100509, avg=315.82, stdev=758.24
    clat percentiles (usec):
     |  1.00th=[  139],  5.00th=[  199], 10.00th=[  205], 20.00th=[  217],
     | 30.00th=[  227], 40.00th=[  233], 50.00th=[  241], 60.00th=[  253],
     | 70.00th=[  270], 80.00th=[  298], 90.00th=[  370], 95.00th=[  516],
     | 99.00th=[ 1384], 99.50th=[ 2864], 99.90th=[ 7136], 99.95th=[ 9408],
     | 99.99th=[40192]
    bw (KB  /s): min= 2952, max=15256, per=12.51%, avg=12474.81, stdev=2609.26
    lat (usec) : 100=0.35%, 250=57.08%, 500=37.25%, 750=2.41%, 1000=1.29%
    lat (msec) : 2=0.93%, 4=0.39%, 10=0.24%, 20=0.02%, 50=0.02%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=2.06%, sys=7.24%, ctx=1501769, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1496285/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=5844.9MB, aggrb=99749KB/s, minb=99749KB/s, maxb=99749KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  sda: ios=1490329/10, merge=152/5, ticks=394100/0, in_queue=393836, util=99.70%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=9033: Mon Dec 16 03:22:39 2019
  write: io=1783.9MB, bw=30443KB/s, iops=7610, runt= 60004msec
    clat (usec): min=74, max=414357, avg=1045.25, stdev=6500.27
     lat (usec): min=75, max=414358, avg=1046.20, stdev=6500.29
    clat percentiles (usec):
     |  1.00th=[   77],  5.00th=[   79], 10.00th=[   80], 20.00th=[   83],
     | 30.00th=[   85], 40.00th=[   88], 50.00th=[   93], 60.00th=[   99],
     | 70.00th=[  110], 80.00th=[  126], 90.00th=[  163], 95.00th=[  247],
     | 99.00th=[37120], 99.50th=[42240], 99.90th=[59136], 99.95th=[69120],
     | 99.99th=[124416]
    bw (KB  /s): min=  354, max= 5447, per=12.53%, avg=3813.73, stdev=898.12
    lat (usec) : 100=60.09%, 250=34.95%, 500=1.55%, 750=0.33%, 1000=0.14%
    lat (msec) : 2=0.30%, 4=0.24%, 10=0.06%, 20=0.01%, 50=2.13%
    lat (msec) : 100=0.20%, 250=0.01%, 500=0.01%
  cpu          : usr=0.62%, sys=2.31%, ctx=924429, majf=0, minf=66
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=456672/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=1783.9MB, aggrb=30442KB/s, minb=30442KB/s, maxb=30442KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  sda: ios=65/456148, merge=144/1721, ticks=8/47868, in_queue=47764, util=78.32%


fio (Сингапур)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29700: Mon Dec 16 01:29:12 2019
  read : io=7460.3MB, bw=127319KB/s, iops=31829, runt= 60001msec
    clat (usec): min=2, max=100538, avg=245.95, stdev=526.10
     lat (usec): min=2, max=100539, avg=246.84, stdev=526.11
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[  181], 20.00th=[  213],
     | 30.00th=[  223], 40.00th=[  229], 50.00th=[  235], 60.00th=[  243],
     | 70.00th=[  258], 80.00th=[  274], 90.00th=[  310], 95.00th=[  354],
     | 99.00th=[  516], 99.50th=[  868], 99.90th=[ 2864], 99.95th=[ 3376],
     | 99.99th=[ 7648]
    bw (KB  /s): min= 7664, max=21344, per=12.51%, avg=15924.29, stdev=1653.02
    lat (usec) : 4=6.65%, 10=1.02%, 20=0.27%, 50=0.02%, 100=0.24%
    lat (usec) : 250=56.60%, 500=34.12%, 750=0.52%, 1000=0.11%
    lat (msec) : 2=0.18%, 4=0.25%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=2.37%, sys=8.65%, ctx=1767874, majf=2, minf=96
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1909821/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7460.3MB, aggrb=127319KB/s, minb=127319KB/s, maxb=127319KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1751526/10, merge=136/5, ticks=379692/0, in_queue=379548, util=98.45%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29726: Mon Dec 16 01:31:16 2019
  write: io=22301MB, bw=380447KB/s, iops=95111, runt= 60026msec
    clat (usec): min=2, max=99818, avg=50.79, stdev=815.47
     lat (usec): min=2, max=99819, avg=56.89, stdev=884.08
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   97], 99.50th=[ 1992], 99.90th=[13248], 99.95th=[18304],
     | 99.99th=[29824]
    bw (KB  /s): min=    1, max=60142, per=12.65%, avg=48143.86, stdev=4034.99
    lat (usec) : 4=50.77%, 10=47.42%, 20=0.47%, 50=0.18%, 100=0.18%
    lat (usec) : 250=0.22%, 500=0.10%, 750=0.04%, 1000=0.03%
    lat (msec) : 2=0.09%, 4=0.14%, 10=0.21%, 20=0.12%, 50=0.04%
    lat (msec) : 100=0.01%
  cpu          : usr=4.24%, sys=6.15%, ctx=116463, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5709171/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=22301MB, aggrb=380446KB/s, minb=380446KB/s, maxb=380446KB/s, mint=60026msec, maxt=60026msec

Disk stats (read/write):
  sda: ios=181/1565641, merge=31/1692, ticks=840/7100268, in_queue=7103928, util=90.92%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29713: Mon Dec 16 01:30:15 2019
  read : io=6622.7MB, bw=113024KB/s, iops=28255, runt= 60001msec
    clat (usec): min=64, max=1003.4K, avg=277.49, stdev=1028.50
     lat (usec): min=65, max=1003.4K, avg=278.34, stdev=1028.52
    clat percentiles (usec):
     |  1.00th=[  113],  5.00th=[  195], 10.00th=[  207], 20.00th=[  213],
     | 30.00th=[  219], 40.00th=[  225], 50.00th=[  233], 60.00th=[  241],
     | 70.00th=[  253], 80.00th=[  274], 90.00th=[  314], 95.00th=[  378],
     | 99.00th=[  868], 99.50th=[ 1688], 99.90th=[ 4960], 99.95th=[ 8032],
     | 99.99th=[43264]
    bw (KB  /s): min=  234, max=19920, per=12.52%, avg=14153.85, stdev=3251.43
    lat (usec) : 100=0.60%, 250=66.91%, 500=30.22%, 750=1.10%, 1000=0.31%
    lat (msec) : 2=0.42%, 4=0.27%, 10=0.14%, 20=0.03%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 2000=0.01%
  cpu          : usr=2.03%, sys=7.84%, ctx=1722385, majf=0, minf=66
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1695388/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=6622.7MB, aggrb=113023KB/s, minb=113023KB/s, maxb=113023KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1689128/10, merge=156/5, ticks=386848/0, in_queue=386776, util=99.63%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=29739: Mon Dec 16 01:32:16 2019
  write: io=2204.8MB, bw=37628KB/s, iops=9406, runt= 60001msec
    clat (usec): min=71, max=106108, avg=844.73, stdev=4994.03
     lat (usec): min=72, max=106109, avg=845.63, stdev=4994.04
    clat percentiles (usec):
     |  1.00th=[   77],  5.00th=[   78], 10.00th=[   79], 20.00th=[   81],
     | 30.00th=[   82], 40.00th=[   84], 50.00th=[   86], 60.00th=[   89],
     | 70.00th=[   95], 80.00th=[  105], 90.00th=[  126], 95.00th=[  161],
     | 99.00th=[33536], 99.50th=[35584], 99.90th=[41216], 99.95th=[45824],
     | 99.99th=[54016]
    bw (KB  /s): min= 2250, max= 5573, per=12.51%, avg=4707.66, stdev=501.18
    lat (usec) : 100=75.26%, 250=21.53%, 500=0.56%, 750=0.15%, 1000=0.06%
    lat (msec) : 2=0.12%, 4=0.10%, 10=0.01%, 20=0.01%, 50=2.19%
    lat (msec) : 100=0.02%, 250=0.01%
  cpu          : usr=0.78%, sys=2.67%, ctx=1141856, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=564422/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2204.8MB, aggrb=37627KB/s, minb=37627KB/s, maxb=37627KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=55/563927, merge=144/1523, ticks=8/44340, in_queue=44224, util=73.69%


fio (Великобритания, Лондон)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28829: Sun Dec 15 16:28:01 2019
  read : io=3631.9MB, bw=61959KB/s, iops=15489, runt= 60023msec
    clat (usec): min=2, max=134462, avg=509.94, stdev=1233.42
     lat (usec): min=2, max=134463, avg=510.93, stdev=1233.79
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[  189], 10.00th=[  207], 20.00th=[  223],
     | 30.00th=[  235], 40.00th=[  247], 50.00th=[  270], 60.00th=[  298],
     | 70.00th=[  346], 80.00th=[  430], 90.00th=[  652], 95.00th=[ 1320],
     | 99.00th=[ 5408], 99.50th=[ 7840], 99.90th=[14528], 99.95th=[17792],
     | 99.99th=[34048]
    bw (KB  /s): min= 1440, max=16184, per=12.55%, avg=7778.51, stdev=3012.08
    lat (usec) : 4=2.91%, 10=0.57%, 20=0.18%, 50=0.02%, 100=0.14%
    lat (usec) : 250=37.38%, 500=43.43%, 750=7.02%, 1000=1.99%
    lat (msec) : 2=2.60%, 4=1.95%, 10=1.55%, 20=0.24%, 50=0.02%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.38%, sys=5.15%, ctx=905144, majf=0, minf=67
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=929741/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=3631.9MB, aggrb=61958KB/s, minb=61958KB/s, maxb=61958KB/s, mint=60023msec, maxt=60023msec

Disk stats (read/write):
  sda: ios=895279/10, merge=144/5, ticks=396812/12, in_queue=396516, util=97.96%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28855: Sun Dec 15 16:30:07 2019
  write: io=19601MB, bw=334417KB/s, iops=83604, runt= 60019msec
    clat (usec): min=2, max=260881, avg=60.05, stdev=1166.81
     lat (usec): min=2, max=260882, avg=66.48, stdev=1253.78
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   48], 99.50th=[ 2128], 99.90th=[15680], 99.95th=[21376],
     | 99.99th=[40192]
    bw (KB  /s): min=    0, max=73357, per=12.86%, avg=43019.84, stdev=11843.30
    lat (usec) : 4=60.71%, 10=37.73%, 20=0.44%, 50=0.13%, 100=0.16%
    lat (usec) : 250=0.13%, 500=0.08%, 750=0.03%, 1000=0.02%
    lat (msec) : 2=0.06%, 4=0.11%, 10=0.21%, 20=0.12%, 50=0.05%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=3.95%, sys=5.77%, ctx=81576, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5017839/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=19601MB, aggrb=334416KB/s, minb=334416KB/s, maxb=334416KB/s, mint=60019msec, maxt=60019msec

Disk stats (read/write):
  sda: ios=699/1536338, merge=60/1385, ticks=532/5432372, in_queue=5430380, util=77.08%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28842: Sun Dec 15 16:29:06 2019
  read : io=4286.8MB, bw=73146KB/s, iops=18286, runt= 60011msec
    clat (usec): min=66, max=157991, avg=431.19, stdev=1243.53
     lat (usec): min=67, max=157992, avg=432.10, stdev=1243.78
    clat percentiles (usec):
     |  1.00th=[  113],  5.00th=[  191], 10.00th=[  203], 20.00th=[  215],
     | 30.00th=[  225], 40.00th=[  235], 50.00th=[  249], 60.00th=[  266],
     | 70.00th=[  290], 80.00th=[  338], 90.00th=[  478], 95.00th=[  788],
     | 99.00th=[ 4640], 99.50th=[ 7456], 99.90th=[14912], 99.95th=[18304],
     | 99.99th=[48384]
    bw (KB  /s): min=  924, max=15136, per=12.56%, avg=9184.53, stdev=3293.69
    lat (usec) : 100=0.62%, 250=50.61%, 500=39.45%, 750=4.09%, 1000=1.23%
    lat (msec) : 2=1.62%, 4=1.06%, 10=1.06%, 20=0.24%, 50=0.03%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.62%, sys=5.73%, ctx=1112738, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1097395/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4286.8MB, aggrb=73146KB/s, minb=73146KB/s, maxb=73146KB/s, mint=60011msec, maxt=60011msec

Disk stats (read/write):
  sda: ios=1096684/10, merge=159/5, ticks=395396/4, in_queue=395132, util=99.05%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28870: Sun Dec 15 16:31:07 2019
  write: io=750168KB, bw=12501KB/s, iops=3125, runt= 60008msec
    clat (usec): min=76, max=396545, avg=2550.32, stdev=14063.14
     lat (usec): min=77, max=396546, avg=2551.50, stdev=14063.56
    clat percentiles (usec):
     |  1.00th=[   81],  5.00th=[   84], 10.00th=[   87], 20.00th=[   94],
     | 30.00th=[  101], 40.00th=[  112], 50.00th=[  129], 60.00th=[  151],
     | 70.00th=[  179], 80.00th=[  243], 90.00th=[  548], 95.00th=[ 3376],
     | 99.00th=[74240], 99.50th=[98816], 99.90th=[164864], 99.95th=[189440],
     | 99.99th=[276480]
    bw (KB  /s): min=  103, max= 4120, per=12.70%, avg=1587.53, stdev=808.38
    lat (usec) : 100=28.22%, 250=52.47%, 500=8.76%, 750=1.96%, 1000=0.91%
    lat (msec) : 2=1.63%, 4=1.65%, 10=0.97%, 20=0.19%, 50=1.17%
    lat (msec) : 100=1.60%, 250=0.47%, 500=0.01%
  cpu          : usr=0.36%, sys=1.58%, ctx=383096, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=187542/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=750168KB, aggrb=12501KB/s, minb=12501KB/s, maxb=12501KB/s, mint=60008msec, maxt=60008msec

Disk stats (read/write):
  sda: ios=232/187357, merge=433/2112, ticks=520/52288, in_queue=52668, util=85.28%


fio (США, Атланта)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=1353: Thu Dec  5 08:14:33 2019
  read : io=5863.2MB, bw=100061KB/s, iops=25015, runt= 60002msec
    clat (usec): min=1, max=109953, avg=316.23, stdev=905.11
     lat (usec): min=1, max=109954, avg=316.70, stdev=905.29
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[   10], 10.00th=[  169], 20.00th=[  195],
     | 30.00th=[  213], 40.00th=[  227], 50.00th=[  245], 60.00th=[  266],
     | 70.00th=[  298], 80.00th=[  342], 90.00th=[  418], 95.00th=[  510],
     | 99.00th=[ 1288], 99.50th=[ 2576], 99.90th=[12480], 99.95th=[17792],
     | 99.99th=[43264]
    bw (KB  /s): min= 4007, max=19840, per=12.49%, avg=12499.98, stdev=3205.40
    lat (usec) : 2=0.12%, 4=1.95%, 10=2.75%, 20=1.10%, 50=0.12%
    lat (usec) : 100=0.27%, 250=46.38%, 500=41.94%, 750=3.30%, 1000=0.66%
    lat (msec) : 2=0.73%, 4=0.38%, 10=0.16%, 20=0.10%, 50=0.03%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.12%, sys=7.37%, ctx=1424468, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1500958/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=5863.2MB, aggrb=100060KB/s, minb=100060KB/s, maxb=100060KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  sda: ios=1407476/10, merge=144/5, ticks=367876/0, in_queue=367508, util=98.27%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=1386: Thu Dec  5 08:16:37 2019
  write: io=30002MB, bw=511974KB/s, iops=127993, runt= 60007msec
    clat (usec): min=1, max=168979, avg=47.39, stdev=843.23
     lat (usec): min=1, max=270942, avg=50.03, stdev=898.26
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    2], 50.00th=[    2], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    3], 90.00th=[    3], 95.00th=[    3],
     | 99.00th=[   16], 99.50th=[  382], 99.90th=[14912], 99.95th=[19328],
     | 99.99th=[29056]
    bw (KB  /s): min=    1, max=89640, per=12.64%, avg=64734.89, stdev=7943.05
    lat (usec) : 2=0.09%, 4=96.74%, 10=2.01%, 20=0.19%, 50=0.16%
    lat (usec) : 100=0.19%, 250=0.10%, 500=0.04%, 750=0.03%, 1000=0.02%
    lat (msec) : 2=0.05%, 4=0.05%, 10=0.15%, 20=0.14%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.71%, sys=7.21%, ctx=89916, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=7680511/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=30002MB, aggrb=511974KB/s, minb=511974KB/s, maxb=511974KB/s, mint=60007msec, maxt=60007msec

Disk stats (read/write):
  sda: ios=149/2401249, merge=59/1413, ticks=224/4709572, in_queue=4705400, util=65.68%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=1366: Thu Dec  5 08:15:37 2019
  read : io=4400.3MB, bw=75092KB/s, iops=18773, runt= 60004msec
    clat (usec): min=54, max=146994, avg=421.38, stdev=1591.75
     lat (usec): min=55, max=146995, avg=422.04, stdev=1592.67
    clat percentiles (usec):
     |  1.00th=[   97],  5.00th=[  159], 10.00th=[  173], 20.00th=[  193],
     | 30.00th=[  209], 40.00th=[  223], 50.00th=[  241], 60.00th=[  266],
     | 70.00th=[  302], 80.00th=[  354], 90.00th=[  462], 95.00th=[  676],
     | 99.00th=[ 4048], 99.50th=[ 9280], 99.90th=[19584], 99.95th=[28032],
     | 99.99th=[60672]
    bw (KB  /s): min= 1760, max=19184, per=12.53%, avg=9406.77, stdev=3435.47
    lat (usec) : 100=1.10%, 250=52.98%, 500=37.72%, 750=3.93%, 1000=1.36%
    lat (msec) : 2=1.36%, 4=0.55%, 10=0.55%, 20=0.37%, 50=0.08%
    lat (msec) : 100=0.02%, 250=0.01%
  cpu          : usr=1.07%, sys=5.29%, ctx=1151545, majf=0, minf=67
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1126460/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4400.3MB, aggrb=75092KB/s, minb=75092KB/s, maxb=75092KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  sda: ios=1124490/19, merge=311/14, ticks=392884/24, in_queue=392608, util=98.99%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=1399: Thu Dec  5 08:17:38 2019
  write: io=1424.9MB, bw=24315KB/s, iops=6078, runt= 60003msec
    clat (usec): min=66, max=168724, avg=1311.68, stdev=7949.99
     lat (usec): min=66, max=168724, avg=1312.29, stdev=7950.64
    clat percentiles (usec):
     |  1.00th=[   70],  5.00th=[   73], 10.00th=[   75], 20.00th=[   78],
     | 30.00th=[   81], 40.00th=[   86], 50.00th=[   91], 60.00th=[  100],
     | 70.00th=[  118], 80.00th=[  167], 90.00th=[  286], 95.00th=[  390],
     | 99.00th=[43776], 99.50th=[58112], 99.90th=[91648], 99.95th=[103936],
     | 99.99th=[125440]
    bw (KB  /s): min=  600, max= 5231, per=12.54%, avg=3048.58, stdev=962.56
    lat (usec) : 100=59.56%, 250=28.19%, 500=8.00%, 750=0.69%, 1000=0.39%
    lat (msec) : 2=0.43%, 4=0.14%, 10=0.14%, 20=0.08%, 50=1.67%
    lat (msec) : 100=0.63%, 250=0.06%
  cpu          : usr=0.39%, sys=2.24%, ctx=738676, majf=0, minf=59
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=364746/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=1424.9MB, aggrb=24315KB/s, minb=24315KB/s, maxb=24315KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  sda: ios=152/363129, merge=338/1944, ticks=264/48728, in_queue=48864, util=79.69%


fio (США, Даллас)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=1700: Wed Dec 11 14:48:11 2019
  read : io=6200.4MB, bw=105812KB/s, iops=26452, runt= 60001msec
    clat (usec): min=2, max=102104, avg=296.47, stdev=729.41
     lat (usec): min=2, max=102105, avg=297.48, stdev=729.66
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[  201], 20.00th=[  223],
     | 30.00th=[  233], 40.00th=[  245], 50.00th=[  258], 60.00th=[  274],
     | 70.00th=[  290], 80.00th=[  314], 90.00th=[  358], 95.00th=[  422],
     | 99.00th=[ 1064], 99.50th=[ 2064], 99.90th=[ 6816], 99.95th=[10048],
     | 99.99th=[36608]
    bw (KB  /s): min= 7008, max=19088, per=12.50%, avg=13230.61, stdev=1790.79
    lat (usec) : 4=4.58%, 10=1.55%, 20=0.51%, 50=0.02%, 100=0.14%
    lat (usec) : 250=36.05%, 500=54.16%, 750=1.36%, 1000=0.53%
    lat (msec) : 2=0.56%, 4=0.36%, 10=0.13%, 20=0.03%, 50=0.02%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=2.03%, sys=8.96%, ctx=1487519, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1587201/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=6200.4MB, aggrb=105811KB/s, minb=105811KB/s, maxb=105811KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1477598/10, merge=134/5, ticks=337536/0, in_queue=337040, util=98.37%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=1726: Wed Dec 11 14:50:15 2019
  write: io=22194MB, bw=378745KB/s, iops=94686, runt= 60006msec
    clat (usec): min=2, max=116106, avg=52.63, stdev=827.76
     lat (usec): min=2, max=116107, avg=58.41, stdev=892.15
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   97], 99.50th=[ 2320], 99.90th=[13888], 99.95th=[19328],
     | 99.99th=[29056]
    bw (KB  /s): min=    1, max=60735, per=12.65%, avg=47903.56, stdev=4565.88
    lat (usec) : 4=45.50%, 10=52.63%, 20=0.53%, 50=0.15%, 100=0.21%
    lat (usec) : 250=0.19%, 500=0.11%, 750=0.05%, 1000=0.03%
    lat (msec) : 2=0.09%, 4=0.14%, 10=0.22%, 20=0.12%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=4.18%, sys=6.17%, ctx=119947, majf=0, minf=74
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5681736/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=22194MB, aggrb=378744KB/s, minb=378744KB/s, maxb=378744KB/s, mint=60006msec, maxt=60006msec

Disk stats (read/write):
  sda: ios=204/1655764, merge=0/1663, ticks=432/5708944, in_queue=5711996, util=75.42%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=1713: Wed Dec 11 14:49:15 2019
  read : io=6961.6MB, bw=118808KB/s, iops=29702, runt= 60001msec
    clat (usec): min=67, max=100512, avg=263.94, stdev=572.86
     lat (usec): min=68, max=100513, avg=264.80, stdev=572.88
    clat percentiles (usec):
     |  1.00th=[  149],  5.00th=[  199], 10.00th=[  205], 20.00th=[  213],
     | 30.00th=[  225], 40.00th=[  231], 50.00th=[  237], 60.00th=[  243],
     | 70.00th=[  258], 80.00th=[  274], 90.00th=[  306], 95.00th=[  346],
     | 99.00th=[  516], 99.50th=[  932], 99.90th=[ 3088], 99.95th=[ 6304],
     | 99.99th=[14784]
    bw (KB  /s): min= 7356, max=17056, per=12.51%, avg=14865.38, stdev=1547.41
    lat (usec) : 100=0.36%, 250=65.31%, 500=33.25%, 750=0.45%, 1000=0.17%
    lat (msec) : 2=0.26%, 4=0.12%, 10=0.07%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=2.30%, sys=9.10%, ctx=1787510, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1782151/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=6961.6MB, aggrb=118808KB/s, minb=118808KB/s, maxb=118808KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1775430/12, merge=142/6, ticks=374772/0, in_queue=374608, util=99.88%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=1739: Wed Dec 11 14:51:16 2019
  write: io=2319.8MB, bw=39588KB/s, iops=9897, runt= 60003msec
    clat (usec): min=72, max=129405, avg=802.66, stdev=4689.80
     lat (usec): min=73, max=129406, avg=803.62, stdev=4689.82
    clat percentiles (usec):
     |  1.00th=[   76],  5.00th=[   78], 10.00th=[   79], 20.00th=[   81],
     | 30.00th=[   82], 40.00th=[   84], 50.00th=[   86], 60.00th=[   88],
     | 70.00th=[   91], 80.00th=[   97], 90.00th=[  111], 95.00th=[  137],
     | 99.00th=[30848], 99.50th=[33024], 99.90th=[37632], 99.95th=[41216],
     | 99.99th=[62720]
    bw (KB  /s): min= 2833, max= 6248, per=12.52%, avg=4956.34, stdev=397.19
    lat (usec) : 100=83.08%, 250=14.10%, 500=0.21%, 750=0.09%, 1000=0.05%
    lat (msec) : 2=0.10%, 4=0.08%, 10=0.01%, 20=0.01%, 50=2.27%
    lat (msec) : 100=0.02%, 250=0.01%
  cpu          : usr=0.88%, sys=2.91%, ctx=1201789, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=593856/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2319.8MB, aggrb=39588KB/s, minb=39588KB/s, maxb=39588KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  sda: ios=56/593337, merge=143/1500, ticks=12/43260, in_queue=43144, util=71.87%


fio (США, Фримонт)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30282: Wed Dec 11 18:48:11 2019
  read : io=7310.9MB, bw=124764KB/s, iops=31190, runt= 60004msec
    clat (usec): min=2, max=108489, avg=250.77, stdev=583.00
     lat (usec): min=2, max=108489, avg=251.65, stdev=583.02
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[  159], 20.00th=[  211],
     | 30.00th=[  223], 40.00th=[  231], 50.00th=[  239], 60.00th=[  247],
     | 70.00th=[  258], 80.00th=[  274], 90.00th=[  306], 95.00th=[  342],
     | 99.00th=[  540], 99.50th=[ 1368], 99.90th=[ 3504], 99.95th=[ 6496],
     | 99.99th=[13888]
    bw (KB  /s): min= 7848, max=20600, per=12.51%, avg=15607.82, stdev=1622.54
    lat (usec) : 4=6.90%, 10=0.84%, 20=0.19%, 50=0.01%, 100=0.37%
    lat (usec) : 250=53.78%, 500=36.77%, 750=0.37%, 1000=0.14%
    lat (msec) : 2=0.23%, 4=0.30%, 10=0.07%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=2.17%, sys=8.32%, ctx=1751322, majf=3, minf=88
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1871578/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7310.9MB, aggrb=124763KB/s, minb=124763KB/s, maxb=124763KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  sda: ios=1719411/10, merge=136/5, ticks=367604/0, in_queue=367292, util=98.42%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30308: Wed Dec 11 18:50:15 2019
  write: io=22228MB, bw=379297KB/s, iops=94824, runt= 60010msec
    clat (usec): min=2, max=71934, avg=51.74, stdev=781.09
     lat (usec): min=3, max=76027, avg=57.84, stdev=846.45
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[  105], 99.50th=[ 2640], 99.90th=[12864], 99.95th=[18304],
     | 99.99th=[28032]
    bw (KB  /s): min=    1, max=57849, per=12.64%, avg=47960.37, stdev=3393.26
    lat (usec) : 4=52.13%, 10=46.07%, 20=0.42%, 50=0.11%, 100=0.22%
    lat (usec) : 250=0.21%, 500=0.11%, 750=0.04%, 1000=0.03%
    lat (msec) : 2=0.09%, 4=0.16%, 10=0.24%, 20=0.11%, 50=0.04%
    lat (msec) : 100=0.01%
  cpu          : usr=4.16%, sys=6.13%, ctx=122895, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5690402/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=22228MB, aggrb=379296KB/s, minb=379296KB/s, maxb=379296KB/s, mint=60010msec, maxt=60010msec

Disk stats (read/write):
  sda: ios=172/1796795, merge=31/1518, ticks=196/5151300, in_queue=5149896, util=69.28%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30295: Wed Dec 11 18:49:15 2019
  read : io=7053.2MB, bw=120372KB/s, iops=30093, runt= 60001msec
    clat (usec): min=67, max=149994, avg=259.65, stdev=564.00
     lat (usec): min=68, max=149995, avg=260.50, stdev=564.01
    clat percentiles (usec):
     |  1.00th=[  118],  5.00th=[  193], 10.00th=[  205], 20.00th=[  217],
     | 30.00th=[  225], 40.00th=[  231], 50.00th=[  239], 60.00th=[  247],
     | 70.00th=[  258], 80.00th=[  270], 90.00th=[  298], 95.00th=[  326],
     | 99.00th=[  442], 99.50th=[  732], 99.90th=[ 3536], 99.95th=[ 6752],
     | 99.99th=[14016]
    bw (KB  /s): min= 7751, max=16512, per=12.51%, avg=15054.93, stdev=1315.85
    lat (usec) : 100=0.47%, 250=63.33%, 500=35.47%, 750=0.24%, 1000=0.10%
    lat (msec) : 2=0.20%, 4=0.09%, 10=0.08%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=2.33%, sys=8.06%, ctx=1846033, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1805615/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7053.2MB, aggrb=120372KB/s, minb=120372KB/s, maxb=120372KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1803762/10, merge=144/5, ticks=382228/0, in_queue=382160, util=99.88%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=30321: Wed Dec 11 18:51:16 2019
  write: io=2204.4MB, bw=37617KB/s, iops=9404, runt= 60005msec
    clat (usec): min=75, max=155277, avg=845.18, stdev=5068.01
     lat (usec): min=76, max=155278, avg=846.06, stdev=5068.02
    clat percentiles (usec):
     |  1.00th=[   79],  5.00th=[   81], 10.00th=[   82], 20.00th=[   84],
     | 30.00th=[   85], 40.00th=[   87], 50.00th=[   90], 60.00th=[   92],
     | 70.00th=[   97], 80.00th=[  104], 90.00th=[  122], 95.00th=[  147],
     | 99.00th=[34048], 99.50th=[36096], 99.90th=[40704], 99.95th=[43776],
     | 99.99th=[55040]
    bw (KB  /s): min= 2514, max= 5664, per=12.51%, avg=4707.09, stdev=458.93
    lat (usec) : 100=74.69%, 250=22.43%, 500=0.32%, 750=0.13%, 1000=0.06%
    lat (msec) : 2=0.13%, 4=0.09%, 10=0.01%, 20=0.01%, 50=2.14%
    lat (msec) : 100=0.02%, 250=0.01%
  cpu          : usr=0.73%, sys=2.64%, ctx=1141286, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=564308/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2204.4MB, aggrb=37617KB/s, minb=37617KB/s, maxb=37617KB/s, mint=60005msec, maxt=60005msec

Disk stats (read/write):
  sda: ios=63/563824, merge=144/1549, ticks=68/44880, in_queue=44828, util=74.54%


fio (США, Ньюарк)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=10448: Thu Dec 12 13:06:37 2019
  read : io=7558.4MB, bw=128994KB/s, iops=32248, runt= 60001msec
    clat (usec): min=1, max=1002.2K, avg=244.98, stdev=946.83
     lat (usec): min=1, max=1002.2K, avg=245.40, stdev=946.83
    clat percentiles (usec):
     |  1.00th=[    1],  5.00th=[    2], 10.00th=[  153], 20.00th=[  183],
     | 30.00th=[  199], 40.00th=[  211], 50.00th=[  223], 60.00th=[  239],
     | 70.00th=[  262], 80.00th=[  298], 90.00th=[  354], 95.00th=[  410],
     | 99.00th=[  596], 99.50th=[  844], 99.90th=[ 2544], 99.95th=[ 2960],
     | 99.99th=[13888]
    bw (KB  /s): min= 4981, max=22968, per=12.52%, avg=16154.17, stdev=1818.71
    lat (usec) : 2=1.77%, 4=3.76%, 10=1.80%, 20=0.40%, 50=0.18%
    lat (usec) : 100=0.47%, 250=57.15%, 500=32.56%, 750=1.32%, 1000=0.19%
    lat (msec) : 2=0.19%, 4=0.20%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 2000=0.01%
  cpu          : usr=1.24%, sys=7.51%, ctx=1795177, majf=2, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1934945/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7558.4MB, aggrb=128994KB/s, minb=128994KB/s, maxb=128994KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1775630/10, merge=144/5, ticks=400756/0, in_queue=400528, util=98.21%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=10474: Thu Dec 12 13:08:40 2019
  write: io=32935MB, bw=561990KB/s, iops=140497, runt= 60011msec
    clat (usec): min=1, max=71142, avg=42.34, stdev=755.46
     lat (usec): min=1, max=71142, avg=44.95, stdev=799.24
    clat percentiles (usec):
     |  1.00th=[    1],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    2], 50.00th=[    2], 60.00th=[    2],
     | 70.00th=[    2], 80.00th=[    3], 90.00th=[    3], 95.00th=[    3],
     | 99.00th=[   14], 99.50th=[  378], 99.90th=[13120], 99.95th=[17536],
     | 99.99th=[26752]
    bw (KB  /s): min=    1, max=90912, per=12.64%, avg=71015.53, stdev=7060.81
    lat (usec) : 2=1.50%, 4=96.15%, 10=1.26%, 20=0.13%, 50=0.11%
    lat (usec) : 100=0.22%, 250=0.10%, 500=0.04%, 750=0.04%, 1000=0.03%
    lat (msec) : 2=0.05%, 4=0.05%, 10=0.16%, 20=0.12%, 50=0.03%
    lat (msec) : 100=0.01%
  cpu          : usr=1.88%, sys=7.08%, ctx=102574, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=8431394/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=32935MB, aggrb=561989KB/s, minb=561989KB/s, maxb=561989KB/s, mint=60011msec, maxt=60011msec

Disk stats (read/write):
  sda: ios=204/2664201, merge=0/1530, ticks=96/5015780, in_queue=5011968, util=66.88%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=10461: Thu Dec 12 13:07:39 2019
  read : io=7272.5MB, bw=124105KB/s, iops=31026, runt= 60002msec
    clat (usec): min=52, max=150788, avg=254.66, stdev=558.22
     lat (usec): min=52, max=150788, avg=255.12, stdev=558.23
    clat percentiles (usec):
     |  1.00th=[  129],  5.00th=[  159], 10.00th=[  169], 20.00th=[  185],
     | 30.00th=[  199], 40.00th=[  209], 50.00th=[  221], 60.00th=[  237],
     | 70.00th=[  262], 80.00th=[  302], 90.00th=[  358], 95.00th=[  418],
     | 99.00th=[  564], 99.50th=[  708], 99.90th=[ 1928], 99.95th=[ 2832],
     | 99.99th=[10816]
    bw (KB  /s): min= 8656, max=22752, per=12.51%, avg=15520.89, stdev=1889.80
    lat (usec) : 100=0.52%, 250=65.45%, 500=32.26%, 750=1.34%, 1000=0.19%
    lat (msec) : 2=0.16%, 4=0.07%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=1.28%, sys=6.40%, ctx=1876818, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1861643/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7272.5MB, aggrb=124105KB/s, minb=124105KB/s, maxb=124105KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  sda: ios=1861380/10, merge=144/5, ticks=390012/0, in_queue=389716, util=99.03%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=10487: Thu Dec 12 13:09:40 2019
  write: io=2191.1MB, bw=37407KB/s, iops=9351, runt= 60004msec
    clat (usec): min=64, max=485184, avg=852.18, stdev=5707.61
     lat (usec): min=65, max=485184, avg=852.64, stdev=5707.61
    clat percentiles (usec):
     |  1.00th=[   69],  5.00th=[   70], 10.00th=[   72], 20.00th=[   73],
     | 30.00th=[   75], 40.00th=[   76], 50.00th=[   79], 60.00th=[   82],
     | 70.00th=[   87], 80.00th=[  104], 90.00th=[  187], 95.00th=[  266],
     | 99.00th=[38656], 99.50th=[40704], 99.90th=[45824], 99.95th=[52480],
     | 99.99th=[88576]
    bw (KB  /s): min=  311, max= 6184, per=12.54%, avg=4691.88, stdev=732.96
    lat (usec) : 100=78.71%, 250=15.08%, 500=3.93%, 750=0.17%, 1000=0.08%
    lat (msec) : 2=0.08%, 4=0.02%, 10=0.01%, 20=0.01%, 50=1.83%
    lat (msec) : 100=0.06%, 250=0.01%, 500=0.01%
  cpu          : usr=0.42%, sys=2.55%, ctx=1133358, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=561138/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2191.1MB, aggrb=37406KB/s, minb=37406KB/s, maxb=37406KB/s, mint=60004msec, maxt=60004msec

Disk stats (read/write):
  sda: ios=55/560849, merge=144/1569, ticks=48/45804, in_queue=45712, util=76.12%


Speedtest (Австралия, Сидней)
Retrieving speedtest.net configuration...
Testing from Linode (172.105.166.138)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by 'Yes' Optus (Sydney) [1.01 km]: 5.273 ms
Testing download speed................................................................................
Download: 3748.73 Mbit/s
Testing upload speed................................................................................................
Upload: 1914.90 Mbit/s


Speedtest (Канада, Торонто)
Retrieving speedtest.net configuration...
Testing from Linode (172.105.27.90)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Bell Mobility (Toronto, ON) [3.45 km]: 4.353 ms
Testing download speed................................................................................
Download: 2291.58 Mbit/s
Testing upload speed................................................................................................
Upload: 1898.46 Mbit/s


Speedtest (Германия, Франкфурт)
Retrieving speedtest.net configuration...
Testing from Linode (172.105.66.241)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Vodafone DE (Frankfurt) [0.89 km]: 2.5 ms
Testing download speed................................................................................
Download: 334.21 Mbit/s
Testing upload speed................................................................................................
Upload: 778.76 Mbit/s


Speedtest (Индия, Мумбаи)
Retrieving speedtest.net configuration...
Testing from Linode (172.105.53.168)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by I-ON (Mumbai) [13.76 km]: 41.212 ms
Testing download speed................................................................................
Download: 2668.93 Mbit/s
Testing upload speed................................................................................................
Upload: 2456.87 Mbit/s


Speedtest (Япония, Токио)
Retrieving speedtest.net configuration...
Testing from Linode (172.104.99.120)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by OPEN Project (via 20G SINET) (Tokyo) [6.34 km]: 2.983 ms
Testing download speed................................................................................
Download: 1514.59 Mbit/s
Testing upload speed................................................................................................
Upload: 742.07 Mbit/s


Speedtest (Сингапур)
Retrieving speedtest.net configuration...
Testing from Linode (139.162.42.89)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by StarHub Mobile Pte Ltd (Singapore) [7.42 km]: 4.323 ms
Testing download speed................................................................................
Download: 2594.09 Mbit/s
Testing upload speed................................................................................................
Upload: 2845.87 Mbit/s


Speedtest (Великобритания, Лондон)
Retrieving speedtest.net configuration...
Testing from Linode (85.159.213.129)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Spitfire Network Services (London) [0.92 km]: 3.546 ms
Testing download speed................................................................................
Download: 1292.02 Mbit/s
Testing upload speed................................................................................................
Upload: 1262.76 Mbit/s


Speedtest (США, Атланта)
Retrieving speedtest.net configuration...
Testing from Linode (45.79.223.127)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Hotwire Fision (Atlanta, GA) [0.37 km]: 1.599 ms
Testing download speed................................................................................
Download: 623.66 Mbit/s
Testing upload speed................................................................................................
Upload: 899.49 Mbit/s


Speedtest (США, Даллас)
Retrieving speedtest.net configuration...
Testing from Linode (45.33.18.235)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Windstream (Dallas, TX) [2.33 km]: 6.618 ms
Testing download speed................................................................................
Download: 1750.90 Mbit/s
Testing upload speed................................................................................................
Upload: 2639.56 Mbit/s


Speedtest (США, Фримонт)
Retrieving speedtest.net configuration...
Testing from Linode (45.79.226.251)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Converse in Code Networks (Fremont, CA) [1.89 km]: 4.107 ms
Testing download speed................................................................................
Download: 683.26 Mbit/s
Testing upload speed................................................................................................
Upload: 746.06 Mbit/s


Speedtest (США, Ньюарк)
Retrieving speedtest.net configuration...
Testing from Linode (45.79.181.215)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Interserver, inc (Secaucus, NJ) [11.03 km]: 5.212 ms
Testing download speed................................................................................
Download: 1773.85 Mbit/s
Testing upload speed................................................................................................
Upload: 2021.52 Mbit/s



Виртуальный сервер от Hetzner:




  • тариф CX11 (20 GB SSD / 1x vCPU / 2 GB RAM / 10000 Mbit/s за 2.99€ в месяц);
  • дата-центры в Германии, Финляндии.

Тесты:


Unixbench (Германия, Фалькенштейн)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: debian-2gb-fsn1-1: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel Xeon Processor (Skylake, IBRS) (4589.2 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   17:44:12 up  1:02,  1 user,  load average: 0.45, 0.12, 0.04; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Wed Dec 04 2019 17:44:12 - 18:12:22
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       31999500.4 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4041.0 MWIPS (9.9 s, 7 samples)
Execl Throughput                               4467.7 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        522204.8 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          132016.7 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1293513.2 KBps  (30.0 s, 2 samples)
Pipe Throughput                              703333.3 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 105055.3 lps   (10.0 s, 7 samples)
Process Creation                               9959.9 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7888.1 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   1021.9 lpm   (60.0 s, 2 samples)
System Call Overhead                         420618.3 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   31999500.4   2742.0
Double-Precision Whetstone                       55.0       4041.0    734.7
Execl Throughput                                 43.0       4467.7   1039.0
File Copy 1024 bufsize 2000 maxblocks          3960.0     522204.8   1318.7
File Copy 256 bufsize 500 maxblocks            1655.0     132016.7    797.7
File Copy 4096 bufsize 8000 maxblocks          5800.0    1293513.2   2230.2
Pipe Throughput                               12440.0     703333.3    565.4
Pipe-based Context Switching                   4000.0     105055.3    262.6
Process Creation                                126.0       9959.9    790.5
Shell Scripts (1 concurrent)                     42.4       7888.1   1860.4
Shell Scripts (8 concurrent)                      6.0       1021.9   1703.2
System Call Overhead                          15000.0     420618.3    280.4
                                                                   ========
System Benchmarks Index Score                                         945.8


Unixbench (Германия, Нюрнберг)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: debian-2gb-nbg1-1: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel Xeon Processor (Skylake, IBRS) (4200.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   17:43:43 up  1:01,  1 user,  load average: 0.68, 0.20, 0.07; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Wed Dec 04 2019 17:43:43 - 18:11:50
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       30981175.0 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3970.6 MWIPS (9.9 s, 7 samples)
Execl Throughput                               4214.2 lps   (29.6 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        455583.2 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          134162.5 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1327033.8 KBps  (30.0 s, 2 samples)
Pipe Throughput                              709339.4 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 102866.3 lps   (10.0 s, 7 samples)
Process Creation                               8830.0 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7109.0 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    888.5 lpm   (60.1 s, 2 samples)
System Call Overhead                         424010.8 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   30981175.0   2654.8
Double-Precision Whetstone                       55.0       3970.6    721.9
Execl Throughput                                 43.0       4214.2    980.0
File Copy 1024 bufsize 2000 maxblocks          3960.0     455583.2   1150.5
File Copy 256 bufsize 500 maxblocks            1655.0     134162.5    810.6
File Copy 4096 bufsize 8000 maxblocks          5800.0    1327033.8   2288.0
Pipe Throughput                               12440.0     709339.4    570.2
Pipe-based Context Switching                   4000.0     102866.3    257.2
Process Creation                                126.0       8830.0    700.8
Shell Scripts (1 concurrent)                     42.4       7109.0   1676.6
Shell Scripts (8 concurrent)                      6.0        888.5   1480.9
System Call Overhead                          15000.0     424010.8    282.7
                                                                   ========
System Benchmarks Index Score                                         901.7


Unixbench (Финляндия, Хельсинки)
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: debian-2gb-hel1-1: GNU/Linux
   OS: GNU/Linux -- 4.9.0-11-amd64 -- #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11)
   Machine: x86_64 (unknown)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel Xeon Processor (Skylake, IBRS) (4200.0 bogomips)
          x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   17:45:00 up  1:02,  1 user,  load average: 0.67, 0.22, 0.07; runlevel 5

------------------------------------------------------------------------
Benchmark Run: Wed Dec 04 2019 17:45:00 - 18:13:09
1 CPU in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       29256935.7 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3758.3 MWIPS (9.9 s, 7 samples)
Execl Throughput                               3536.8 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        416729.8 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          116120.6 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       1211197.2 KBps  (30.0 s, 2 samples)
Pipe Throughput                              636599.3 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  92831.7 lps   (10.0 s, 7 samples)
Process Creation                               8632.2 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   6427.6 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    784.8 lpm   (60.1 s, 2 samples)
System Call Overhead                         378836.2 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   29256935.7   2507.0
Double-Precision Whetstone                       55.0       3758.3    683.3
Execl Throughput                                 43.0       3536.8    822.5
File Copy 1024 bufsize 2000 maxblocks          3960.0     416729.8   1052.3
File Copy 256 bufsize 500 maxblocks            1655.0     116120.6    701.6
File Copy 4096 bufsize 8000 maxblocks          5800.0    1211197.2   2088.3
Pipe Throughput                               12440.0     636599.3    511.7
Pipe-based Context Switching                   4000.0      92831.7    232.1
Process Creation                                126.0       8632.2    685.1
Shell Scripts (1 concurrent)                     42.4       6427.6   1515.9
Shell Scripts (8 concurrent)                      6.0        784.8   1308.0
System Call Overhead                          15000.0     378836.2    252.6
                                                                   ========
System Benchmarks Index Score                                         817.1


dd (Германия, Фалькенштейн)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.1357 s, 685 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.41475 s, 889 MB/s


dd (Германия, Нюрнберг)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.34572 s, 494 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 61.8634 s, 34.7 MB/s


dd (Финляндия, Хельсинки)
Блоками по 1M
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.91158 s, 549 MB/s

Блоками по 64K
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 561.943 s, 3.8 MB/s


fio (Германия, Фалькенштейн)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=27954: Wed Dec  4 18:13:32 2019
  read : io=12935MB, bw=220758KB/s, iops=55189, runt= 60001msec
    clat (usec): min=1, max=15276, avg=138.82, stdev=194.23
     lat (usec): min=2, max=15277, avg=139.89, stdev=194.55
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[   68], 60.00th=[  189],
     | 70.00th=[  215], 80.00th=[  251], 90.00th=[  326], 95.00th=[  414],
     | 99.00th=[  660], 99.50th=[  812], 99.90th=[ 2192], 99.95th=[ 3056],
     | 99.99th=[ 4448]
    bw (KB  /s): min=13160, max=52192, per=12.50%, avg=27592.76, stdev=9637.34
    lat (usec) : 2=0.13%, 4=41.65%, 10=6.64%, 20=1.32%, 50=0.21%
    lat (usec) : 100=0.19%, 250=29.76%, 500=17.46%, 750=1.99%, 1000=0.37%
    lat (msec) : 2=0.18%, 4=0.09%, 10=0.02%, 20=0.01%
  cpu          : usr=2.75%, sys=7.24%, ctx=1708991, majf=2, minf=110
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=3311429/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=12935MB, aggrb=220758KB/s, minb=220758KB/s, maxb=220758KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1641059/31, merge=0/26, ticks=326344/444, in_queue=326668, util=97.95%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=27993: Wed Dec  4 18:15:39 2019
  write: io=30828MB, bw=526046KB/s, iops=131511, runt= 60009msec
    clat (usec): min=1, max=72109, avg=40.58, stdev=834.02
     lat (usec): min=2, max=72109, avg=44.54, stdev=898.63
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[    8], 99.50th=[   12], 99.90th=[15936], 99.95th=[20096],
     | 99.99th=[31104]
    bw (KB  /s): min=    1, max=80631, per=12.58%, avg=66166.54, stdev=7286.63
    lat (usec) : 2=0.01%, 4=74.79%, 10=24.56%, 20=0.28%, 50=0.05%
    lat (usec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.02%, 10=0.08%, 20=0.11%, 50=0.06%
    lat (msec) : 100=0.01%
  cpu          : usr=4.73%, sys=6.73%, ctx=50571, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=7891869/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=30828MB, aggrb=526045KB/s, minb=526045KB/s, maxb=526045KB/s, mint=60009msec, maxt=60009msec

Disk stats (read/write):
  sda: ios=527/424265, merge=89/1974, ticks=692/2002184, in_queue=2002388, util=27.50%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=27977: Wed Dec  4 18:14:37 2019
  read : io=7711.1MB, bw=131613KB/s, iops=32903, runt= 60002msec
    clat (usec): min=61, max=10698, avg=237.50, stdev=110.59
     lat (usec): min=62, max=10706, avg=238.41, stdev=110.73
    clat percentiles (usec):
     |  1.00th=[  117],  5.00th=[  141], 10.00th=[  157], 20.00th=[  177],
     | 30.00th=[  191], 40.00th=[  201], 50.00th=[  213], 60.00th=[  227],
     | 70.00th=[  245], 80.00th=[  274], 90.00th=[  342], 95.00th=[  418],
     | 99.00th=[  612], 99.50th=[  724], 99.90th=[ 1144], 99.95th=[ 1608],
     | 99.99th=[ 2928]
    bw (KB  /s): min= 8408, max=19592, per=12.50%, avg=16453.29, stdev=2036.10
    lat (usec) : 100=0.10%, 250=71.70%, 500=25.83%, 750=1.95%, 1000=0.28%
    lat (msec) : 2=0.11%, 4=0.03%, 10=0.01%, 20=0.01%
  cpu          : usr=2.15%, sys=6.16%, ctx=1978828, majf=0, minf=80
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1974259/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7711.1MB, aggrb=131612KB/s, minb=131612KB/s, maxb=131612KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  sda: ios=1969675/13, merge=0/7, ticks=379064/4, in_queue=378660, util=99.83%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=28016: Wed Dec  4 18:16:39 2019
  write: io=1513.9MB, bw=25834KB/s, iops=6458, runt= 60005msec
    clat (usec): min=71, max=55895, avg=1232.03, stdev=6253.29
     lat (usec): min=71, max=55898, avg=1233.17, stdev=6253.36
    clat percentiles (usec):
     |  1.00th=[   83],  5.00th=[   87], 10.00th=[   92], 20.00th=[   99],
     | 30.00th=[  106], 40.00th=[  114], 50.00th=[  125], 60.00th=[  143],
     | 70.00th=[  163], 80.00th=[  191], 90.00th=[  249], 95.00th=[  362],
     | 99.00th=[39168], 99.50th=[42240], 99.90th=[46336], 99.95th=[47872],
     | 99.99th=[51456]
    bw (KB  /s): min=  660, max= 4936, per=12.51%, avg=3231.59, stdev=699.18
    lat (usec) : 100=20.10%, 250=70.08%, 500=6.16%, 750=0.58%, 1000=0.05%
    lat (msec) : 2=0.03%, 4=0.02%, 10=0.01%, 20=0.01%, 50=2.97%
    lat (msec) : 100=0.02%
  cpu          : usr=0.64%, sys=2.58%, ctx=787097, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=387547/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=1513.9MB, aggrb=25834KB/s, minb=25834KB/s, maxb=25834KB/s, mint=60005msec, maxt=60005msec

Disk stats (read/write):
  sda: ios=680/387270, merge=0/1946, ticks=836/48344, in_queue=49052, util=80.31%


fio (Германия, Нюрнберг)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=4283: Wed Dec  4 18:14:01 2019
  read : io=10029MB, bw=171161KB/s, iops=42790, runt= 60001msec
    clat (usec): min=1, max=50938, avg=181.05, stdev=348.78
     lat (usec): min=2, max=50938, avg=182.04, stdev=349.01
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    4], 10.00th=[    5], 20.00th=[    6],
     | 30.00th=[    7], 40.00th=[    9], 50.00th=[  157], 60.00th=[  203],
     | 70.00th=[  243], 80.00th=[  302], 90.00th=[  426], 95.00th=[  564],
     | 99.00th=[  988], 99.50th=[ 1272], 99.90th=[ 2928], 99.95th=[ 3888],
     | 99.99th=[12608]
    bw (KB  /s): min=12264, max=34672, per=12.51%, avg=21413.37, stdev=6618.44
    lat (usec) : 2=0.01%, 4=4.65%, 10=35.58%, 20=6.15%, 50=1.07%
    lat (usec) : 100=0.10%, 250=24.16%, 500=21.47%, 750=4.58%, 1000=1.27%
    lat (msec) : 2=0.78%, 4=0.14%, 10=0.03%, 20=0.02%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=2.19%, sys=8.10%, ctx=1364825, majf=2, minf=95
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2567457/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=10029MB, aggrb=171160KB/s, minb=171160KB/s, maxb=171160KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1337530/8, merge=0/4, ticks=280012/0, in_queue=279880, util=97.89%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=4309: Wed Dec  4 18:16:07 2019
  write: io=30705MB, bw=523948KB/s, iops=130987, runt= 60010msec
    clat (usec): min=1, max=76630, avg=41.59, stdev=835.90
     lat (usec): min=2, max=82084, avg=45.14, stdev=893.25
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    5],
     | 99.00th=[    8], 99.50th=[   13], 99.90th=[16064], 99.95th=[20096],
     | 99.99th=[30080]
    bw (KB  /s): min=    1, max=83288, per=12.63%, avg=66191.01, stdev=7467.72
    lat (usec) : 2=0.01%, 4=69.29%, 10=29.98%, 20=0.36%, 50=0.04%
    lat (usec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.02%, 10=0.08%, 20=0.11%, 50=0.06%
    lat (msec) : 100=0.01%
  cpu          : usr=4.42%, sys=7.05%, ctx=56571, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=7860536/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=30705MB, aggrb=523948KB/s, minb=523948KB/s, maxb=523948KB/s, mint=60010msec, maxt=60010msec

Disk stats (read/write):
  sda: ios=138/439786, merge=0/1844, ticks=132/2046436, in_queue=2045992, util=28.66%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=4296: Wed Dec  4 18:15:06 2019
  read : io=7458.3MB, bw=127281KB/s, iops=31820, runt= 60003msec
    clat (usec): min=62, max=19424, avg=245.43, stdev=172.33
     lat (usec): min=62, max=19425, avg=246.39, stdev=172.54
    clat percentiles (usec):
     |  1.00th=[  122],  5.00th=[  143], 10.00th=[  155], 20.00th=[  171],
     | 30.00th=[  183], 40.00th=[  197], 50.00th=[  211], 60.00th=[  229],
     | 70.00th=[  251], 80.00th=[  290], 90.00th=[  370], 95.00th=[  458],
     | 99.00th=[  700], 99.50th=[  836], 99.90th=[ 1528], 99.95th=[ 2192],
     | 99.99th=[ 6304]
    bw (KB  /s): min=12104, max=19336, per=12.50%, avg=15914.07, stdev=1323.40
    lat (usec) : 100=0.04%, 250=69.70%, 500=26.60%, 750=2.89%, 1000=0.50%
    lat (msec) : 2=0.21%, 4=0.03%, 10=0.02%, 20=0.01%
  cpu          : usr=2.11%, sys=6.34%, ctx=1914023, majf=0, minf=66
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1909305/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=7458.3MB, aggrb=127280KB/s, minb=127280KB/s, maxb=127280KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  sda: ios=1904100/8, merge=0/5, ticks=362412/0, in_queue=362124, util=99.79%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=4323: Wed Dec  4 18:17:08 2019
  write: io=1437.8MB, bw=24535KB/s, iops=6133, runt= 60006msec
    clat (usec): min=59, max=78562, avg=1296.69, stdev=6139.93
     lat (usec): min=60, max=78563, avg=1297.96, stdev=6140.01
    clat percentiles (usec):
     |  1.00th=[   71],  5.00th=[   75], 10.00th=[   79], 20.00th=[   88],
     | 30.00th=[  100], 40.00th=[  114], 50.00th=[  131], 60.00th=[  151],
     | 70.00th=[  173], 80.00th=[  207], 90.00th=[  290], 95.00th=[  470],
     | 99.00th=[35584], 99.50th=[39168], 99.90th=[45824], 99.95th=[48896],
     | 99.99th=[55552]
    bw (KB  /s): min=  718, max= 5150, per=12.50%, avg=3067.24, stdev=665.86
    lat (usec) : 100=29.91%, 250=56.93%, 500=8.50%, 750=0.97%, 1000=0.13%
    lat (msec) : 2=0.07%, 4=0.02%, 10=0.02%, 20=0.02%, 50=3.39%
    lat (msec) : 100=0.04%
  cpu          : usr=0.72%, sys=2.84%, ctx=749031, majf=0, minf=73
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=368068/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=1437.8MB, aggrb=24535KB/s, minb=24535KB/s, maxb=24535KB/s, mint=60006msec, maxt=60006msec

Disk stats (read/write):
  sda: ios=418/367714, merge=0/2000, ticks=376/46720, in_queue=46952, util=77.54%


fio (Финляндия, Хельсинки)
Read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=16484: Wed Dec  4 18:23:39 2019
  read : io=9472.7MB, bw=161663KB/s, iops=40415, runt= 60001msec
    clat (usec): min=1, max=134988, avg=191.81, stdev=499.59
     lat (usec): min=2, max=134989, avg=192.84, stdev=499.98
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[    4], 20.00th=[    6],
     | 30.00th=[    8], 40.00th=[   10], 50.00th=[  157], 60.00th=[  207],
     | 70.00th=[  249], 80.00th=[  318], 90.00th=[  458], 95.00th=[  612],
     | 99.00th=[ 1064], 99.50th=[ 1368], 99.90th=[ 3280], 99.95th=[ 4320],
     | 99.99th=[15424]
    bw (KB  /s): min= 2974, max=37808, per=12.49%, avg=20198.33, stdev=6446.07
    lat (usec) : 2=0.01%, 4=8.45%, 10=31.44%, 20=6.28%, 50=1.26%
    lat (usec) : 100=0.06%, 250=22.59%, 500=21.67%, 750=5.44%, 1000=1.60%
    lat (msec) : 2=0.99%, 4=0.14%, 10=0.05%, 20=0.02%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=2.26%, sys=8.06%, ctx=1284108, majf=2, minf=113
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2424986/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=9472.7MB, aggrb=161663KB/s, minb=161663KB/s, maxb=161663KB/s, mint=60001msec, maxt=60001msec

Disk stats (read/write):
  sda: ios=1264059/8, merge=0/4, ticks=269492/0, in_queue=269092, util=97.51%

Write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=16510: Wed Dec  4 18:25:44 2019
  write: io=27890MB, bw=475947KB/s, iops=118986, runt= 60006msec
    clat (usec): min=1, max=124945, avg=46.69, stdev=899.56
     lat (usec): min=2, max=124946, avg=50.31, stdev=954.85
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    5],
     | 99.00th=[   10], 99.50th=[   18], 99.90th=[16064], 99.95th=[20608],
     | 99.99th=[31104]
    bw (KB  /s): min=    1, max=75300, per=12.61%, avg=59998.47, stdev=7600.10
    lat (usec) : 2=0.01%, 4=49.88%, 10=48.95%, 20=0.69%, 50=0.11%
    lat (usec) : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.02%, 10=0.09%, 20=0.13%, 50=0.07%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=4.33%, sys=7.01%, ctx=51459, majf=0, minf=83
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=7139920/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=27890MB, aggrb=475947KB/s, minb=475947KB/s, maxb=475947KB/s, mint=60006msec, maxt=60006msec

Disk stats (read/write):
  sda: ios=76/415614, merge=61/1682, ticks=120/1827512, in_queue=1827116, util=26.02%

Direct read
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=16497: Wed Dec  4 18:24:44 2019
  read : io=6984.9MB, bw=119201KB/s, iops=29800, runt= 60003msec
    clat (usec): min=67, max=20518, avg=261.82, stdev=203.48
     lat (usec): min=67, max=20518, avg=262.86, stdev=203.69
    clat percentiles (usec):
     |  1.00th=[  119],  5.00th=[  141], 10.00th=[  153], 20.00th=[  173],
     | 30.00th=[  187], 40.00th=[  203], 50.00th=[  219], 60.00th=[  239],
     | 70.00th=[  266], 80.00th=[  314], 90.00th=[  418], 95.00th=[  524],
     | 99.00th=[  772], 99.50th=[  916], 99.90th=[ 1816], 99.95th=[ 3088],
     | 99.99th=[ 7904]
    bw (KB  /s): min=10992, max=18064, per=12.50%, avg=14901.36, stdev=1197.05
    lat (usec) : 100=0.05%, 250=64.41%, 500=29.73%, 750=4.67%, 1000=0.78%
    lat (msec) : 2=0.28%, 4=0.05%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=2.08%, sys=6.62%, ctx=1793721, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1788109/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=6984.9MB, aggrb=119201KB/s, minb=119201KB/s, maxb=119201KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  sda: ios=1785701/10, merge=0/6, ticks=352024/0, in_queue=351460, util=99.73%

Direct write
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
...
fio-2.8
Starting 8 processes
benchmark: Laying out IO file(s) (1 file(s) / 256MB)

benchmark: (groupid=0, jobs=8): err= 0: pid=16523: Wed Dec  4 18:26:45 2019
  write: io=1400.5MB, bw=23899KB/s, iops=5974, runt= 60003msec
    clat (usec): min=60, max=100206, avg=1331.72, stdev=6337.35
     lat (usec): min=61, max=100207, avg=1332.97, stdev=6337.58
    clat percentiles (usec):
     |  1.00th=[   72],  5.00th=[   78], 10.00th=[   83], 20.00th=[   92],
     | 30.00th=[  102], 40.00th=[  116], 50.00th=[  133], 60.00th=[  151],
     | 70.00th=[  175], 80.00th=[  211], 90.00th=[  302], 95.00th=[  466],
     | 99.00th=[36608], 99.50th=[39168], 99.90th=[47360], 99.95th=[54528],
     | 99.99th=[72192]
    bw (KB  /s): min=  809, max= 4445, per=12.50%, avg=2986.79, stdev=509.90
    lat (usec) : 100=27.73%, 250=58.01%, 500=9.62%, 750=0.92%, 1000=0.13%
    lat (msec) : 2=0.09%, 4=0.04%, 10=0.03%, 20=0.01%, 50=3.36%
    lat (msec) : 100=0.07%, 250=0.01%
  cpu          : usr=0.71%, sys=2.72%, ctx=729250, majf=0, minf=81
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=358505/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=1400.5MB, aggrb=23899KB/s, minb=23899KB/s, maxb=23899KB/s, mint=60003msec, maxt=60003msec

Disk stats (read/write):
  sda: ios=138/358207, merge=0/2164, ticks=168/47172, in_queue=47212, util=78.36%


Speedtest (Германия, Фалькенштейн)
Retrieving speedtest.net configuration...
Testing from Hetzner Online GmbH (78.47.44.61)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by noris network AG (Nuremberg) [8.42 km]: 4.678 ms
Testing download speed................................................................................
Download: 1188.20 Mbit/s
Testing upload speed................................................................................................
Upload: 1315.40 Mbit/s


Speedtest (Германия, Нюрнберг)
Retrieving speedtest.net configuration...
Testing from Hetzner Online GmbH (88.99.185.76)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Gemeindewerke Numbrecht GmbH (Numbrecht) [143.00 km]: 10.404 ms
Testing download speed................................................................................
Download: 781.57 Mbit/s
Testing upload speed................................................................................................
Upload: 487.99 Mbit/s


Speedtest (Финляндия, Хельсинки)
Retrieving speedtest.net configuration...
Testing from Hetzner Online GmbH (95.217.11.133)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Cumulus Technologies (Helsinki) [0.18 km]: 1.283 ms
Testing download speed................................................................................
Download: 666.33 Mbit/s
Testing upload speed................................................................................................
Upload: 669.90 Mbit/s



Результаты (картинки кликабельны):


Unixbench (попугаи)




Скорость чтения/записи dd (MB/s)


1M



64K



Количество IOPS в разных режимах работы fio


Random read



Random write



Random direct read



Random direct write



Скорость сети от speedtest (Mbit/s)



Синий — download, красный — upload.

Как мне показалось, наиболее стабильные результаты показали Linode и Vultr. Hetzner немного отстает, но он и относится к несколько другой ценовой категории. DigitalOcean же в некоторых тестах (в частности Unixbench) уступил остальным, хотя в целом этот хостер мне очень нравится.

Комментарии (12)


  1. Arech
    18.12.2019 16:40
    +2

    Я так понимаю, что локальное время сервера напрочь игнорировалось во время тестов?
    Поясню: люди часто выбирают датацентр хостера поближе к локации своих клиентов. Поэтому когда в Америке день, американцы флудят свои сервера и они крепко нагружены. В это же время в Азии — ночь, в Европе поздний вечер, нагрузка минимальна. И наоборот. Таким образом не учитывая лок.время получаем сравнение тёплого с мягким, которое ничего не стоит.


    1. edo1h
      18.12.2019 19:27

      в любом случае одиночный тест ни о чём не говорит, для объективности нужна серия тестов (локальное время — далеко не единственная причина изменения производительности)


  1. BorisTheAnimal
    18.12.2019 18:01

    Можно еще в тесты добавить амазоновский lightsail на будущее.


  1. hololoev
    18.12.2019 19:22
    +2

    Из личного опыта:
    Hetzner — просто непрерывные лаги в дц
    DO — с ними со старта, зажали безлимитный трафик, который обещали всем, кто с ними со старта. Но в целом всё ок
    Linode — +- то же что и ДО, только входящий канал до 40Гб/с. Исходящий фильтруется, нельзя юзать хпинг, в отличии от ДО. Но у ДО входящий 1Гб/с


  1. gecube
    18.12.2019 22:46
    -1

    Выводы какие? Тест сам сделан как-то небрежно. Методика так себе. Не говоря уже о том, что есть не отраженные в методике параметры — качество поддержки, лаг по сети (толку от быстрых дисков, если сеть медленная?) и многое другое.
    Я уж не говорю о том, что не у всех провайдеров "одинаковые" виртуалки матчатся друг в друга. По разным причинам. И мне кажется, что самое надёжное — это все приводить к деньгам.
    А ещё мне очень нравится scaleway / ovh


  1. GomboTs
    18.12.2019 23:32

    Когда мне потребовался небольшой сервер для личных проектов, DO привлек отличными обучающими материалами; но после тестового периода пришлось их послать нафиг, когда они для $5 сервера попросили скан паспорта.
    Ушел к Vultr, нареканий нет. К DO заглядываю за инструкциями :D


    1. denaspireone
      19.12.2019 09:38
      +1

      Это у вас наверное разница в ФИО аккаунта и владельцем карты? Т.к. у меня ничего такого не спрашивали за 4 года ни разу.


  1. arheops
    19.12.2019 00:11

    То, что тесты Vultr(заявлены и реально используются одинаковые хосты на всех локациях) показывает разные значения, говорит о спорности методики.
    У меня Vultr выдает практически одинаковые значения тестов для одинаковых типов инстансов, если инстанс имеет больше 8гб памяти. На малых полная чехарда.


    1. Tolmy
      19.12.2019 17:50

      Потому что машины под дешевые тарифные планы перегружены больше всех, поэтому есть большой шанс попасть на физический сервер с «неудобными соседями». Чем дороже тарифный план, то, как правило, под всё больший запас берут VDS, и поэтому меньше вероятность нарваться на перегруженный сервер. Хотя я несколько раз нарывался и на $40/month инстансах. Проще всего занять ещё один инстанс, протестить его за неделю, и если всё ок, переехать на него.


      1. arheops
        19.12.2019 17:59

        Я то как раз понимаю, но в статье то самые дешевые.


  1. multed
    19.12.2019 17:42

    Раньше как-то раньше делал у себя в бложеке тесты vps от разных хостеров.
    Но в итоге цифры сильно зависят от соседей, времени суток, и других вещей.
    Цифры какие-то показывает, но гарантии, что на твоём хост-сервере не поселится шумный сосед нет никакой.
    Или ты сам можешь стать этим шумным соседом.
    Или ты делаешь тесты, пока сосед шумит.


    1. Tolmy
      19.12.2019 17:52

      Или ты делаешь тесты вместе с соседом. А все остальные на этом же физическом сервере в это время пищат, «что за фигня?». Azure в этом отношении честнее — там всем всё изначально зарезано и ограничено, и если пообещали 200 IOPS, то они будут. Хоть это и печально обычно, и не вызывает ничего, кроме раздражения.