Все модели Raspberry pi до 4B обязательно требовали microSD, поскольку именно на ней располагался загрузчик. Raspberry Pi 4 же имеет EEPROM (Electrically Erasable Programmable Read-Only Memory) объемом 512KB. Таким образом, загрузчик с microSD переехал в EEPROM, что сделало возможным загрузку непосредственно с USB-накопителя или даже по сети без microSD. В данной заметке описан процесс установки Ubuntu 20.04 на SSD, подключенный через USB 3.0.

Несколько таких Raspberry pi используются в качестве управляющих узлов моего домашнего кластера. Как можно будет убедится из тестов приведенных в конце записи, SSD-диск подключенный к raspberry pi посредством USB 3.0 обеспечивает гораздо большую производительность, чем microSD (ожидаемо).

Но для начала, нам все-таки нужна карта microSD с Raspberry Pi OS. Дело в том, что поддержку загрузки непосредственно с USB-накопителя добавили в прошивку EEPROM относительно недавно. Для тех модулей, которые есть у меня, потребовалось обновление.

Итак, после загрузки в Raspberry Pi OS обновляем все пакеты и переключаем канал обновления EEPROM с критичного на стабильный.

apt update
apt full-upgrade
sed -i 's/^FIRMWARE_RELEASE_STATUS=.*/FIRMWARE_RELEASE_STATUS="stable"/' /etc/default/rpi-eeprom-update

После чего обновляем EEPROM и перезагружаемся:

rpi-eeprom-update -a -d
BCM2711 detected
Dedicated VL805 EEPROM detected
BOOTFS /boot
*** INSTALLING EEPROM UPDATES ***
BOOTLOADER: update available
CURRENT: Tue 10 Sep 2019 10:41:50 AM UTC (1568112110)
 LATEST: Thu 03 Sep 2020 12:11:43 PM UTC (1599135103)
 FW DIR: /lib/firmware/raspberrypi/bootloader/stable
VL805: update available
CURRENT: 000137ad
 LATEST: 000138a1
BOOTFS /boot
EEPROM updates pending. Please reboot to apply the update.

reboot

Проверяем конфиг после обновления:

vcgencmd bootloader_config
[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
ENABLE_SELF_UPDATE=1
DISABLE_HDMI=0
BOOT_ORDER=0xf41

Пора установить Ubuntu 20.04 на SSD:

wget https://cdimage.ubuntu.com/releases/20.04.1/release/ubuntu-20.04.1-preinstalled-server-arm64+raspi.img.xz
xz -d < ubuntu-20.04.1-preinstalled-server-arm64+raspi.img.xz - | dd of=/dev/sda bs=1M status=progress
partprobe

mkdir /mnt/root
mount /dev/sda2 /mnt/root
mount /dev/sda1 /mnt/root/boot/firmware

Большая часть дальнейших действий позаимствована на официальном форуме raspberrypi.org.

К сожалению, данная схема не работает со сжатым ядром, поэтому нам необходимо его распаковать:

cd /mnt/root/boot/firmware
zcat vmlinuz > vmlinux

А так же приводим секцию config.txt к виду:

[pi4]
gpu_mem=16
max_framebuffers=2
dtoverlay=vc4-fkms-v3d
boot_delay
kernel=vmlinux
initramfs initrd.img followkernel

Автоматизируем распаковку ядра при обновлении с помощью auto_decompress_kernel:

cat <<EOF > auto_decompress_kernel
#!/bin/bash -e

#Set Variables
BTPATH=/boot/firmware
CKPATH=$BTPATH/vmlinuz
DKPATH=$BTPATH/vmlinux

#Check if compression needs to be done.
if [ -e $BTPATH/check.md5 ]; then
	if md5sum --status --ignore-missing -c $BTPATH/check.md5; then
	echo -e "\e[32mFiles have not changed, Decompression not needed\e[0m"
	exit 0
	else echo -e "\e[31mHash failed, kernel will be compressed\e[0m"
	fi
fi

#Backup the old decompressed kernel
mv $DKPATH $DKPATH.bak

if [ ! $? == 0 ]; then
	echo -e "\e[31mDECOMPRESSED KERNEL BACKUP FAILED!\e[0m"
	exit 1
else 	echo -e "\e[32mDecompressed kernel backup was successful\e[0m"
fi

#Decompress the new kernel
echo "Decompressing kernel: "$CKPATH".............."

zcat $CKPATH > $DKPATH

if [ ! $? == 0 ]; then
	echo -e "\e[31mKERNEL FAILED TO DECOMPRESS!\e[0m"
	exit 1
else
        echo -e "\e[32mKernel Decompressed Succesfully\e[0m"
fi

#Hash the new kernel for checking
md5sum $CKPATH $DKPATH > $BTPATH/check.md5

if [ ! $? == 0 ]; then
	echo -e "\e[31mMD5 GENERATION FAILED!\e[0m"
	else echo -e "\e[32mMD5 generated Succesfully\e[0m"
fi

#Exit
exit 0
EOF

Запускаем скрипт при каждой манипуляции с пакетами:

cat <<EOF > /mnt/root/etc/apt/apt.conf.d/999_decompress_rpi_kernel
DPkg::Post-Invoke {"/bin/bash /boot/firmware/auto_decompress_kernel"; };
EOF
chmod +x /mnt/root/etc/apt/apt.conf.d/999_decompress_rpi_kernel

На этом все. Выключаем систему и вынимаем SD-карту, больше она нам не понадобится. Далее ОС уже будет грузится с SSD.

Тест SSD vs microSD

Ради чего все это было? Если вы дочитали до этого места, значит вам интересно, какой выигрыш в производительности можно получить в сравнении с топовой microSD. Результаты, которые получились у меня, приведены в сводной таблице ниже.

Тестовая конфигурация

При тестировании использовалась следующая конфигурация:

• Raspberry Pi 4 model B 4GB
• GeeekPi Raspberry X857 mSATA SSD Shield
• Kingspec mSATA SSD 128 ГБ
• SanDisk Extreme Pro 64G microSD

Результаты тестов

	IOPS	BW, MiB/s	lat, ms
линейное чтение
microSD	10	43.3	2935
SSD	94	379	336
случайное чтение
microSD	1875	7.5	0.5
SSD	5185	20.3	0.2
линейная запись
microSD	6	25.5	4979.09
SSD	69	277	336
случайная запись
microSD	163	0.6	6.1
SSD	551	2.2	1.8

Сырые результаты тестов

SSD

fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=read -runtime=60 -filename=/dev/sda
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=380MiB/s][r=95 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1168: Wed Sep 16 20:24:17 2020
  read: IOPS=94, BW=379MiB/s (397MB/s)(22.2GiB/60078msec)
    slat (usec): min=396, max=22200, avg=10518.14, stdev=3092.86
    clat (msec): min=69, max=445, avg=326.21, stdev=11.54
     lat (msec): min=78, max=457, avg=336.73, stdev=11.57
    clat percentiles (msec):
     |  1.00th=[  321],  5.00th=[  321], 10.00th=[  321], 20.00th=[  321],
     | 30.00th=[  321], 40.00th=[  326], 50.00th=[  326], 60.00th=[  330],
     | 70.00th=[  334], 80.00th=[  334], 90.00th=[  334], 95.00th=[  334],
     | 99.00th=[  334], 99.50th=[  338], 99.90th=[  393], 99.95th=[  426],
     | 99.99th=[  447]
   bw (  KiB/s): min=81920, max=401408, per=99.54%, avg=386013.07, stdev=28372.15, samples=120
   iops        : min=   20, max=   98, avg=94.13, stdev= 6.93, samples=120
  lat (msec)   : 100=0.05%, 250=0.25%, 500=99.70%
  cpu          : usr=0.26%, sys=5.84%, ctx=7075, majf=0, minf=32791
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=99.5%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=5688,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=379MiB/s (397MB/s), 379MiB/s-379MiB/s (397MB/s-397MB/s), io=22.2GiB (23.9GB), run=60078-60078msec

Disk stats (read/write):
  sda: ios=45355/0, merge=0/0, ticks=3252406/0, in_queue=3033920, util=99.87%

fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -filename=/dev/sda
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=108MiB/s][w=27 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1155: Wed Sep 16 20:20:57 2020
  write: IOPS=69, BW=277MiB/s (291MB/s)(16.3GiB/60277msec); 0 zone resets
    slat (usec): min=1484, max=77985, avg=14324.74, stdev=7642.49
    clat (msec): min=271, max=1390, avg=446.22, stdev=199.27
     lat (msec): min=305, max=1422, avg=460.55, stdev=205.50
    clat percentiles (msec):
     |  1.00th=[  376],  5.00th=[  380], 10.00th=[  384], 20.00th=[  384],
     | 30.00th=[  388], 40.00th=[  388], 50.00th=[  393], 60.00th=[  393],
     | 70.00th=[  397], 80.00th=[  397], 90.00th=[  405], 95.00th=[ 1167],
     | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1234], 99.95th=[ 1318],
     | 99.99th=[ 1385]
   bw (  KiB/s): min=98304, max=335872, per=99.67%, avg=283024.26, stdev=85726.24, samples=120
   iops        : min=   24, max=   82, avg=68.99, stdev=20.90, samples=120
  lat (msec)   : 500=92.56%, 750=0.36%, 1000=0.43%
  cpu          : usr=5.45%, sys=3.32%, ctx=7301, majf=0, minf=25
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.3%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4179,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=277MiB/s (291MB/s), 277MiB/s-277MiB/s (291MB/s-291MB/s), io=16.3GiB (17.5GB), run=60277-60277msec

Disk stats (read/write):
  sda: ios=117/33425, merge=0/0, ticks=76/3276312, in_queue=3106450, util=99.43%

fio -ioengine=libaio -sync=1 -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=1 -rw=randread -runtime=60 -filename=/dev/sda
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=22.1MiB/s][r=5667 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1148: Wed Sep 16 20:18:19 2020
  read: IOPS=5185, BW=20.3MiB/s (21.2MB/s)(1215MiB/60001msec)
    slat (usec): min=24, max=134, avg=29.69, stdev= 5.88
    clat (usec): min=13, max=27815, avg=155.23, stdev=72.73
     lat (usec): min=120, max=27982, avg=186.02, stdev=75.01
    clat percentiles (usec):
     |  1.00th=[   97],  5.00th=[   98], 10.00th=[   99], 20.00th=[  112],
     | 30.00th=[  143], 40.00th=[  145], 50.00th=[  147], 60.00th=[  153],
     | 70.00th=[  159], 80.00th=[  167], 90.00th=[  221], 95.00th=[  289],
     | 99.00th=[  343], 99.50th=[  351], 99.90th=[  383], 99.95th=[  388],
     | 99.99th=[  396]
   bw (  KiB/s): min=12808, max=23208, per=99.90%, avg=20719.71, stdev=3476.07, samples=119
   iops        : min= 3202, max= 5802, avg=5179.91, stdev=869.02, samples=119
  lat (usec)   : 20=0.01%, 50=0.01%, 100=11.41%, 250=80.32%, 500=8.27%
  lat (usec)   : 750=0.01%
  lat (msec)   : 2=0.01%, 50=0.01%
  cpu          : usr=4.18%, sys=31.36%, ctx=311343, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=311124,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=20.3MiB/s (21.2MB/s), 20.3MiB/s-20.3MiB/s (21.2MB/s-21.2MB/s), io=1215MiB (1274MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=310369/0, merge=0/0, ticks=48674/0, in_queue=20, util=99.84%

fio -ioengine=libaio -sync=1 -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=60 -filename=/dev/sda
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=2192KiB/s][w=548 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1124: Wed Sep 16 20:16:02 2020
  write: IOPS=551, BW=2207KiB/s (2260kB/s)(129MiB/60001msec); 0 zone resets
    slat (usec): min=39, max=678, avg=81.91, stdev= 7.52
    clat (usec): min=1061, max=10623, avg=1713.08, stdev=196.50
     lat (usec): min=1639, max=10709, avg=1799.11, stdev=196.79
    clat percentiles (usec):
     |  1.00th=[ 1631],  5.00th=[ 1647], 10.00th=[ 1647], 20.00th=[ 1647],
     | 30.00th=[ 1647], 40.00th=[ 1663], 50.00th=[ 1663], 60.00th=[ 1663],
     | 70.00th=[ 1663], 80.00th=[ 1680], 90.00th=[ 2057], 95.00th=[ 2057],
     | 99.00th=[ 2212], 99.50th=[ 2474], 99.90th=[ 2868], 99.95th=[ 6128],
     | 99.99th=[ 6456]
   bw (  KiB/s): min= 2160, max= 2320, per=99.98%, avg=2206.55, stdev=19.14, samples=120
   iops        : min=  540, max=  580, avg=551.50, stdev= 4.81, samples=120
  lat (msec)   : 2=88.03%, 4=11.89%, 10=0.08%, 20=0.01%
  cpu          : usr=1.03%, sys=4.56%, ctx=66286, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,33112,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2207KiB/s (2260kB/s), 2207KiB/s-2207KiB/s (2260kB/s-2260kB/s), io=129MiB (136MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=116/33012, merge=0/0, ticks=48/57647, in_queue=10, util=99.97%

microSD

fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=read -runtime=60 -filename=/dev/mmcblk0
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [R(1)][4.2%][r=16.0MiB/s][r=4 IOPS][eta 23m:03s]
test: (groupid=0, jobs=1): err= 0: pid=2120: Wed Sep 16 18:20:31 2020
  read: IOPS=10, BW=43.3MiB/s (45.4MB/s)(2660MiB/61469msec)
    slat (msec): min=5, max=131, avg=90.24, stdev=13.51
    clat (msec): min=1441, max=4215, avg=2845.52, stdev=214.83
     lat (msec): min=1532, max=4308, avg=2935.76, stdev=221.38
    clat percentiles (msec):
     |  1.00th=[ 1737],  5.00th=[ 2802], 10.00th=[ 2836], 20.00th=[ 2836],
     | 30.00th=[ 2869], 40.00th=[ 2869], 50.00th=[ 2869], 60.00th=[ 2869],
     | 70.00th=[ 2869], 80.00th=[ 2869], 90.00th=[ 2903], 95.00th=[ 2903],
     | 99.00th=[ 3675], 99.50th=[ 3943], 99.90th=[ 4212], 99.95th=[ 4212],
     | 99.99th=[ 4212]
   bw (  KiB/s): min= 8192, max=49152, per=99.32%, avg=44010.88, stdev=5224.40, samples=118
   iops        : min=    2, max=   12, avg=10.70, stdev= 1.29, samples=118
  lat (msec)   : 2000=1.95%, >=2000=98.05%
  cpu          : usr=0.02%, sys=0.62%, ctx=772, majf=0, minf=32791
  IO depths    : 1=0.2%, 2=0.3%, 4=0.6%, 8=1.2%, 16=2.4%, 32=95.3%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=665,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=43.3MiB/s (45.4MB/s), 43.3MiB/s-43.3MiB/s (45.4MB/s-45.4MB/s), io=2660MiB (2789MB), run=61469-61469msec

Disk stats (read/write):
  mmcblk0: ios=5319/0, merge=0/0, ticks=7555533/0, in_queue=7544900, util=35.06%

fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -filename=/dev/mmcblk0
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [W(1)][2.4%][eta 42m:01s]
test: (groupid=0, jobs=1): err= 0: pid=2145: Wed Sep 16 18:22:57 2020
  write: IOPS=6, BW=25.5MiB/s (26.7MB/s)(1588MiB/62279msec); 0 zone resets
    slat (usec): min=499, max=448407, avg=151333.36, stdev=67559.61
    clat (msec): min=2174, max=7178, avg=4827.75, stdev=1185.82
     lat (msec): min=2177, max=7445, avg=4979.09, stdev=1227.35
    clat percentiles (msec):
     |  1.00th=[ 2366],  5.00th=[ 3373], 10.00th=[ 3910], 20.00th=[ 4044],
     | 30.00th=[ 4111], 40.00th=[ 4245], 50.00th=[ 4279], 60.00th=[ 4463],
     | 70.00th=[ 5269], 80.00th=[ 6409], 90.00th=[ 6745], 95.00th=[ 6946],
     | 99.00th=[ 7148], 99.50th=[ 7148], 99.90th=[ 7148], 99.95th=[ 7148],
     | 99.99th=[ 7148]
   bw (  KiB/s): min= 8192, max=40960, per=98.13%, avg=25622.02, stdev=9590.32, samples=117
   iops        : min=    2, max=   10, avg= 6.19, stdev= 2.32, samples=117
  lat (msec)   : >=2000=100.00%
  cpu          : usr=0.51%, sys=0.17%, ctx=497, majf=0, minf=22
  IO depths    : 1=0.3%, 2=0.5%, 4=1.0%, 8=2.0%, 16=4.0%, 32=92.2%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.0%, 32=0.3%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,397,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=25.5MiB/s (26.7MB/s), 25.5MiB/s-25.5MiB/s (26.7MB/s-26.7MB/s), io=1588MiB (1665MB), run=62279-62279msec

Disk stats (read/write):
  mmcblk0: ios=0/3185, merge=0/0, ticks=0/7594168, in_queue=7587800, util=21.04%

fio -ioengine=libaio -sync=1 -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=1 -rw=randread -runtime=60 -filename=/dev/mmcblk0
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=7784KiB/s][r=1946 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2168: Wed Sep 16 18:24:21 2020
  read: IOPS=1875, BW=7500KiB/s (7680kB/s)(439MiB/60001msec)
    slat (usec): min=30, max=173, avg=38.10, stdev= 1.42
    clat (usec): min=212, max=51870, avg=492.15, stdev=723.30
     lat (usec): min=251, max=51909, avg=530.71, stdev=723.32
    clat percentiles (usec):
     |  1.00th=[  420],  5.00th=[  424], 10.00th=[  424], 20.00th=[  424],
     | 30.00th=[  429], 40.00th=[  429], 50.00th=[  429], 60.00th=[  429],
     | 70.00th=[  429], 80.00th=[  429], 90.00th=[  685], 95.00th=[  824],
     | 99.00th=[  840], 99.50th=[  848], 99.90th=[ 5473], 99.95th=[19268],
     | 99.99th=[36439]
   bw (  KiB/s): min= 4696, max= 7944, per=99.96%, avg=7497.35, stdev=661.73, samples=119
   iops        : min= 1174, max= 1986, avg=1874.34, stdev=165.43, samples=119
  lat (usec)   : 250=0.01%, 500=87.07%, 750=6.37%, 1000=6.36%
  lat (msec)   : 2=0.02%, 4=0.03%, 10=0.06%, 20=0.04%, 50=0.04%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.28%, sys=5.51%, ctx=225012, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=112502,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=7500KiB/s (7680kB/s), 7500KiB/s-7500KiB/s (7680kB/s-7680kB/s), io=439MiB (461MB), run=60001-60001msec

Disk stats (read/write):
  mmcblk0: ios=112481/0, merge=0/0, ticks=57454/0, in_queue=2180, util=96.77%

fio -ioengine=libaio -sync=1 -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=60 -filename=/dev/mmcblk0
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=804KiB/s][w=201 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2176: Wed Sep 16 18:25:56 2020
  write: IOPS=163, BW=653KiB/s (669kB/s)(38.3MiB/60001msec); 0 zone resets
    slat (usec): min=39, max=160, avg=40.88, stdev= 2.31
    clat (usec): min=1391, max=137877, avg=6078.68, stdev=9078.56
     lat (usec): min=1431, max=137918, avg=6120.11, stdev=9078.57
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    5], 60.00th=[    7],
     | 70.00th=[    7], 80.00th=[    7], 90.00th=[    7], 95.00th=[    7],
     | 99.00th=[   13], 99.50th=[  112], 99.90th=[  125], 99.95th=[  126],
     | 99.99th=[  138]
   bw (  KiB/s): min=  456, max=  928, per=99.96%, avg=652.76, stdev=122.86, samples=119
   iops        : min=  114, max=  232, avg=163.18, stdev=30.73, samples=119
  lat (msec)   : 2=0.01%, 4=6.67%, 10=92.25%, 20=0.26%, 50=0.14%
  lat (msec)   : 100=0.05%, 250=0.61%
  cpu          : usr=0.14%, sys=0.48%, ctx=19598, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,9799,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=653KiB/s (669kB/s), 653KiB/s-653KiB/s (669kB/s-669kB/s), io=38.3MiB (40.1MB), run=60001-60001msec

Disk stats (read/write):
  mmcblk0: ios=0/9795, merge=0/0, ticks=0/59750, in_queue=44232, util=65.49%