Raspberry pi 4b direct USB boot

All Raspberry pi models up to 4 required microSD because the microSD had a bootloader. Raspberry Pi 4 has 512KB EEPROM (Electrically Erasable Programmable Read-Only Memory). Thus, the bootloader from microSD moved to EEPROM, making it possible to boot directly from a USB drive or even over a network without a microSD. This post describes the installation process for Ubuntu 20.04 on an SSD connected via USB 3.0.

Several of these Raspberry pis are used as control nodes for my home cluster. As shown from the tests given at the end of this post, an SSD-drive connected to raspberry pi via USB 3.0 provides much higher performance than a microSD (expectedly).

But first, we still need a microSD card with a Raspberry Pi OS. The support of direct USB boot was added to the EEPROM firmware recently. For modules that I have, an update was required.

After booting into Raspberry Pi OS, we have to update all packages and switch the EEPROM update channel from critical to stable:

apt update
apt full-upgrade
sed -i 's/^FIRMWARE_RELEASE_STATUS=.*/FIRMWARE_RELEASE_STATUS="stable"/' /etc/default/rpi-eeprom-update

Then we will update the EEPROM and reboot:

rpi-eeprom-update -a -d
BCM2711 detected
Dedicated VL805 EEPROM detected
BOOTFS /boot
*** INSTALLING EEPROM UPDATES ***
BOOTLOADER: update available
CURRENT: Tue 10 Sep 2019 10:41:50 AM UTC (1568112110)
 LATEST: Thu 03 Sep 2020 12:11:43 PM UTC (1599135103)
 FW DIR: /lib/firmware/raspberrypi/bootloader/stable
VL805: update available
CURRENT: 000137ad
 LATEST: 000138a1
BOOTFS /boot
EEPROM updates pending. Please reboot to apply the update.

reboot

Checking the config after the update:

vcgencmd bootloader_config
[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
ENABLE_SELF_UPDATE=1
DISABLE_HDMI=0
BOOT_ORDER=0xf41

It’s time to install Ubuntu 20.04 on SSD:

wget https://cdimage.ubuntu.com/releases/20.04.1/release/ubuntu-20.04.1-preinstalled-server-arm64+raspi.img.xz
xz -d < ubuntu-20.04.1-preinstalled-server-arm64+raspi.img.xz - | dd of=/dev/sda bs=1M status=progress
partprobe

mkdir /mnt/root
mount /dev/sda2 /mnt/root
mount /dev/sda1 /mnt/root/boot/firmware

Most of the further actions are borrowed from the official forum raspberrypi.org.

Unfortunately, direct USB boot does not work with a compressed kernel, so we need to unpack it:

cd /mnt/root/boot/firmware
zcat vmlinuz > vmlinux

And also we make the section of config.txt to:

[pi4]
gpu_mem=16
max_framebuffers=2
dtoverlay=vc4-fkms-v3d
boot_delay
kernel=vmlinux
initramfs initrd.img followkernel

Automating kernel unpacking when updating with script auto_decompress_kernel:

cat <<EOF > auto_decompress_kernel
#!/bin/bash -e

#Set Variables
BTPATH=/boot/firmware
CKPATH=$BTPATH/vmlinuz
DKPATH=$BTPATH/vmlinux

#Check if compression needs to be done.
if [ -e $BTPATH/check.md5 ]; then
	if md5sum --status --ignore-missing -c $BTPATH/check.md5; then
	echo -e "\e[32mFiles have not changed, Decompression not needed\e[0m"
	exit 0
	else echo -e "\e[31mHash failed, kernel will be compressed\e[0m"
	fi
fi

#Backup the old decompressed kernel
mv $DKPATH $DKPATH.bak

if [ ! $? == 0 ]; then
	echo -e "\e[31mDECOMPRESSED KERNEL BACKUP FAILED!\e[0m"
	exit 1
else 	echo -e "\e[32mDecompressed kernel backup was successful\e[0m"
fi

#Decompress the new kernel
echo "Decompressing kernel: "$CKPATH".............."

zcat $CKPATH > $DKPATH

if [ ! $? == 0 ]; then
	echo -e "\e[31mKERNEL FAILED TO DECOMPRESS!\e[0m"
	exit 1
else
        echo -e "\e[32mKernel Decompressed Succesfully\e[0m"
fi

#Hash the new kernel for checking
md5sum $CKPATH $DKPATH > $BTPATH/check.md5

if [ ! $? == 0 ]; then
	echo -e "\e[31mMD5 GENERATION FAILED!\e[0m"
	else echo -e "\e[32mMD5 generated Succesfully\e[0m"
fi

#Exit
exit 0
EOF

Run this script for every package manipulation:

cat <<EOF > /mnt/root/etc/apt/apt.conf.d/999_decompress_rpi_kernel
DPkg::Post-Invoke {"/bin/bash /boot/firmware/auto_decompress_kernel"; };
EOF
chmod +x /mnt/root/etc/apt/apt.conf.d/999_decompress_rpi_kernel

After this, turn off the system and take out the SD card. We will never need this again. Further, the OS will be loaded from USB.

SSD vs microSD performance

What was it all for? If you've read this far, you may like to know how what performance will take SSD on raspberry pi compares to a top-end microSD card. The results that I got are shown in the summary table below.

Test configuration

The following components were used during testing:

Test results

  IOPS BW, MiB/s lat, ms
sequential reads
microSD 10 43.3 2935
SSD 94 379 336
random reads
microSD 1875 7.5 0.5
SSD 5185 20.3 0.2
sequential writes
microSD 6 25.5 4979.09
SSD 69 277 336
random writes
microSD 163 0.6 6.1
SSD 551 2.2 1.8

Raw test results

SSD

fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=read -runtime=60 -filename=/dev/sda
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=380MiB/s][r=95 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1168: Wed Sep 16 20:24:17 2020
  read: IOPS=94, BW=379MiB/s (397MB/s)(22.2GiB/60078msec)
    slat (usec): min=396, max=22200, avg=10518.14, stdev=3092.86
    clat (msec): min=69, max=445, avg=326.21, stdev=11.54
     lat (msec): min=78, max=457, avg=336.73, stdev=11.57
    clat percentiles (msec):
     |  1.00th=[  321],  5.00th=[  321], 10.00th=[  321], 20.00th=[  321],
     | 30.00th=[  321], 40.00th=[  326], 50.00th=[  326], 60.00th=[  330],
     | 70.00th=[  334], 80.00th=[  334], 90.00th=[  334], 95.00th=[  334],
     | 99.00th=[  334], 99.50th=[  338], 99.90th=[  393], 99.95th=[  426],
     | 99.99th=[  447]
   bw (  KiB/s): min=81920, max=401408, per=99.54%, avg=386013.07, stdev=28372.15, samples=120
   iops        : min=   20, max=   98, avg=94.13, stdev= 6.93, samples=120
  lat (msec)   : 100=0.05%, 250=0.25%, 500=99.70%
  cpu          : usr=0.26%, sys=5.84%, ctx=7075, majf=0, minf=32791
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=99.5%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=5688,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=379MiB/s (397MB/s), 379MiB/s-379MiB/s (397MB/s-397MB/s), io=22.2GiB (23.9GB), run=60078-60078msec

Disk stats (read/write):
  sda: ios=45355/0, merge=0/0, ticks=3252406/0, in_queue=3033920, util=99.87%

fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -filename=/dev/sda
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=108MiB/s][w=27 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1155: Wed Sep 16 20:20:57 2020
  write: IOPS=69, BW=277MiB/s (291MB/s)(16.3GiB/60277msec); 0 zone resets
    slat (usec): min=1484, max=77985, avg=14324.74, stdev=7642.49
    clat (msec): min=271, max=1390, avg=446.22, stdev=199.27
     lat (msec): min=305, max=1422, avg=460.55, stdev=205.50
    clat percentiles (msec):
     |  1.00th=[  376],  5.00th=[  380], 10.00th=[  384], 20.00th=[  384],
     | 30.00th=[  388], 40.00th=[  388], 50.00th=[  393], 60.00th=[  393],
     | 70.00th=[  397], 80.00th=[  397], 90.00th=[  405], 95.00th=[ 1167],
     | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1234], 99.95th=[ 1318],
     | 99.99th=[ 1385]
   bw (  KiB/s): min=98304, max=335872, per=99.67%, avg=283024.26, stdev=85726.24, samples=120
   iops        : min=   24, max=   82, avg=68.99, stdev=20.90, samples=120
  lat (msec)   : 500=92.56%, 750=0.36%, 1000=0.43%
  cpu          : usr=5.45%, sys=3.32%, ctx=7301, majf=0, minf=25
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.3%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4179,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=277MiB/s (291MB/s), 277MiB/s-277MiB/s (291MB/s-291MB/s), io=16.3GiB (17.5GB), run=60277-60277msec

Disk stats (read/write):
  sda: ios=117/33425, merge=0/0, ticks=76/3276312, in_queue=3106450, util=99.43%

fio -ioengine=libaio -sync=1 -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=1 -rw=randread -runtime=60 -filename=/dev/sda
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=22.1MiB/s][r=5667 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1148: Wed Sep 16 20:18:19 2020
  read: IOPS=5185, BW=20.3MiB/s (21.2MB/s)(1215MiB/60001msec)
    slat (usec): min=24, max=134, avg=29.69, stdev= 5.88
    clat (usec): min=13, max=27815, avg=155.23, stdev=72.73
     lat (usec): min=120, max=27982, avg=186.02, stdev=75.01
    clat percentiles (usec):
     |  1.00th=[   97],  5.00th=[   98], 10.00th=[   99], 20.00th=[  112],
     | 30.00th=[  143], 40.00th=[  145], 50.00th=[  147], 60.00th=[  153],
     | 70.00th=[  159], 80.00th=[  167], 90.00th=[  221], 95.00th=[  289],
     | 99.00th=[  343], 99.50th=[  351], 99.90th=[  383], 99.95th=[  388],
     | 99.99th=[  396]
   bw (  KiB/s): min=12808, max=23208, per=99.90%, avg=20719.71, stdev=3476.07, samples=119
   iops        : min= 3202, max= 5802, avg=5179.91, stdev=869.02, samples=119
  lat (usec)   : 20=0.01%, 50=0.01%, 100=11.41%, 250=80.32%, 500=8.27%
  lat (usec)   : 750=0.01%
  lat (msec)   : 2=0.01%, 50=0.01%
  cpu          : usr=4.18%, sys=31.36%, ctx=311343, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=311124,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=20.3MiB/s (21.2MB/s), 20.3MiB/s-20.3MiB/s (21.2MB/s-21.2MB/s), io=1215MiB (1274MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=310369/0, merge=0/0, ticks=48674/0, in_queue=20, util=99.84%

fio -ioengine=libaio -sync=1 -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=60 -filename=/dev/sda
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=2192KiB/s][w=548 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1124: Wed Sep 16 20:16:02 2020
  write: IOPS=551, BW=2207KiB/s (2260kB/s)(129MiB/60001msec); 0 zone resets
    slat (usec): min=39, max=678, avg=81.91, stdev= 7.52
    clat (usec): min=1061, max=10623, avg=1713.08, stdev=196.50
     lat (usec): min=1639, max=10709, avg=1799.11, stdev=196.79
    clat percentiles (usec):
     |  1.00th=[ 1631],  5.00th=[ 1647], 10.00th=[ 1647], 20.00th=[ 1647],
     | 30.00th=[ 1647], 40.00th=[ 1663], 50.00th=[ 1663], 60.00th=[ 1663],
     | 70.00th=[ 1663], 80.00th=[ 1680], 90.00th=[ 2057], 95.00th=[ 2057],
     | 99.00th=[ 2212], 99.50th=[ 2474], 99.90th=[ 2868], 99.95th=[ 6128],
     | 99.99th=[ 6456]
   bw (  KiB/s): min= 2160, max= 2320, per=99.98%, avg=2206.55, stdev=19.14, samples=120
   iops        : min=  540, max=  580, avg=551.50, stdev= 4.81, samples=120
  lat (msec)   : 2=88.03%, 4=11.89%, 10=0.08%, 20=0.01%
  cpu          : usr=1.03%, sys=4.56%, ctx=66286, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,33112,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2207KiB/s (2260kB/s), 2207KiB/s-2207KiB/s (2260kB/s-2260kB/s), io=129MiB (136MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=116/33012, merge=0/0, ticks=48/57647, in_queue=10, util=99.97%

microSD

fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=read -runtime=60 -filename=/dev/mmcblk0
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [R(1)][4.2%][r=16.0MiB/s][r=4 IOPS][eta 23m:03s]
test: (groupid=0, jobs=1): err= 0: pid=2120: Wed Sep 16 18:20:31 2020
  read: IOPS=10, BW=43.3MiB/s (45.4MB/s)(2660MiB/61469msec)
    slat (msec): min=5, max=131, avg=90.24, stdev=13.51
    clat (msec): min=1441, max=4215, avg=2845.52, stdev=214.83
     lat (msec): min=1532, max=4308, avg=2935.76, stdev=221.38
    clat percentiles (msec):
     |  1.00th=[ 1737],  5.00th=[ 2802], 10.00th=[ 2836], 20.00th=[ 2836],
     | 30.00th=[ 2869], 40.00th=[ 2869], 50.00th=[ 2869], 60.00th=[ 2869],
     | 70.00th=[ 2869], 80.00th=[ 2869], 90.00th=[ 2903], 95.00th=[ 2903],
     | 99.00th=[ 3675], 99.50th=[ 3943], 99.90th=[ 4212], 99.95th=[ 4212],
     | 99.99th=[ 4212]
   bw (  KiB/s): min= 8192, max=49152, per=99.32%, avg=44010.88, stdev=5224.40, samples=118
   iops        : min=    2, max=   12, avg=10.70, stdev= 1.29, samples=118
  lat (msec)   : 2000=1.95%, >=2000=98.05%
  cpu          : usr=0.02%, sys=0.62%, ctx=772, majf=0, minf=32791
  IO depths    : 1=0.2%, 2=0.3%, 4=0.6%, 8=1.2%, 16=2.4%, 32=95.3%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=665,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=43.3MiB/s (45.4MB/s), 43.3MiB/s-43.3MiB/s (45.4MB/s-45.4MB/s), io=2660MiB (2789MB), run=61469-61469msec

Disk stats (read/write):
  mmcblk0: ios=5319/0, merge=0/0, ticks=7555533/0, in_queue=7544900, util=35.06%

fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -filename=/dev/mmcblk0
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [W(1)][2.4%][eta 42m:01s]
test: (groupid=0, jobs=1): err= 0: pid=2145: Wed Sep 16 18:22:57 2020
  write: IOPS=6, BW=25.5MiB/s (26.7MB/s)(1588MiB/62279msec); 0 zone resets
    slat (usec): min=499, max=448407, avg=151333.36, stdev=67559.61
    clat (msec): min=2174, max=7178, avg=4827.75, stdev=1185.82
     lat (msec): min=2177, max=7445, avg=4979.09, stdev=1227.35
    clat percentiles (msec):
     |  1.00th=[ 2366],  5.00th=[ 3373], 10.00th=[ 3910], 20.00th=[ 4044],
     | 30.00th=[ 4111], 40.00th=[ 4245], 50.00th=[ 4279], 60.00th=[ 4463],
     | 70.00th=[ 5269], 80.00th=[ 6409], 90.00th=[ 6745], 95.00th=[ 6946],
     | 99.00th=[ 7148], 99.50th=[ 7148], 99.90th=[ 7148], 99.95th=[ 7148],
     | 99.99th=[ 7148]
   bw (  KiB/s): min= 8192, max=40960, per=98.13%, avg=25622.02, stdev=9590.32, samples=117
   iops        : min=    2, max=   10, avg= 6.19, stdev= 2.32, samples=117
  lat (msec)   : >=2000=100.00%
  cpu          : usr=0.51%, sys=0.17%, ctx=497, majf=0, minf=22
  IO depths    : 1=0.3%, 2=0.5%, 4=1.0%, 8=2.0%, 16=4.0%, 32=92.2%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.0%, 32=0.3%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,397,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=25.5MiB/s (26.7MB/s), 25.5MiB/s-25.5MiB/s (26.7MB/s-26.7MB/s), io=1588MiB (1665MB), run=62279-62279msec

Disk stats (read/write):
  mmcblk0: ios=0/3185, merge=0/0, ticks=0/7594168, in_queue=7587800, util=21.04%

fio -ioengine=libaio -sync=1 -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=1 -rw=randread -runtime=60 -filename=/dev/mmcblk0
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=7784KiB/s][r=1946 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2168: Wed Sep 16 18:24:21 2020
  read: IOPS=1875, BW=7500KiB/s (7680kB/s)(439MiB/60001msec)
    slat (usec): min=30, max=173, avg=38.10, stdev= 1.42
    clat (usec): min=212, max=51870, avg=492.15, stdev=723.30
     lat (usec): min=251, max=51909, avg=530.71, stdev=723.32
    clat percentiles (usec):
     |  1.00th=[  420],  5.00th=[  424], 10.00th=[  424], 20.00th=[  424],
     | 30.00th=[  429], 40.00th=[  429], 50.00th=[  429], 60.00th=[  429],
     | 70.00th=[  429], 80.00th=[  429], 90.00th=[  685], 95.00th=[  824],
     | 99.00th=[  840], 99.50th=[  848], 99.90th=[ 5473], 99.95th=[19268],
     | 99.99th=[36439]
   bw (  KiB/s): min= 4696, max= 7944, per=99.96%, avg=7497.35, stdev=661.73, samples=119
   iops        : min= 1174, max= 1986, avg=1874.34, stdev=165.43, samples=119
  lat (usec)   : 250=0.01%, 500=87.07%, 750=6.37%, 1000=6.36%
  lat (msec)   : 2=0.02%, 4=0.03%, 10=0.06%, 20=0.04%, 50=0.04%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.28%, sys=5.51%, ctx=225012, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=112502,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=7500KiB/s (7680kB/s), 7500KiB/s-7500KiB/s (7680kB/s-7680kB/s), io=439MiB (461MB), run=60001-60001msec

Disk stats (read/write):
  mmcblk0: ios=112481/0, merge=0/0, ticks=57454/0, in_queue=2180, util=96.77%

fio -ioengine=libaio -sync=1 -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=60 -filename=/dev/mmcblk0
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=804KiB/s][w=201 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2176: Wed Sep 16 18:25:56 2020
  write: IOPS=163, BW=653KiB/s (669kB/s)(38.3MiB/60001msec); 0 zone resets
    slat (usec): min=39, max=160, avg=40.88, stdev= 2.31
    clat (usec): min=1391, max=137877, avg=6078.68, stdev=9078.56
     lat (usec): min=1431, max=137918, avg=6120.11, stdev=9078.57
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    5], 60.00th=[    7],
     | 70.00th=[    7], 80.00th=[    7], 90.00th=[    7], 95.00th=[    7],
     | 99.00th=[   13], 99.50th=[  112], 99.90th=[  125], 99.95th=[  126],
     | 99.99th=[  138]
   bw (  KiB/s): min=  456, max=  928, per=99.96%, avg=652.76, stdev=122.86, samples=119
   iops        : min=  114, max=  232, avg=163.18, stdev=30.73, samples=119
  lat (msec)   : 2=0.01%, 4=6.67%, 10=92.25%, 20=0.26%, 50=0.14%
  lat (msec)   : 100=0.05%, 250=0.61%
  cpu          : usr=0.14%, sys=0.48%, ctx=19598, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,9799,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=653KiB/s (669kB/s), 653KiB/s-653KiB/s (669kB/s-669kB/s), io=38.3MiB (40.1MB), run=60001-60001msec

Disk stats (read/write):
  mmcblk0: ios=0/9795, merge=0/0, ticks=0/59750, in_queue=44232, util=65.49%

Leave a comment

Your email address will not be published. Required fields are marked *

Spelling error report

The following text will be sent to our editors: