Table of Contents
Linux Multi-Queue Block IO Queueing Mechanism (blk-mq)
blk-mq (Multi-Queue Block IO Queueing Mechanism) is a new framework for the Linux block layer that was introduced with Linux Kernel 3.13 and has become feature-complete with Kernel 3.16.
Blk-mq allows for over 15 million IOPS with high-performance flash devices (e.g. PCIe SSDs) on multi-socket servers, though even single and dual socket servers also benefit considerably from blk-mq.
To use a device with blk-mq, the device must support the respective driver.
Overview of blk-mq
Blk-mq integrates into the storage stack and provides basic functions to device drivers for mapping I/O enquiries to multiple queues.
The tasks are distributed across multiple threads and therefore to multiple CPU cores (per-core software queues).
Blk-mq compatible drivers request count for parallel hardware queues a device supports.
All device drivers that use the previous block I/O layer continue to work independently of blk-mq.
Device Drivers
| Driver | Device Name | Supported Devices |
| null_blk | /dev/nullb* | none (test drivers) |
| virtio-blk | /dev/vd* | Virtual guest drivers |
| mtip32xx | /dev/rssd* | Micron RealSSD PCIe |
| scsi (scsi_mq) | /dev/sd* | e.g. SAS and SATA SSDs/HDDs |
| NVMe | /dev/nvme* | e.g. Intel SSD DC P3600 DC P3700 Series |
| rbd | /dev/rdb* | RADOS Block Device (Ceph) |
| ubi/block | /dev/ubiblock* | |
| loop | /dev/loop* | Loopback-Device |
| dm / dm-mpath |
Enable blk-mq
Enabling blk-mq must happen at boot time. You need to add scsi_mod.use_blk_mq=1 to your lilo.conf
- /etc/lilo.conf
# LILO configuration file # generated by 'liloconfig' # # Start LILO global section # Append any additional kernel parameters: append=" scsi_mod.use_blk_mq=1"
Automatic IO scheduler selection
udev rules can be used to select IO schedulers for different device types.
- /etc/udev/rules.d/60-ioscheduler.rules
# set deadline scheduler for non-rotating disks ACTION=="add|change", KERNEL=="sd[a-z]", TEST!="queue/rotational", ATTR{queue/scheduler}="deadline" ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="bfq" # set cfq scheduler for rotating disks ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="cfq"
Sources
* Originally written by lamerix
* Source https://www.thomas-krenn.com/en/wiki/Linux_Multi-Queue_Block_IO_Queueing_Mechanism_(blk-mq)
* Source http://kernel.dk/blk-mq.pdf
* Source https://lwn.net/Articles/552904/
