Tracing a Block I/O Request End-to-End
Follow a write() syscall from the VFS layer through bio construction, blk-mq staging queues, the mq-deadline scheduler, NVMe command submission, and interrupt-driven completion.
Walkthrough
// annotated source at every step
write() enters the kernel via the syscall entry path
A userspace write() call crosses into the kernel through the syscall entry path (see the Syscall Entry tutorial). The kernel dispatches to sys_write() → ksys_write() → vfs_write(). The VFS layer resolves the file's inode and calls the filesystem's write_iter() method — for ext4 this is ext4_file_write_iter(). At this point we are still in the VFS layer; no block I/O has been issued yet.
// fs/read_write.c
ssize_t ksys_write(unsigned int fd, const char __user *buf, size_t count)
{
struct fd f = fdget_pos(fd);
// ...
ret = vfs_write(f.file, buf, count, &pos);
// ...
}Filesystem buffers data in the page cache
ext4_file_write_iter() calls generic_perform_write(), which locates or allocates page cache pages for the target file range via grab_cache_page_write_begin(). The filesystem copies userspace data into these pages with iov_iter_copy_from_user_atomic(). The pages are now dirty — they hold the new data but it has not yet been written to disk. The writeback subsystem will eventually flush them, or the process can call fsync() to force immediate writeback.
// mm/filemap.c
ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
{
// ...
status = a_ops->write_begin(file, mapping, pos, bytes, &page, &fsdata);
copied = copy_page_from_iter_atomic(page, offset, bytes, i);
a_ops->write_end(file, mapping, pos, bytes, copied, page, fsdata);
// ...
}Writeback submits dirty pages as a bio
The kernel's writeback thread (or fsync()) calls ext4_writepages() → mpage_writepages(). This function iterates over dirty pages and constructs a struct bio — the fundamental I/O descriptor in the block layer. bio_add_page() appends each dirty page as a bio_vec segment. The bio records the target block device, the starting sector on disk, the I/O direction (WRITE), and a bi_end_io callback that will fire on completion. submit_bio() hands the bio to the block layer.
// block/bio.c
struct bio *bio_alloc(gfp_t gfp_mask, unsigned short nr_iovecs)
{
// allocates bio + bio_vec array
}
// mm/mpage.c — simplified
bio = mpage_alloc(bdev, first_block, nr_pages, GFP_NOFS);
bio_add_page(bio, page, PAGE_SIZE, 0);
submit_bio(bio);blk-mq places the request in a per-CPU software queue
submit_bio() calls blk_mq_submit_bio(). blk-mq allocates a struct request from the request pool and copies the bio's sector/page information into it. The request is inserted into the per-CPU software staging queue (struct blk_mq_ctx) without acquiring any global lock — this is the key scalability win of blk-mq over the legacy single-queue design. If a plug is active (blk_start_plug() was called), the request stays in the plug list for batching; otherwise blk_mq_run_hw_queue() is called immediately.
// block/blk-mq.c
void blk_mq_submit_bio(struct bio *bio)
{
struct blk_mq_ctx *ctx = blk_mq_get_ctx(q);
struct request *rq = blk_mq_get_request(q, bio, &data);
// copy bio → rq
blk_mq_bio_to_request(rq, bio, nr_segs);
// insert into per-CPU sw queue
blk_mq_insert_request(rq, 0);
blk_mq_run_hw_queue(hctx, false);
}mq-deadline scheduler sorts and dispatches the request
blk_mq_run_hw_queue() calls the I/O scheduler's dispatch function. With mq-deadline, deadline_dispatch_requests() is called. mq-deadline maintains two red-black trees — one sorted by sector (for spatial locality) and one sorted by deadline (to prevent starvation). It picks the next request by checking whether any request has exceeded its deadline (500 ms for reads, 5 s for writes); if not, it picks the next sector-ordered request for sequential throughput. The chosen request is removed from the scheduler queues and handed to the driver.
// block/mq-deadline.c
static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx)
{
struct deadline_data *dd = hctx->queue->elevator->elevator_data;
// check for expired deadlines first
rq = deadline_check_fifo(dd, data_dir);
if (!rq)
rq = deadline_next_request(dd, data_dir);
return rq;
}NVMe driver writes the command to the submission queue
The NVMe driver's queue_rq() callback — nvme_queue_rq() — is called with the dispatched request. It translates the block-layer request into an NVMe command (struct nvme_rw_command) specifying the namespace ID, starting LBA, and number of blocks. The command is written to the tail of the NVMe submission queue (SQ) in host memory. A single MMIO write to the SQ doorbell register signals the NVMe controller that a new command is available. The controller fetches the command via PCIe DMA and begins executing it on the flash media.
// drivers/nvme/host/pci.c
static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
struct nvme_ns *ns = hctx->queue->queuedata;
struct nvme_rw_command *cmnd = nvme_setup_rw(ns, req);
// write command to SQ tail
nvme_sq_copy_cmd(nvmeq, cmnd);
// ring the doorbell
writel(nvmeq->sq_tail, nvmeq->q_db);
return BLK_STS_OK;
}NVMe controller completes; MSI-X interrupt fires
When the NVMe controller finishes writing the data to flash, it writes a completion entry to the completion queue (CQ) in host memory and raises an MSI-X interrupt. The CPU's interrupt handler — nvme_irq() — is invoked. It reads the CQ head, finds the completion entry, and calls nvme_handle_cqe(). This resolves the original request and calls blk_mq_complete_request(), which schedules the request's completion on the submitting CPU via a softirq to avoid cross-CPU cache thrashing.
// drivers/nvme/host/pci.c
static irqreturn_t nvme_irq(int irq, void *data)
{
struct nvme_queue *nvmeq = data;
// read CQ entries
while (nvme_cqe_pending(nvmeq)) {
nvme_handle_cqe(nvmeq, &nvmeq->cq_head);
}
// update CQ head doorbell
writel(nvmeq->cq_head, nvmeq->q_db + nvmeq->dev->db_stride);
return IRQ_HANDLED;
}bio completion propagates back to the filesystem
blk_mq_complete_request() calls the request's end_io function, which calls bio_endio(). bio_endio() invokes the bio's bi_end_io callback — set by the filesystem when the bio was constructed. For a writeback bio, this callback clears the PG_writeback flag on each page, marks the page clean, and wakes any processes waiting on writeback completion (e.g., a process blocked in fsync()). The write() syscall that initiated this chain can now return to userspace — the data is durably on disk.
// block/bio.c
void bio_endio(struct bio *bio)
{
// call filesystem's bi_end_io
if (bio->bi_end_io)
bio->bi_end_io(bio);
}
// mm/page-writeback.c — writeback completion
static void end_page_writeback(struct page *page)
{
ClearPageWriteback(page);
wake_up_page(page, PG_writeback);
}Knowledge Check
// click each question to reveal the answer
Source References
// annotated entry points · Linux 7.2