Skip to content

verify: fix bandwidth asymmetry and memory leak with norandommap - #2107

Merged
axboe merged 2 commits into
axboe:masterfrom
malikoyv:pr-speed-up-norandommap-read-bw
Jun 15, 2026
Merged

axboe merged 2 commits into
axboe:masterfrom
malikoyv:pr-speed-up-norandommap-read-bw

Conversation

@malikoyv

@malikoyv malikoyv commented Jun 9, 2026

Copy link
Copy Markdown

When norandommap is set, io_pieces are stored in an RB tree sorted by offset to detect overlapping writes. This caused two problems with verify_backlog:

  1. get_next_verify() only checked rb_first() for a completed entry. If
    the lowest-offset IO was still in-flight, it gave up entirely even
    though completed entries existed at higher offsets. This starved
    verification reads, causing severe read/write bandwidth asymmetry
    (e.g. 40 MiB/s read vs 700 MiB/s write).

  2. When an in-flight io_piece was evicted from the RB tree due to an
    overlapping write, the IP_F_ONRB flag was not cleared. On IO
    completion the entry appeared to still belong to the tree, so it was
    never freed. This caused unbounded memory growth over long runs.

Fix (1) by walking the tree with rb_next() to find the first non in-flight entry instead of giving up at rb_first().

Fix (2) by clearing IP_F_ONRB when evicting an in-flight entry from the tree, and detecting the orphaned entry on IO completion to free it.

Neither issue affects the linked-list path used when norandommap is not set, since list ordering matches submission order and no overlap eviction occurs.

@vincentkfu

Copy link
Copy Markdown
Collaborator

This seems like two distinct issues. Please split the PR into two patches, one for each issue.

Yehor Malikov added 2 commits June 10, 2026 09:25
When norandommap is set, io_pieces are stored in an RB tree sorted by
offset. get_next_verify() only checked rb_first() for a completed
entry. If the lowest-offset IO was still in-flight, it returned
"nothing" even though completed entries existed at higher offsets. This
starved verification reads during verify_backlog processing, causing
severe read/write bandwidth asymmetry (e.g. 40 MiB/s read vs 700 MiB/s
write) and unbounded io_piece accumulation in the tree.

Fix this by walking the tree with rb_next() to find the first completed
entry instead of giving up at rb_first().

This does not affect the linked-list path used when norandommap is not
set, since list ordering matches submission order.

Signed-off-by: Yehor Malikov <Yehor.Malikov@solidigm.com>
When an in-flight io_piece is evicted from the RB tree due to an
overlapping write in log_io_piece(), the entry cannot be freed
immediately. However, the IP_F_ONRB flag was not cleared on eviction.
On IO completion, the entry still appeared to belong to the tree, so
it was never freed.

Clear IP_F_ONRB when evicting an in-flight entry from the tree. On IO
completion, detect the orphaned io_piece (neither IP_F_ONRB nor
IP_F_ONLIST set) and free it.

This leak only occurs with norandommap or softrandommap where
fio_offset_overlap_risk() is true, causing io_pieces to be stored in
the RB tree with overlap detection.

Signed-off-by: Yehor Malikov <Yehor.Malikov@solidigm.com>
@malikoyv
malikoyv force-pushed the pr-speed-up-norandommap-read-bw branch from 047c174 to c0e4b6e Compare June 10, 2026 07:26
@vincentkfu
vincentkfu requested a review from Copilot June 15, 2026 15:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes online verification backlog handling when norandommap/overlap detection stores io_pieces in an RB tree, addressing both verification starvation (read/write bandwidth asymmetry) and a long-run memory leak.

Changes:

  • verify.c: Walk the RB tree (rb_next()) to locate the first non-in-flight io_piece instead of giving up at rb_first().
  • iolog.c: Clear IP_F_ONRB when evicting an overlapping in-flight io_piece from the RB tree.
  • io_u.c: Detect an evicted/orphaned io_piece at IO completion and free it to prevent leaks.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
verify.c Avoid verify starvation by scanning RB tree for the first completed entry.
iolog.c Fix eviction bookkeeping by clearing IP_F_ONRB for removed entries.
io_u.c Free orphaned io_pieces on completion to prevent unbounded memory growth.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread io_u.c
Comment on lines +2170 to +2174
if (io_u->error) {
unlog_io_piece(td, io_u);
else {
atomic_store_release(&io_u->ipo->flags,
io_u->ipo->flags & ~IP_F_IN_FLIGHT);
} else {
unsigned int flags = io_u->ipo->flags & ~IP_F_IN_FLIGHT;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A better solution would be to decrement td->io_hist_len in unlog_io_piece() only if the io_piece is removed from the tree or list.

@axboe
axboe merged commit fb5b6ae into axboe:master Jun 15, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

⌂ 换网址