Skip to content

verify state does not properly handle out-of-order write completions #1950

Closed

Description

@riley-pstg

Description of the bug:
When testing data integrity across power cycles / reboots with trigger and verify_state_load=1, fio does not properly handle out-of-order completions (e.g. due to tail latency). This can lead to fio attempting to verify a block that was never written.

When saving/loading state files for verification, fio keeps track of only a finite number of recent write completions in thread_io_list::comps. The verify code (verify_state_should_stop(), specifically) assumes that any write issued before the writes recorded in comps must have been successfully completed. This is not generally true for iodepth > 1 and if writes are completed out-of-order.

Consider this example: a single write stalls while all other writes complete normally. This condition can persist for an arbitrarily long time, and the stalled write will fall out of the range "covered" by comps. Saving state at this point (e.g. via a trigger) and halting the workload (e.g. via power-cycling the machine) will result in that stalled write being verified when the state is loaded despite the fact that the write may have never completed.

It seems like a dynamically-sized data structure is needed to keep track of any completion "gaps" which may be arbitrarily out-of-order. It does not seem generally possible with a fixed-size data structure (but maybe I am missing something).

Environment:

$ cat /etc/redhat-release 
CentOS Stream release 9
$ uname -a
Linux hyper-ba563a-1 6.11.1-gb9fe45adbd3e #20 SMP Mon Jun  2 21:55:22 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

fio version: 3.40

Reproduction steps
Note that this was reproduced consistently only by injecting artificial tail latency (1 second every 100 writes) via a proprietary ublk-based software storage driver. It may be possible to construct a similar setup with dm. I was able to reproduce the issue in 174 / 200 tests. It is also important to trim the entire space between each test.

$ cat trim.fio 
[global]
iodepth=16
direct=1
group_reporting=1
numjobs=1
ioengine=io_uring
[job0]
rw=trim
filename=/dev/ublkb0
blocksize=1M
size=5%
randseed=43747
new_group=1

$ cat write.fio 
[global]
iodepth=16
direct=1
group_reporting=1
numjobs=1
ioengine=io_uring
rw=randwrite
runtime=240
time_based
verify=md5
[job0]
rw=randwrite
filename=/dev/ublkb0
blocksize=128k
size=5%
randseed=43747
new_group=1

$ cat read.fio
[global]
iodepth=16
direct=1
group_reporting=1
numjobs=1
ioengine=io_uring
rw=randread
runtime=180
time_based
verify=md5
verify_backlog=10000000
verify_state_load=1
verify_async=4
verify_fatal=1
verify_dump=1
[job0]
filename=/dev/ublkb0
blocksize=128k
size=5%
randseed=43747
new_group=1

$ fio trim.fio

$ fio write.fio --trigger-timeout=120 --trigger='ipmitool chassis power cycle'

... reboot ...

$ fio read.fio
verify: bad magic header 0, wanted acca at file /dev/ublkb0 offset 177436229632, length 131072 (requested block: offset=177436229632, length=131072)

Activity

  1. vincentkfu commented on Jul 22, 2025

    @vincentkfu
    Collaborator

    Shinichiro sought to reduce the severity of this problem earlier this year by doubling the number of completions recorded in the verify state file for async workloads: https://lore.kernel.org/fio/20250213052510.1474423-1-shinichiro.kawasaki@wdc.com/

    @kawasaki Do you have any additional thoughts about this?

  2. kawasaki commented on Jul 22, 2025

    @kawasaki
    Contributor

    My fix was a simple one, and it was enough to avoid my failure scenario which was observed with real storage devices. I expected and hoped that it would be enough to avoid similar failures for other users. However, it turned out that guess was wrong, especially with artificial devices and error injections as described in this GitHub issue.

    Actually I had another idea when I worked on my failure scenario:

    For the write workload:

    • Record last N writes issued, instead of writes completed
    • Invalidate the recorded writes when they completed
    • To record a new write, find out an invalidated write in the buffer and overwrite it

    For the read verify workload:

    • Check if each read/verify IO targets the offsets of all valid writes recorded in the verify state. If so, stop verifying. (This will add some overhead, since now this check is done only for the last N=depth reads).

    This way, we will record writes as many as the queue depth for each file, then, there is no need to double the buffer. On top of that, we will not need to change the write record buffer size dynamically for the stalled writes either. However, this solution will add some code complexity, since write state handling is required at both write issues and write completions. Also, it adds read verify overhead as I noted above. Then I concluded that this solution approach is not worth trying and I worked on the simpler "double the buffer" solution.

    I guess the idea above will avoid the the failure reported, and could be more feasible than dynamically increasing the write record buffer size. (Please note that I just have this idea in my mind, and am not 100% sure that it will work).

  3. sitsofe commented on Jul 22, 2025

    @sitsofe
    Collaborator

    I agree with @kawasaki's idea. Unless you are going to record every I/O individually, it is insufficient to go off completions alone because an "infinite stall" means a region is indeterminate. However, could you go off inflight I/Os instead assuming you recorded them all? My thinking is that if you take all inflight I/Os and look at their sequence numbers you can deduce the highest completed sequence number by finding whichever inflight I/O has the highest sequence and then assuming all inflight I/Os represent gaps. However maybe there's a corner case when you near completion...

  4. kawasaki commented on Jul 22, 2025

    @kawasaki
    Contributor

    @sitsofe Thanks for the comment. I'm not so sure if I catch what you mean with the word "inflight I/Os", but I guess we are on a same page, probably. My idea is to record writes as many as the queue depth, and invalidate the record of the completed writes. In other words, I meant recording only the inflight writes.

    It sounds a good idea to record sequence numbers. I guess you mean "io_u->numberio". By recording this sequence number of inflight writes, it will be easier to implement the handling at the completion (we can find the corresponding entry saved at the issue by looking up the sequence number). And as you say, the read/verify workload does not need to check the recorded writes until its sequence number reach to the smallest sequence number recorded. With this idea, my concern about the read/verify overhead is not valid.

    It sounds interesting to work on the fix, but I myself do not have time to afford at this moment. If someone can work on it, it will be great.

  5. sitsofe commented on Jul 22, 2025

    @sitsofe
    Collaborator

    @kawasaki:

    My idea is to record writes as many as the queue depth, and invalidate the record of the completed writes. In other words, I meant recording only the inflight writes.

    That's what I mean too but you've worded it more concisely and clearly!

    It sounds a good idea to record sequence numbers. I guess you mean "io_u->numberio"

    Yes.

    It sounds interesting to work on the fix, but I myself do not have time to afford at this moment. If someone can work on it, it will be great.

    I'm afraid I'm in a similar position (I don't really have time to work on this and I'm not familiar with that part of the existing code) but I agree it would be good for someone to submit something for review - I like the idea of making fio more useful!

  6. added
    triagedIssue cause is understood but a patch is needed to fix it
    patcheswelcomeNo one internal is working on this but an external contributor's help is very welcome!
    on Jul 22, 2025
  7. riley-pstg commented on Jul 22, 2025

    @riley-pstg
    Author

    Thank you for the discussion folks! That idea makes sense to me - we can keep track of the "gaps" in the set of completed sequence numbers. The number of gaps should be at most the iodepth (each "inflight" write creates a potential gap), so we can use a simple fixed-size data structure similar to the existing circular buffer. All offsets less than the minimum gap should be verified on the read side, as well as all offsets between the minimum and maximum gaps that are not tracked as gaps.

    This is much simpler than what I had in mind, so I think I can take a stab at it. I also have a reliable reproduction which makes testing easier.

  8. sitsofe commented on Aug 10, 2025

    @sitsofe
    Collaborator

    @riley-pstg:
    Nicely done!

    I know 1e6ca39 added an additional fix and my only other thought is whether we could calculate the earliest inflight numberio so as to reduce the number of comparisons in https://github.com/axboe/fio/blob/master/verify.c#L1923-L1930 in the non-stuck I/O case (which is hopefully more common).

    At any rate I consider this issue resolved thanks to your PR.

    (Also nice to know that Pure Storage are using fio in some capacity)

  9. noclip-code commented on Aug 11, 2025

    @noclip-code
    Contributor

    Yes good idea - it should be straightforward to cache the minimum inflight value when the verify state is loaded.

    We use fio as a load gen for performance benchmarks in a few places. It's a great tool!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    patcheswelcomeNo one internal is working on this but an external contributor's help is very welcome!triagedIssue cause is understood but a patch is needed to fix it

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址