Repository navigation
Description
Description of the bug:
When testing data integrity across power cycles / reboots with trigger and verify_state_load=1, fio does not properly handle out-of-order completions (e.g. due to tail latency). This can lead to fio attempting to verify a block that was never written.
When saving/loading state files for verification, fio keeps track of only a finite number of recent write completions in thread_io_list::comps. The verify code (verify_state_should_stop(), specifically) assumes that any write issued before the writes recorded in comps must have been successfully completed. This is not generally true for iodepth > 1 and if writes are completed out-of-order.
Consider this example: a single write stalls while all other writes complete normally. This condition can persist for an arbitrarily long time, and the stalled write will fall out of the range "covered" by comps. Saving state at this point (e.g. via a trigger) and halting the workload (e.g. via power-cycling the machine) will result in that stalled write being verified when the state is loaded despite the fact that the write may have never completed.
It seems like a dynamically-sized data structure is needed to keep track of any completion "gaps" which may be arbitrarily out-of-order. It does not seem generally possible with a fixed-size data structure (but maybe I am missing something).
Environment:
$ cat /etc/redhat-release
CentOS Stream release 9
$ uname -a
Linux hyper-ba563a-1 6.11.1-gb9fe45adbd3e #20 SMP Mon Jun 2 21:55:22 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
fio version: 3.40
Reproduction steps
Note that this was reproduced consistently only by injecting artificial tail latency (1 second every 100 writes) via a proprietary ublk-based software storage driver. It may be possible to construct a similar setup with dm. I was able to reproduce the issue in 174 / 200 tests. It is also important to trim the entire space between each test.
$ cat trim.fio
[global]
iodepth=16
direct=1
group_reporting=1
numjobs=1
ioengine=io_uring
[job0]
rw=trim
filename=/dev/ublkb0
blocksize=1M
size=5%
randseed=43747
new_group=1
$ cat write.fio
[global]
iodepth=16
direct=1
group_reporting=1
numjobs=1
ioengine=io_uring
rw=randwrite
runtime=240
time_based
verify=md5
[job0]
rw=randwrite
filename=/dev/ublkb0
blocksize=128k
size=5%
randseed=43747
new_group=1
$ cat read.fio
[global]
iodepth=16
direct=1
group_reporting=1
numjobs=1
ioengine=io_uring
rw=randread
runtime=180
time_based
verify=md5
verify_backlog=10000000
verify_state_load=1
verify_async=4
verify_fatal=1
verify_dump=1
[job0]
filename=/dev/ublkb0
blocksize=128k
size=5%
randseed=43747
new_group=1
$ fio trim.fio
$ fio write.fio --trigger-timeout=120 --trigger='ipmitool chassis power cycle'
... reboot ...
$ fio read.fio
verify: bad magic header 0, wanted acca at file /dev/ublkb0 offset 177436229632, length 131072 (requested block: offset=177436229632, length=131072)
Activity
vincentkfu commentedon Jul 22, 2025
Shinichiro sought to reduce the severity of this problem earlier this year by doubling the number of completions recorded in the verify state file for async workloads: https://lore.kernel.org/fio/20250213052510.1474423-1-shinichiro.kawasaki@wdc.com/
@kawasaki Do you have any additional thoughts about this?
kawasaki commentedon Jul 22, 2025
My fix was a simple one, and it was enough to avoid my failure scenario which was observed with real storage devices. I expected and hoped that it would be enough to avoid similar failures for other users. However, it turned out that guess was wrong, especially with artificial devices and error injections as described in this GitHub issue.
Actually I had another idea when I worked on my failure scenario:
For the write workload:
For the read verify workload:
This way, we will record writes as many as the queue depth for each file, then, there is no need to double the buffer. On top of that, we will not need to change the write record buffer size dynamically for the stalled writes either. However, this solution will add some code complexity, since write state handling is required at both write issues and write completions. Also, it adds read verify overhead as I noted above. Then I concluded that this solution approach is not worth trying and I worked on the simpler "double the buffer" solution.
I guess the idea above will avoid the the failure reported, and could be more feasible than dynamically increasing the write record buffer size. (Please note that I just have this idea in my mind, and am not 100% sure that it will work).
sitsofe commentedon Jul 22, 2025
I agree with @kawasaki's idea. Unless you are going to record every I/O individually, it is insufficient to go off completions alone because an "infinite stall" means a region is indeterminate. However, could you go off inflight I/Os instead assuming you recorded them all? My thinking is that if you take all inflight I/Os and look at their sequence numbers you can deduce the highest completed sequence number by finding whichever inflight I/O has the highest sequence and then assuming all inflight I/Os represent gaps. However maybe there's a corner case when you near completion...
kawasaki commentedon Jul 22, 2025
@sitsofe Thanks for the comment. I'm not so sure if I catch what you mean with the word "inflight I/Os", but I guess we are on a same page, probably. My idea is to record writes as many as the queue depth, and invalidate the record of the completed writes. In other words, I meant recording only the inflight writes.
It sounds a good idea to record sequence numbers. I guess you mean "io_u->numberio". By recording this sequence number of inflight writes, it will be easier to implement the handling at the completion (we can find the corresponding entry saved at the issue by looking up the sequence number). And as you say, the read/verify workload does not need to check the recorded writes until its sequence number reach to the smallest sequence number recorded. With this idea, my concern about the read/verify overhead is not valid.
It sounds interesting to work on the fix, but I myself do not have time to afford at this moment. If someone can work on it, it will be great.
sitsofe commentedon Jul 22, 2025
@kawasaki:
That's what I mean too but you've worded it more concisely and clearly!
Yes.
I'm afraid I'm in a similar position (I don't really have time to work on this and I'm not familiar with that part of the existing code) but I agree it would be good for someone to submit something for review - I like the idea of making fio more useful!
riley-pstg commentedon Jul 22, 2025
Thank you for the discussion folks! That idea makes sense to me - we can keep track of the "gaps" in the set of completed sequence numbers. The number of gaps should be at most the iodepth (each "inflight" write creates a potential gap), so we can use a simple fixed-size data structure similar to the existing circular buffer. All offsets less than the minimum gap should be verified on the read side, as well as all offsets between the minimum and maximum gaps that are not tracked as gaps.
This is much simpler than what I had in mind, so I think I can take a stab at it. I also have a reliable reproduction which makes testing easier.
verify: rework write tracking for use with verify_save_state()
verify: plumb inflight write information through verify state
verify: rework write tracking for use with verify_save_state()
verify: plumb inflight write information through verify state
verify: plumb inflight write information through verify state
verify: rework write tracking for use with verify_save_state()
verify: plumb inflight write information through verify state
verify: rework write tracking for use with verify_save_state()
verify: plumb inflight write information through verify state
verify: rework write tracking for use with verify_save_state()
verify: plumb inflight write information through verify state
verify: rework write tracking for use with verify_save_state()
verify: plumb inflight write information through verify state
sitsofe commentedon Aug 10, 2025
@riley-pstg:
Nicely done!
I know 1e6ca39 added an additional fix and my only other thought is whether we could calculate the earliest inflight numberio so as to reduce the number of comparisons in https://github.com/axboe/fio/blob/master/verify.c#L1923-L1930 in the non-stuck I/O case (which is hopefully more common).
At any rate I consider this issue resolved thanks to your PR.
(Also nice to know that Pure Storage are using fio in some capacity)
noclip-code commentedon Aug 11, 2025
Yes good idea - it should be straightforward to cache the minimum inflight value when the verify state is loaded.
We use fio as a load gen for performance benchmarks in a few places. It's a great tool!
verify: rework write tracking for use with verify_save_state()
verify: plumb inflight write information through verify state