You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
When using libaio/io_uring engine and executing experimental_verify FIO generates writes instead of reads #1955
I have read the GitHub issues section of REPORTING-BUGS.
Description of the bug:
When executing verify_only with experimental_verify, FIO generates writes to the device/file.
Environment: Linux using libaio engine
fio version: 3.33, but it might be valid for master
Reproduction steps
Run FIO with verify_only together with experimental_verify. We expect FIO to perform only READ operations for data verification. However, we are observing only WRITE operations. The issue might be related to the experimental_verify procedure: when it changes the I/O direction (from WRITE to READ), it may not trigger the prep io function in the I/O engine
In all cases, once fio gets to the verification phase it will report that it switched to issuing reads, but checking via iostat shows that it's only sending writes to the tested device.
Can you make this happen with a sequential write (rw=write) workload?
Can you make it happen with a single write by limit the size (size=4k)?
Can you make it happen with ioengine=sync?
Can you make it happen with direct=1 (although doing this may be tough if you detecting writes by device stats as opposed to through some sort of trace)
Can you make it happen without group_reporting?
Can you make it happen without serialize_overlap=1?
Can you make it happen with a plain file?
After removing all the above the final questions is: can you make it happen without experimental_verify=1? [mmkayPL has stated below that this option is required to reproduce the issue]
The reason for doing all the above is that we always want the minimum number of options that make a problem happen so it's easier for other to reproduce and so there are less places to look in.
@mmkayPL your job is going to do writes because by default fio does the write job followed by a separate read verification pass (see do_verify). The job @mariuszbarczak gave sets verify_only so I would not expect that to do any writes.
@mmkayPL your job is going to do writes because by default fio does the write job followed by a separate read verification pass (see do_verify). The job @mariuszbarczak gave sets verify_only so I would not expect that to do any writes.
Yes, I'm aware that without verify_only there are two passes (a write and then read pass) done by fio. The issue is that when in the verification pass (whether via verify_only set to 1, or waiting until the writes are done) even though fio is reporting that it's doing verification/reads and in the end report it's mentioning that it has been doing reads, if you check under debugger/with iostat what is actually being sent to the device it's always writes during the verification phase and fio never actually reads from the device. Running without experimental_verify works as expected (you can see reads with iostat).
I'll try to answer all other questions tomorrow.
@mmkayPL OK thanks for reiterating. If you are able to get the run to be small it would also be useful to see the output of fio when using --debug=io as the first command line option and having that uploaded here as a text attachment.
Yes the problem occurs on both random and sequential workloads
4096 size is a bit hard to tell, but I do seem to see a tick of read activity with iostat (though it could be udev, I will have to change my test setup a bit to 100% confirm this
The issue doesn't appear with ioengine=sync
I'm assuming you meant direct=0, in which case as you suspected, it's a bit hard to tell
Group reporting setting can be removed, and the issue still appears
Serialize overlap setting can be removed, and the issue still appears
The issue is reproducible on a plain file
The issue appears at iodepth=1
While I don't think this is the proper fix, we found that adding the following change to engines/libaio.c fio_libaio_queue function (after fio_ro_check) would work:
if (io_u->ddir == DDIR_READ) {
io_u->iocb.aio_lio_opcode = IO_CMD_PREAD;
}
I've been doing these experiments on commit 306d898 (so roughly master at the time of the creation of the issue) and config was cut down to this:
The issue doesn't appear with posixaio, but it does appear with io_uring (and just to doublecheck, io_uring with experimental_verify turned off works correctly).
changed the title [-]When using libaio engine and executing experimental_verify FIO generates writes instead of reads[/-][+]When using libaio/io_uring engine and executing experimental_verify FIO generates writes instead of reads[/+]on Aug 12, 2025
So we can see the first (dry run) I/O being filled with a data direction of 1 (which is write) and prepped with a data direction of 1. However we see the second I/O done by verification being filled and prepped with a data direction of 1 and then queued with a data direction of 0 (read) followed by a commit(). You can see that the libaio ioengine sets the I/O's direction in an ancillary structure (iocb which is filled out by io_prep_*() functions) at prep time (see https://github.com/axboe/fio/blob/fio-3.40/engines/libaio.c#L118-L148 ).
Basically any async ioengine that sets the data direction at prep time (rather than at queue time like the posixaio ioengine does) is going to hit this issue. I think to fix it you would have to move (some/all of?) the "rewrite to the ddir to read" logic to somewhere like get_io_u() because you need to have changed the ddir in both the case where the ioengine doesn't have a .prep (which rules out (only) doing it in td_io_prep()) but before an ioengine's .prep function is called.
We have a question regarding the experimental_verify parameter.
We currently use fio’s verification feature for data integrity testing, and we’re considering enabling experimental_verify in our verification path.
Could you please clarify how stable or reliable experimental_verify is at this point?
Is it safe to rely on it for accurate verification, or would you recommend avoiding it in production tests for now?
@yehormalikov I'm not the best person to ask as I don't use fio in anger on a regular basis (so you may want to hear from others). From what I know experimental_verify been around for a while but you can see the issue above was only reported and fixed recently so I'd say most people aren't using it. I think there may be cases it handles poorly (verification when multiple passes are done, handling the case where a write I/O was stalled) that regular verification can but I haven't verified whether my supposition is correct. If your production workload is do a single full pass of writes then verify later with no weirdness in-between I would imagine experimental_verify would work well and with a far far lower memory overhead than regular verification.
@ankit-sam and I went through the standard verify feature several months ago fixing issues and adding a test script. So I would have to say that non-experimental verify is better tested. However, if experimental verify fits your use case, then I would be willing to use it in production after validating it. Just make sure that you understand how regular verify and experimental verify differ. Regular verify saves all the IOs in a data structure for the verification phase whereas experimental_verify resets the random seeds and re-generates the sequence of offsets for the verification phase. There can be issues like the overwrites that @sitsofe mentioned where experimental_verify may fail.
@ankit-sam and I went through the standard verify feature several months ago fixing issues and adding a test script. So I would have to say that non-experimental verify is better tested. However, if experimental verify fits your use case, then I would be willing to use it in production after validating it. Just make sure that you understand how regular verify and experimental verify differ. Regular verify saves all the IOs in a data structure for the verification phase whereas experimental_verify resets the random seeds and re-generates the sequence of offsets for the verification phase. There can be issues like the overwrites that @sitsofe mentioned where experimental_verify may fail.
Thank you both for the detailed clarifications. We understand how experimental_verify operates and appreciate its advantages - particularly the fact that it avoids storing I/O metadata in memory, which makes it much more scalable for large-capacity tests.
That said, the “experimental” designation makes us somewhat cautious to rely on it in production validation flows. We’d like to understand whether there are any ongoing efforts or future plans to stabilize and promote experimental_verify to a some standard verification parameter, or if it’s expected to remain in its current state for the foreseeable future.
Any insights on its long-term roadmap or intended level of support would be greatly appreciated.
experimental_verify has existed for at least a decade. It may have been experimental when it was first added but it's just a regular feature with an unfortunate name by now.
I do not know of any efforts currently underway to overhaul it or comprehensively assess it.
Activity
mariuszbarczak commentedon Jul 31, 2025
Update: Confirmed, this is observed on the master branch too.
Note: FIO reports reads, however iostat shows writes
mmkayPL commentedon Jul 31, 2025
In addition, the issue also occurs in sequential workloads, or in random workloads with or without the norandommap option:
In all cases, once fio gets to the verification phase it will report that it switched to issuing reads, but checking via iostat shows that it's only sending writes to the tested device.
sitsofe commentedon Aug 9, 2025
@mariuszbarczak, @mmkayPL OK let's try and cut this down further.
rw=write) workload?size=4k)?ioengine=sync?direct=1(although doing this may be tough if you detecting writes by device stats as opposed to through some sort of trace)group_reporting?serialize_overlap=1?After removing all the above the final questions is: can you make it happen without[mmkayPL has stated below that this option is required to reproduce the issue]experimental_verify=1?The reason for doing all the above is that we always want the minimum number of options that make a problem happen so it's easier for other to reproduce and so there are less places to look in.
sitsofe commentedon Aug 9, 2025
@mmkayPL your job is going to do writes because by default fio does the write job followed by a separate read verification pass (see
do_verify). The job @mariuszbarczak gave setsverify_onlyso I would not expect that to do any writes.mmkayPL commentedon Aug 10, 2025
Yes, I'm aware that without verify_only there are two passes (a write and then read pass) done by fio. The issue is that when in the verification pass (whether via verify_only set to 1, or waiting until the writes are done) even though fio is reporting that it's doing verification/reads and in the end report it's mentioning that it has been doing reads, if you check under debugger/with iostat what is actually being sent to the device it's always writes during the verification phase and fio never actually reads from the device. Running without experimental_verify works as expected (you can see reads with iostat).
I'll try to answer all other questions tomorrow.
sitsofe commentedon Aug 10, 2025
@mmkayPL OK thanks for reiterating. If you are able to get the run to be small it would also be useful to see the output of fio when using
--debug=ioas the first command line option and having that uploaded here as a text attachment.mmkayPL commentedon Aug 11, 2025
So to answer the rest of the questions:
ioengine=syncdirect=0, in which case as you suspected, it's a bit hard to tellWhile I don't think this is the proper fix, we found that adding the following change to engines/libaio.c fio_libaio_queue function (after fio_ro_check) would work:
I've been doing these experiments on commit 306d898 (so roughly master at the time of the creation of the issue) and config was cut down to this:
sitsofe commentedon Aug 11, 2025
@mmkayPL: Thanks for minimising the test case. Next question: does it happen when using the
posixaioorio_uringioengines?mmkayPL commentedon Aug 12, 2025
The issue doesn't appear with posixaio, but it does appear with io_uring (and just to doublecheck, io_uring with experimental_verify turned off works correctly).
[-]When using libaio engine and executing experimental_verify FIO generates writes instead of reads[/-][+]When using libaio/io_uring engine and executing experimental_verify FIO generates writes instead of reads[/+]sitsofe commentedon Aug 12, 2025
I can reproduce the issue with the following:
An excerpt from the output:
So we can see the first (dry run) I/O being filled with a data direction of 1 (which is write) and prepped with a data direction of 1. However we see the second I/O done by verification being filled and prepped with a data direction of 1 and then queued with a data direction of 0 (read) followed by a
commit(). You can see that the libaio ioengine sets the I/O's direction in an ancillary structure (iocbwhich is filled out byio_prep_*()functions) at prep time (see https://github.com/axboe/fio/blob/fio-3.40/engines/libaio.c#L118-L148 ).Basically any async ioengine that sets the data direction at prep time (rather than at queue time like the posixaio ioengine does) is going to hit this issue. I think to fix it you would have to move (some/all of?) the "rewrite to the
ddirto read" logic to somewhere likeget_io_u()because you need to have changed theddirin both the case where the ioengine doesn't have a.prep(which rules out (only) doing it intd_io_prep()) but before an ioengine's.prepfunction is called.vincentkfu commentedon Oct 30, 2025
@sitsofe Does #2007 seem like a reasonable fix to you?
backend.c: fix async ioengine prep/queue direction mismatch in experi…
sitsofe commentedon Oct 30, 2025
@vincentkfu Yes! It has a good commit message, it's more elegant than what I proposed and it only has a double prep as the downside.
backend.c: fix async ioengine prep/queue direction mismatch in experi…
yehormalikov commentedon Nov 3, 2025
@sitsofe, @vincentkfu,
We have a question regarding the experimental_verify parameter.
We currently use fio’s verification feature for data integrity testing, and we’re considering enabling experimental_verify in our verification path.
Could you please clarify how stable or reliable experimental_verify is at this point?
Is it safe to rely on it for accurate verification, or would you recommend avoiding it in production tests for now?
sitsofe commentedon Nov 3, 2025
@yehormalikov I'm not the best person to ask as I don't use fio in anger on a regular basis (so you may want to hear from others). From what I know
experimental_verifybeen around for a while but you can see the issue above was only reported and fixed recently so I'd say most people aren't using it. I think there may be cases it handles poorly (verification when multiple passes are done, handling the case where a write I/O was stalled) that regular verification can but I haven't verified whether my supposition is correct. If your production workload is do a single full pass of writes then verify later with no weirdness in-between I would imagineexperimental_verifywould work well and with a far far lower memory overhead than regular verification.vincentkfu commentedon Nov 4, 2025
@ankit-sam and I went through the standard verify feature several months ago fixing issues and adding a test script. So I would have to say that non-experimental verify is better tested. However, if experimental verify fits your use case, then I would be willing to use it in production after validating it. Just make sure that you understand how regular verify and experimental verify differ. Regular verify saves all the IOs in a data structure for the verification phase whereas experimental_verify resets the random seeds and re-generates the sequence of offsets for the verification phase. There can be issues like the overwrites that @sitsofe mentioned where experimental_verify may fail.
yehormalikov commentedon Nov 4, 2025
Thank you both for the detailed clarifications. We understand how experimental_verify operates and appreciate its advantages - particularly the fact that it avoids storing I/O metadata in memory, which makes it much more scalable for large-capacity tests.
That said, the “experimental” designation makes us somewhat cautious to rely on it in production validation flows. We’d like to understand whether there are any ongoing efforts or future plans to stabilize and promote experimental_verify to a some standard verification parameter, or if it’s expected to remain in its current state for the foreseeable future.
Any insights on its long-term roadmap or intended level of support would be greatly appreciated.
vincentkfu commentedon Nov 4, 2025
experimental_verify has existed for at least a decade. It may have been experimental when it was first added but it's just a regular feature with an unfortunate name by now.
I do not know of any efforts currently underway to overhaul it or comprehensively assess it.
backend.c: fix async ioengine prep/queue direction mismatch in experi…