Skip to content

When using libaio/io_uring engine and executing experimental_verify FIO generates writes instead of reads #1955

Closed

Description

@mariuszbarczak

Description of the bug:
When executing verify_only with experimental_verify, FIO generates writes to the device/file.

Environment: Linux using libaio engine

fio version: 3.33, but it might be valid for master

Reproduction steps
Run FIO with verify_only together with experimental_verify. We expect FIO to perform only READ operations for data verification. However, we are observing only WRITE operations. The issue might be related to the experimental_verify procedure: when it changes the I/O direction (from WRITE to READ), it may not trigger the prep io function in the I/O engine

FIO config

[global] 
ioengine=libaio 
iodepth=32 
direct=1 
readwrite=randwrite 
filename=/dev/vda 
serialize_overlap=1 
experimental_verify=1 
do_verify=1 
verify=md5 
group_reporting 
verify_only 
norandommap

[job] 

Activity

  1. mariuszbarczak commented on Jul 31, 2025

    @mariuszbarczak
    Author

    Update: Confirmed, this is observed on the master branch too.
    Note: FIO reports reads, however iostat shows writes

  2. mmkayPL commented on Jul 31, 2025

    @mmkayPL
    Contributor

    In addition, the issue also occurs in sequential workloads, or in random workloads with or without the norandommap option:

    ioengine=libaio
    iodepth=32
    direct=1
    readwrite=write
    filename=/dev/nvme0n1
    experimental_verify=1
    do_verify=1
    verify=md5
    group_reporting
    size=10GiB
    [job]
    

    In all cases, once fio gets to the verification phase it will report that it switched to issuing reads, but checking via iostat shows that it's only sending writes to the tested device.

  3. sitsofe commented on Aug 9, 2025

    @sitsofe
    Collaborator

    @mariuszbarczak, @mmkayPL OK let's try and cut this down further.

    • Can you make this happen with a sequential write (rw=write) workload?
    • Can you make it happen with a single write by limit the size (size=4k)?
    • Can you make it happen with ioengine=sync?
    • Can you make it happen with direct=1 (although doing this may be tough if you detecting writes by device stats as opposed to through some sort of trace)
    • Can you make it happen without group_reporting?
    • Can you make it happen without serialize_overlap=1?
    • Can you make it happen with a plain file?

    After removing all the above the final questions is: can you make it happen without experimental_verify=1? [mmkayPL has stated below that this option is required to reproduce the issue]

    The reason for doing all the above is that we always want the minimum number of options that make a problem happen so it's easier for other to reproduce and so there are less places to look in.

  4. sitsofe commented on Aug 9, 2025

    @sitsofe
    Collaborator

    @mmkayPL your job is going to do writes because by default fio does the write job followed by a separate read verification pass (see do_verify). The job @mariuszbarczak gave sets verify_only so I would not expect that to do any writes.

  5. mmkayPL commented on Aug 10, 2025

    @mmkayPL
    Contributor

    @mmkayPL your job is going to do writes because by default fio does the write job followed by a separate read verification pass (see do_verify). The job @mariuszbarczak gave sets verify_only so I would not expect that to do any writes.

    Yes, I'm aware that without verify_only there are two passes (a write and then read pass) done by fio. The issue is that when in the verification pass (whether via verify_only set to 1, or waiting until the writes are done) even though fio is reporting that it's doing verification/reads and in the end report it's mentioning that it has been doing reads, if you check under debugger/with iostat what is actually being sent to the device it's always writes during the verification phase and fio never actually reads from the device. Running without experimental_verify works as expected (you can see reads with iostat).
    I'll try to answer all other questions tomorrow.

  6. sitsofe commented on Aug 10, 2025

    @sitsofe
    Collaborator

    @mmkayPL OK thanks for reiterating. If you are able to get the run to be small it would also be useful to see the output of fio when using --debug=io as the first command line option and having that uploaded here as a text attachment.

  7. mmkayPL commented on Aug 11, 2025

    @mmkayPL
    Contributor

    So to answer the rest of the questions:

    • Yes the problem occurs on both random and sequential workloads
    • 4096 size is a bit hard to tell, but I do seem to see a tick of read activity with iostat (though it could be udev, I will have to change my test setup a bit to 100% confirm this
    • The issue doesn't appear with ioengine=sync
    • I'm assuming you meant direct=0, in which case as you suspected, it's a bit hard to tell
    • Group reporting setting can be removed, and the issue still appears
    • Serialize overlap setting can be removed, and the issue still appears
    • The issue is reproducible on a plain file
    • The issue appears at iodepth=1

    While I don't think this is the proper fix, we found that adding the following change to engines/libaio.c fio_libaio_queue function (after fio_ro_check) would work:

    if (io_u->ddir == DDIR_READ) {
        io_u->iocb.aio_lio_opcode = IO_CMD_PREAD;
    }
    

    I've been doing these experiments on commit 306d898 (so roughly master at the time of the creation of the issue) and config was cut down to this:

    [global]
    ioengine=libaio
    direct=1
    readwrite=write
    filename=/dev/nvme0n1
    experimental_verify=1
    verify=md5
    size=1G
    [job]
    
  8. sitsofe commented on Aug 11, 2025

    @sitsofe
    Collaborator

    @mmkayPL: Thanks for minimising the test case. Next question: does it happen when using the posixaio or io_uring ioengines?

  9. mmkayPL commented on Aug 12, 2025

    @mmkayPL
    Contributor

    The issue doesn't appear with posixaio, but it does appear with io_uring (and just to doublecheck, io_uring with experimental_verify turned off works correctly).

  10. changed the title [-]When using libaio engine and executing experimental_verify FIO generates writes instead of reads[/-] [+]When using libaio/io_uring engine and executing experimental_verify FIO generates writes instead of reads[/+] on Aug 12, 2025
  11. added and removed
    needreporterinfoWaiting on information from the issue reporter
    on Aug 12, 2025
  12. sitsofe commented on Aug 12, 2025

    @sitsofe
    Collaborator

    I can reproduce the issue with the following:

    sudo modprobe null_blk gb=1
    sudo chmod a+rw /dev/nullb0
    before=($(cat /sys/block/nullb0/stat | awk '{ print $1" "$5 }'))
    fio --debug=io --minimal --ioengine=libaio --direct=1 --rw=write --experimental_verify=1 --verify=md5 --size=4k --filename=/dev/nullb0 --verify_only --name=job; after=($(cat /sys/block/nullb0/stat | awk '{ print $1" "$5 }'))
    echo read=$(( after[0] - before[0] )) write=$(( after[1] - before[1] ))
    

    An excerpt from the output:

    [...]
    fio: only root may flush block devices. Cache flush bypassed!
    fio: cache invalidation of /dev/nullb0 failed: Permission denied
    io       24995 fill: io_u 0x5a3132284c00: off=0x0,len=0x1000,ddir=1,file=/dev/nullb0
    io       24995 prep: io_u 0x5a3132284c00: off=0x0,len=0x1000,ddir=1,file=/dev/nullb0
    io       24995 prep: io_u 0x5a3132284c00: ret=0
    io       24995 complete: io_u 0x5a3132284c00: off=0x0,len=0x1000,ddir=1,file=/dev/nullb0
    io       24995 drop page cache /dev/nullb0
    fio: cache invalidation of /dev/nullb0 failed: Permission denied
    io       24995 fill: io_u 0x5a3132284c00: off=0x0,len=0x1000,ddir=1,file=/dev/nullb0
    io       24995 prep: io_u 0x5a3132284c00: off=0x0,len=0x1000,ddir=1,file=/dev/nullb0
    io       24995 prep: io_u 0x5a3132284c00: ret=0
    io       24995 queue: io_u 0x5a3132284c00: off=0x0,len=0x1000,ddir=0,file=/dev/nullb0
    io       24995 calling ->commit(), depth 1
    io       24995 io_u_queued_complete: min=1
    io       24995 getevents: 1
    io       24995 complete: io_u 0x5a3132284c00: off=0x0,len=0x1000,ddir=0,file=/dev/nullb0
    [...]
    read=0 write=1
    

    So we can see the first (dry run) I/O being filled with a data direction of 1 (which is write) and prepped with a data direction of 1. However we see the second I/O done by verification being filled and prepped with a data direction of 1 and then queued with a data direction of 0 (read) followed by a commit(). You can see that the libaio ioengine sets the I/O's direction in an ancillary structure (iocb which is filled out by io_prep_*() functions) at prep time (see https://github.com/axboe/fio/blob/fio-3.40/engines/libaio.c#L118-L148 ).

    Basically any async ioengine that sets the data direction at prep time (rather than at queue time like the posixaio ioengine does) is going to hit this issue. I think to fix it you would have to move (some/all of?) the "rewrite to the ddir to read" logic to somewhere like get_io_u() because you need to have changed the ddir in both the case where the ioengine doesn't have a .prep (which rules out (only) doing it in td_io_prep()) but before an ioengine's .prep function is called.

  13. added
    triagedIssue cause is understood but a patch is needed to fix it
    patcheswelcomeNo one internal is working on this but an external contributor's help is very welcome!
    and removed on Aug 12, 2025
  14. vincentkfu commented on Oct 30, 2025

    @vincentkfu
    Collaborator

    @sitsofe Does #2007 seem like a reasonable fix to you?

  15. sitsofe commented on Oct 30, 2025

    @sitsofe
    Collaborator

    @vincentkfu Yes! It has a good commit message, it's more elegant than what I proposed and it only has a double prep as the downside.

  16. yehormalikov commented on Nov 3, 2025

    @yehormalikov
    Contributor

    @sitsofe, @vincentkfu,

    We have a question regarding the experimental_verify parameter.
    We currently use fio’s verification feature for data integrity testing, and we’re considering enabling experimental_verify in our verification path.

    Could you please clarify how stable or reliable experimental_verify is at this point?
    Is it safe to rely on it for accurate verification, or would you recommend avoiding it in production tests for now?

  17. sitsofe commented on Nov 3, 2025

    @sitsofe
    Collaborator

    @yehormalikov I'm not the best person to ask as I don't use fio in anger on a regular basis (so you may want to hear from others). From what I know experimental_verify been around for a while but you can see the issue above was only reported and fixed recently so I'd say most people aren't using it. I think there may be cases it handles poorly (verification when multiple passes are done, handling the case where a write I/O was stalled) that regular verification can but I haven't verified whether my supposition is correct. If your production workload is do a single full pass of writes then verify later with no weirdness in-between I would imagine experimental_verify would work well and with a far far lower memory overhead than regular verification.

  18. vincentkfu commented on Nov 4, 2025

    @vincentkfu
    Collaborator

    @ankit-sam and I went through the standard verify feature several months ago fixing issues and adding a test script. So I would have to say that non-experimental verify is better tested. However, if experimental verify fits your use case, then I would be willing to use it in production after validating it. Just make sure that you understand how regular verify and experimental verify differ. Regular verify saves all the IOs in a data structure for the verification phase whereas experimental_verify resets the random seeds and re-generates the sequence of offsets for the verification phase. There can be issues like the overwrites that @sitsofe mentioned where experimental_verify may fail.

  19. yehormalikov commented on Nov 4, 2025

    @yehormalikov
    Contributor

    @ankit-sam and I went through the standard verify feature several months ago fixing issues and adding a test script. So I would have to say that non-experimental verify is better tested. However, if experimental verify fits your use case, then I would be willing to use it in production after validating it. Just make sure that you understand how regular verify and experimental verify differ. Regular verify saves all the IOs in a data structure for the verification phase whereas experimental_verify resets the random seeds and re-generates the sequence of offsets for the verification phase. There can be issues like the overwrites that @sitsofe mentioned where experimental_verify may fail.

    Thank you both for the detailed clarifications. We understand how experimental_verify operates and appreciate its advantages - particularly the fact that it avoids storing I/O metadata in memory, which makes it much more scalable for large-capacity tests.

    That said, the “experimental” designation makes us somewhat cautious to rely on it in production validation flows. We’d like to understand whether there are any ongoing efforts or future plans to stabilize and promote experimental_verify to a some standard verification parameter, or if it’s expected to remain in its current state for the foreseeable future.

    Any insights on its long-term roadmap or intended level of support would be greatly appreciated.

  20. vincentkfu commented on Nov 4, 2025

    @vincentkfu
    Collaborator

    experimental_verify has existed for at least a decade. It may have been experimental when it was first added but it's just a regular feature with an unfortunate name by now.
    I do not know of any efforts currently underway to overhaul it or comprehensively assess it.

  21. removed
    triagedIssue cause is understood but a patch is needed to fix it
    patcheswelcomeNo one internal is working on this but an external contributor's help is very welcome!
    on Nov 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址