Skip to content

workqueue: fix threads stall when running with io_submit_mode offload - #2053

Merged
axboe merged 1 commit into
axboe:masterfrom
tomas-winkler-sndk:offload_fix
Feb 5, 2026
Merged

axboe merged 1 commit into
axboe:masterfrom
tomas-winkler-sndk:offload_fix

Conversation

@tomas-winkler-sndk

Copy link
Copy Markdown
Contributor

When starting a submit worker, start_worker() sets the worker state to IDLE immediately after pthread_create():

sw->flags = SW_F_IDLE;

However, worker_thread() may run and set SW_F_RUNNING very early in its own startup path:

pthread_mutex_lock(&sw->lock);
    sw->flags |= SW_F_RUNNING;
    if (ret)
        sw->flags |= SW_F_ERROR;
    pthread_mutex_unlock(&sw->lock);

If worker_thread() wins the race and sets SW_F_RUNNING before start_worker() assigns SW_F_IDLE, the unconditional assignment in start_worker() clobbers previously set bits and leaves the worker flags as only SW_F_IDLE. As a result, workqueue_init() waits forever for all workers to report SW_F_RUNNING:

  thread_main()
    -> rate_submit_init()
      -> workqueue_init()
         ...
         running = 0;
         for (i = 0; i < wq->max_workers; i++) {
             pthread_mutex_lock(&sw->lock);

             if (sw->flags & SW_F_RUNNING)
                 running++;
             if (sw->flags & SW_F_ERROR)
                 error++;
             pthread_mutex_unlock(&sw->lock);
         }

         if (error || running == wq->max_workers)
             break;

         pthread_cond_wait(&wq->flush_cond, &wq->flush_lock);

Because SW_F_RUNNING was overwritten, running never reaches
wq->max_workers, and the init loop blocks on flush_cond indefinitely.

Please confirm that your commit message(s) follow these guidelines:

  1. First line is a commit title, a descriptive one-liner for the change
  2. Empty second line
  3. Commit message body that explains why the change is useful. Break lines that
    aren't something like a URL at 72-74 chars.
  4. Empty line
  5. Signed-off-by: Real Name real@email.com

Reminders:

  1. If you modify struct thread_options, also make corresponding changes in
    cconv.c and bump FIO_SERVER_VER in server.h
  2. If you change the ioengine interface (hooks, flags, etc), remember to bump
    FIO_IOOPS_VERSION in ioengines.h.

When starting a submit worker, start_worker() sets the worker state to
IDLE immediately after pthread_create():

sw->flags = SW_F_IDLE;

However, worker_thread() may run and set SW_F_RUNNING very early in its
own startup path:

	pthread_mutex_lock(&sw->lock);
        sw->flags |= SW_F_RUNNING;
        if (ret)
            sw->flags |= SW_F_ERROR;
        pthread_mutex_unlock(&sw->lock);

If worker_thread() wins the race and sets SW_F_RUNNING before
start_worker() assigns SW_F_IDLE, the unconditional assignment in
start_worker() clobbers previously set bits and leaves the worker flags
as *only* SW_F_IDLE. As a result, workqueue_init() waits forever for all
workers to report SW_F_RUNNING:

      thread_main()
        -> rate_submit_init()
          -> workqueue_init()
             ...
             running = 0;
             for (i = 0; i < wq->max_workers; i++) {
                 pthread_mutex_lock(&sw->lock);

                 if (sw->flags & SW_F_RUNNING)
                     running++;
                 if (sw->flags & SW_F_ERROR)
                     error++;
                 pthread_mutex_unlock(&sw->lock);
             }

             if (error || running == wq->max_workers)
                 break;

             pthread_cond_wait(&wq->flush_cond, &wq->flush_lock);

Because SW_F_RUNNING was overwritten, `running` never reaches
    `wq->max_workers`, and the init loop blocks on `flush_cond` indefinitely.

Signed-off-by: Chana Zaks <chana.zaks@sandisk.com>
Signed-off-by: Tomas Winkler <tomas.winkler@sandisk.com>
@axboe

axboe commented Feb 5, 2026

Copy link
Copy Markdown
Owner

Nice work! Great description and the fix looks obviously correct.

@axboe
axboe merged commit a8ab726 into axboe:master Feb 5, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

⌂ 换网址