Skip to content

feat(whisper): add gpu_required to refuse the CPU fallback - #1096

Open
someonewithpc wants to merge 3 commits into
peteonrails:devfrom
someonewithpc:feat/whisper-gpu-required
Open

someonewithpc wants to merge 3 commits into
peteonrails:devfrom
someonewithpc:feat/whisper-gpu-required

Conversation

@someonewithpc

Copy link
Copy Markdown

When a GPU build finds no GPU, whisper.cpp loads the model on the CPU without telling the user. My eGPU dropped off, the laptop GPU failed to initialize, and Voxtype (Vulkan build) restarted with the NVIDIA ICD selected. whisper.cpp logged devices = 1, backends = 1 and CPU total size = 573.40 MB for large-v3-turbo q5_0, where a normal start logs devices = 2, backends = 2 and Vulkan0 total size = 573.40 MB. A 1.5 s clip then took 133 s to transcribe, and a 136 s dictation was still going after 9 minutes. On the GPU a dictation takes 0.2 to 0.5 s

This adds a [whisper] gpu_required option. It's off by default, so nothing changes unless you set it

[whisper]
gpu_required = true

VOXTYPE_GPU_REQUIRED and --gpu-required can override it

Before loading the model, Voxtype reads ggml's device list through whisper-rs's raw-api feature (ggml_backend_dev_count, ggml_backend_dev_type). If the list has no GPU or integrated GPU, or lacks the one gpu_device names, it fails with a new TranscribeError::GpuUnavailable. whisper_backend_init_gpu uses the same rule, so the check refuses the loads that would have gone to the CPU and nothing else, and it doesn't parse log output. The decision is a small function over device kinds, so its unit tests run without a GPU

With no GPU:

  • The daemon checks at startup, after the gpu_wait_secs wait, with or without on_demand_loading. It logs a warning, skips loading the model and keeps running, so nothing pops up while nobody is dictating. When a recording would start, from the hotkey or voxtype record start, it shows a notification and exits with an error instead of recording. ggml lists devices once per process ([Bug] Whisper GPU backend is chosen once at first model load; a not-yet-ready GPU pins the daemon to CPU until restart (Linux) #611), so a daemon that started without a GPU would never see one appear. The systemd unit's Restart=on-failure starts it again 5 s later, and the new process checks again
  • With gpu_isolation, a new worker loads the model for every recording, so the daemon skips the startup check and never opens the GPU driver itself. The worker runs the check and flags the error in its JSON reply. The daemon shows the same notification and discards the recording. It also passes --gpu-required on to the worker
  • When a model load fails this way at recording time, the notification comes with the error sound the daemon already plays
  • voxtype transcribe prints the error and exits 1

The notification uses send_notification with critical urgency, as the streaming error notification does, and honors show_engine_icon from [output.notification]. Error notifications have no on/off switch today, so I didn't add one

The option only covers local Whisper. A CPU-only build can't satisfy it, so it fails the check too. Parakeet, Moonshine and the other ONNX engines pick execution providers through ONNX Runtime, and the GGUF engines run transcribe.cpp in their own worker. They ignore the option, and each would need its own check

The three commits add the option and the check, the daemon's startup check and exit at the next dictation, and the gpu_isolation worker plumbing

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Testing

  • I have tested these changes locally
  • I have run cargo test and all tests pass
  • I have run cargo clippy -- -D warnings with no warnings
  • I have run cargo fmt

Documentation

  • I have updated documentation as needed
  • No documentation changes are needed

Pre-merge checklist

  • This PR targets the dev branch (not main)
  • Wait for ci-success to pass before marking Ready for Review (draft mode is encouraged while CI is red)

Additional Notes

I tested a gpu-vulkan build on the same machine while both NVIDIA GPUs were still down, with large-v3-turbo q5_0 and a 5.7 s recording.

  • NVIDIA ICD, with VK_DRIVER_FILES pointing at nvidia_icd.json, which loads but finds no GPU. voxtype transcribe logs ggml devices: CPU (Cpu) and fails at once. The daemon logs the warning at startup, posts nothing and stays up. About 0.1 s after voxtype record start it exits with status 1, with or without on_demand_loading, and posts one notification with critical urgency and the engine icon. With gpu_isolation the worker replies {"ok":false,"error":"No GPU found: the GPU backend enumerated no devices","gpu_unavailable":true}, and voxtype transcribe reports the GPU error rather than a generic load failure
  • Radeon iGPU (RADV). Voxtype logs ggml devices: Vulkan0 (Gpu), CPU (Cpu) once and loads the model on Vulkan0. The daemon reaches "ready for voice input" and keeps running after voxtype record start, and voxtype transcribe returns the text

When a GPU build's backend enumerates no device, whisper.cpp loads the
model on the CPU without an error. After a GPU failed to initialize,
large-v3-turbo on Vulkan took 133 s for a 1.5 s clip instead of under
half a second, and nothing said why

With [whisper] gpu_required = true (VOXTYPE_GPU_REQUIRED,
--gpu-required), WhisperTranscriber::new reads ggml's device list
before loading the model and fails with GpuUnavailable when it holds no
GPU or integrated GPU, or not the one gpu_device asks for. That follows
whisper_backend_init_gpu, so the check refuses exactly the loads that
would have landed on the CPU. The decision is a pure function over
device kinds, tested without a GPU

The device list comes from ggml_backend_dev_count and friends through
whisper-rs's raw-api feature, which only re-exports whisper-rs-sys.
A CPU-only build never asks whisper.cpp for a GPU, so it always fails
the check. Other engines don't read the option
ggml lists its devices once per process (peteonrails#611), so a daemon that starts
without a GPU will never get one, preloaded or not. With gpu_required
set, the daemon now checks the device list at startup, after the
gpu_wait_secs wait, with on_demand_loading as well. Without a GPU it
logs a warning, skips loading the model and keeps running. When a
recording would start, from the hotkey or `voxtype record start`, it
shows a notification and exits with an error instead, and the systemd
unit's Restart=on-failure starts a fresh process that looks again. The
user only hears about it when they try to dictate

The notification goes through send_notification with the configured
engine icon and critical urgency, like the streaming error. A model
load that fails with GpuUnavailable at recording time gets the same
notification next to the error sound it already played

The device list is logged once per process, since the startup check
and the model load would otherwise both print it

gpu_isolation is left out of the startup check: its workers are fresh
processes that do their own check, and enumerating devices in the
daemon would open the GPU driver there
…rker

With gpu_isolation the model loads in a worker process, and a load
error reaches the daemon as a string in InitFailed, so the daemon could
not tell a missing GPU from a missing model file

The worker now flags a GpuUnavailable error with gpu_unavailable in its
JSON reply, and SubprocessTranscriber turns that back into
GpuUnavailable. The daemon shows the no-GPU notification when a
transcription fails with it. The field is left out of other replies,
and a parent that doesn't know it ignores it

SubprocessTranscriber also passes --gpu-required to the worker. The
config file and VOXTYPE_GPU_REQUIRED reach it already, but the daemon's
CLI flag didn't

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ 换网址