Repository navigation
feat(whisper): add gpu_required to refuse the CPU fallback - #1096
Open
someonewithpc wants to merge 3 commits into
Open
someonewithpc wants to merge 3 commits into
someonewithpc wants to merge 3 commits into
Conversation
When a GPU build's backend enumerates no device, whisper.cpp loads the model on the CPU without an error. After a GPU failed to initialize, large-v3-turbo on Vulkan took 133 s for a 1.5 s clip instead of under half a second, and nothing said why With [whisper] gpu_required = true (VOXTYPE_GPU_REQUIRED, --gpu-required), WhisperTranscriber::new reads ggml's device list before loading the model and fails with GpuUnavailable when it holds no GPU or integrated GPU, or not the one gpu_device asks for. That follows whisper_backend_init_gpu, so the check refuses exactly the loads that would have landed on the CPU. The decision is a pure function over device kinds, tested without a GPU The device list comes from ggml_backend_dev_count and friends through whisper-rs's raw-api feature, which only re-exports whisper-rs-sys. A CPU-only build never asks whisper.cpp for a GPU, so it always fails the check. Other engines don't read the option
ggml lists its devices once per process (peteonrails#611), so a daemon that starts without a GPU will never get one, preloaded or not. With gpu_required set, the daemon now checks the device list at startup, after the gpu_wait_secs wait, with on_demand_loading as well. Without a GPU it logs a warning, skips loading the model and keeps running. When a recording would start, from the hotkey or `voxtype record start`, it shows a notification and exits with an error instead, and the systemd unit's Restart=on-failure starts a fresh process that looks again. The user only hears about it when they try to dictate The notification goes through send_notification with the configured engine icon and critical urgency, like the streaming error. A model load that fails with GpuUnavailable at recording time gets the same notification next to the error sound it already played The device list is logged once per process, since the startup check and the model load would otherwise both print it gpu_isolation is left out of the startup check: its workers are fresh processes that do their own check, and enumerating devices in the daemon would open the GPU driver there
…rker With gpu_isolation the model loads in a worker process, and a load error reaches the daemon as a string in InitFailed, so the daemon could not tell a missing GPU from a missing model file The worker now flags a GpuUnavailable error with gpu_unavailable in its JSON reply, and SubprocessTranscriber turns that back into GpuUnavailable. The daemon shows the no-GPU notification when a transcription fails with it. The field is left out of other replies, and a parent that doesn't know it ignores it SubprocessTranscriber also passes --gpu-required to the worker. The config file and VOXTYPE_GPU_REQUIRED reach it already, but the daemon's CLI flag didn't
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a GPU build finds no GPU, whisper.cpp loads the model on the CPU without telling the user. My eGPU dropped off, the laptop GPU failed to initialize, and Voxtype (Vulkan build) restarted with the NVIDIA ICD selected. whisper.cpp logged
devices = 1, backends = 1andCPU total size = 573.40 MBforlarge-v3-turboq5_0, where a normal start logsdevices = 2, backends = 2andVulkan0 total size = 573.40 MB. A 1.5 s clip then took 133 s to transcribe, and a 136 s dictation was still going after 9 minutes. On the GPU a dictation takes 0.2 to 0.5 sThis adds a
[whisper] gpu_requiredoption. It's off by default, so nothing changes unless you set itVOXTYPE_GPU_REQUIREDand--gpu-requiredcan override itBefore loading the model, Voxtype reads ggml's device list through whisper-rs's
raw-apifeature (ggml_backend_dev_count,ggml_backend_dev_type). If the list has no GPU or integrated GPU, or lacks the onegpu_devicenames, it fails with a newTranscribeError::GpuUnavailable.whisper_backend_init_gpuuses the same rule, so the check refuses the loads that would have gone to the CPU and nothing else, and it doesn't parse log output. The decision is a small function over device kinds, so its unit tests run without a GPUWith no GPU:
gpu_wait_secswait, with or withouton_demand_loading. It logs a warning, skips loading the model and keeps running, so nothing pops up while nobody is dictating. When a recording would start, from the hotkey orvoxtype record start, it shows a notification and exits with an error instead of recording. ggml lists devices once per process ([Bug] Whisper GPU backend is chosen once at first model load; a not-yet-ready GPU pins the daemon to CPU until restart (Linux) #611), so a daemon that started without a GPU would never see one appear. The systemd unit'sRestart=on-failurestarts it again 5 s later, and the new process checks againgpu_isolation, a new worker loads the model for every recording, so the daemon skips the startup check and never opens the GPU driver itself. The worker runs the check and flags the error in its JSON reply. The daemon shows the same notification and discards the recording. It also passes--gpu-requiredon to the workervoxtype transcribeprints the error and exits 1The notification uses
send_notificationwith critical urgency, as the streaming error notification does, and honorsshow_engine_iconfrom[output.notification]. Error notifications have no on/off switch today, so I didn't add oneThe option only covers local Whisper. A CPU-only build can't satisfy it, so it fails the check too. Parakeet, Moonshine and the other ONNX engines pick execution providers through ONNX Runtime, and the GGUF engines run transcribe.cpp in their own worker. They ignore the option, and each would need its own check
The three commits add the option and the check, the daemon's startup check and exit at the next dictation, and the gpu_isolation worker plumbing
Type of Change
Testing
cargo testand all tests passcargo clippy -- -D warningswith no warningscargo fmtDocumentation
Pre-merge checklist
devbranch (notmain)ci-successto pass before marking Ready for Review (draft mode is encouraged while CI is red)Additional Notes
I tested a
gpu-vulkanbuild on the same machine while both NVIDIA GPUs were still down, withlarge-v3-turboq5_0 and a 5.7 s recording.VK_DRIVER_FILESpointing atnvidia_icd.json, which loads but finds no GPU.voxtype transcribelogsggml devices: CPU (Cpu)and fails at once. The daemon logs the warning at startup, posts nothing and stays up. About 0.1 s aftervoxtype record startit exits with status 1, with or withouton_demand_loading, and posts one notification with critical urgency and the engine icon. Withgpu_isolationthe worker replies{"ok":false,"error":"No GPU found: the GPU backend enumerated no devices","gpu_unavailable":true}, andvoxtype transcribereports the GPU error rather than a generic load failureggml devices: Vulkan0 (Gpu), CPU (Cpu)once and loads the model onVulkan0. The daemon reaches "ready for voice input" and keeps running aftervoxtype record start, andvoxtype transcribereturns the text