You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Pre-download mini-benchmark for in-browser (LLM) inference performance #5468
Describe the feature and the current behaviour/state
At the moment, for Gen AI use cases in the browser e.g. Gemma 2B with the MediaPipe LLM Inference API, there's no way for a developer to know ahead of time whether the model can actually run on the device within reasonable times. This is an issue because:
For Gen AI, the model download is really large (1.3GB almost for Gemma 2B, which is manyfold the recommended web app size)
Running an inference on devices that have low spec or too much operations already running may be really slow, or even crash a device (on mobile).
This leads to a subpar UX where a user may have waited to download a large model that can't actually run inferences within reasonable times on their device, or that may even crash their device.
What if we ran a mini-benchmark ahead of model download? This is beaufortfrancois@'s idea he suggested for Transformers.js: huggingface/transformers.js#545 (comment).
This would involve running the model code with zeroed-out weights.
Will this change the current API? How?
Yes, as we'd want to expose to developers the output of the mini-benchmark. This output may be abstracted behind a few dev-friendly performance buckets e.g. high, medium, low. Developers could overlay their own logic based on that output.
Hello @maudnals, thanks for this suggestion! We are still in a pretty early phase for our API, and what you have described is definitely on our roadmap. We need to outline our hardware support in our documentation, including expected performance numbers. I do think that an automatic validation during the client setup can help here as well.
Activity
schmidt-sebastian commentedon Oct 31, 2024
Hello @maudnals, thanks for this suggestion! We are still in a pretty early phase for our API, and what you have described is definitely on our roadmap. We need to outline our hardware support in our documentation, including expected performance numbers. I do think that an automatic validation during the client setup can help here as well.
File internal bug b/376673043