DEV Community

#vllm

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
Why vLLM and SGLang might not be the best option to run AI agents
nahuel990 profile

Why vLLM and SGLang might not be the best option to run AI agents

#ai #agents #vllm #rust
3
Comments 1
4 min read
PagedAttention and the KV Cache, Plainly: Why Memory, Not FLOPs, Caps Your Concurrency
danielsamfdo profile

PagedAttention and the KV Cache, Plainly: Why Memory, Not FLOPs, Caps Your Concurrency

#vllm #llm #googlecloud #machinelearning
Comments
11 min read
llm-compressor for vLLM: The FP8/AWQ Deployment Tradeoff
jangwook_kim_e31e7291ad98 profile

llm-compressor for vLLM: The FP8/AWQ Deployment Tradeoff

#quantization #vllm #fp8 #awq
Comments
9 min read
vLLM 0.33, Plainly: CUDA 13, Model Runner V2, and What Breaks on Upgrade
danielsamfdo profile

vLLM 0.33, Plainly: CUDA 13, Model Runner V2, and What Breaks on Upgrade

#vllm #llm #googlecloud #machinelearning
Comments
9 min read
Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack
wolfsea2357 profile

Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack

#ai #opensource #llm #vllm
1
Comments 1
15 min read
verl GRPO + vLLM Rollouts: GPU Memory Settings for Single-Node Deployment
jangwook_kim_e31e7291ad98 profile

verl GRPO + vLLM Rollouts: GPU Memory Settings for Single-Node Deployment

#rlposttraining #grpo #vllm #gpumemory
Comments
12 min read
vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe
wolfsea2357 profile

vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe

#ai #machinelearning #llm #vllm
Comments
15 min read
Managing Agent GPU Kernels: vLLM's Kimi K3 Fast-Track RFC
rammehta1899 profile

Managing Agent GPU Kernels: vLLM's Kimi K3 Fast-Track RFC

#vllm #gpukernels #cutedsl #codereview
Comments
1 min read
vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap
wolfsea2357 profile

vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap

#ai #python #llm #vllm
Comments 1
13 min read
vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap
wolfsea2357 profile

vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap

#ai #machinelearning #llm #vllm
Comments
14 min read
The Local vLLM Speculative Decoding Blueprint: How I Quadrupled DeepSeek-R1 Inference to 48 Tokens/Sec on Consumer GPUs with...
lifehacknexus profile

The Local vLLM Speculative Decoding Blueprint: How I Quadrupled DeepSeek-R1 Inference to 48 Tokens/Sec on Consumer GPUs with...

#aitoolsproductivity #aitools #deepseekr1 #vllm
Comments
14 min read
Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data
wolfsea2357 profile

Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data

#ai #machinelearning #vllm #llm
Comments 1
13 min read
FP8 Cut Our TPOT by 25%. At High Concurrency, It Also Changed the Answers.
huilin_ye_f804826bf674576 profile

FP8 Cut Our TPOT by 25%. At High Concurrency, It Also Changed the Answers.

#llm #vllm #quantization #benchmarking
Comments
3 min read
Architecting On-Premise Mistral Large 4 for Enterprise SLAs: A Systems Blueprint
burra_harish_d574a4c7327a profile

Architecting On-Premise Mistral Large 4 for Enterprise SLAs: A Systems Blueprint

#architecture #ai #vllm #infrastructure
Comments
4 min read
Six Serving Engines on One Cloud TPU v5e Chip: Gemma 4 E2B

SGLang-JAX wins the speed test
xbill profile

Six Serving Engines on One Cloud TPU v5e Chip: Gemma 4 E2B

#googlecloud #tpu #vllm #jax
26
Comments 5
12 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
⌂ 换网址