DEV Community

#vllm

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap
wolfsea2357 profile

vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap

#ai #machinelearning #llm #vllm
Comments
14 min read
The Local vLLM Speculative Decoding Blueprint: How I Quadrupled DeepSeek-R1 Inference to 48 Tokens/Sec on Consumer GPUs with...
lifehacknexus profile

The Local vLLM Speculative Decoding Blueprint: How I Quadrupled DeepSeek-R1 Inference to 48 Tokens/Sec on Consumer GPUs with...

#aitoolsproductivity #aitools #deepseekr1 #vllm
Comments
14 min read
Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data
wolfsea2357 profile

Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data

#ai #machinelearning #vllm #llm
Comments 1
13 min read
FP8 Cut Our TPOT by 25%. At High Concurrency, It Also Changed the Answers.
huilin_ye_f804826bf674576 profile

FP8 Cut Our TPOT by 25%. At High Concurrency, It Also Changed the Answers.

#llm #vllm #quantization #benchmarking
Comments
3 min read
Architecting On-Premise Mistral Large 4 for Enterprise SLAs: A Systems Blueprint
burra_harish_d574a4c7327a profile

Architecting On-Premise Mistral Large 4 for Enterprise SLAs: A Systems Blueprint

#architecture #ai #vllm #infrastructure
Comments
4 min read
Same RTX 3090, 22x cheaper per token: what concurrency does to vLLM cost
qaisermehdi profile

Same RTX 3090, 22x cheaper per token: what concurrency does to vLLM cost

#vllm #llm #gpu #mlops
Comments
2 min read
RTX 3090 vs A100 for a 7B model on vLLM: the cheaper GPU depends on your answer time
qaisermehdi profile

RTX 3090 vs A100 for a 7B model on vLLM: the cheaper GPU depends on your answer time

#vllm #llm #mlops #gpu
Comments 1
2 min read
vLLM Constrained Decoding: Fix Tool Call Schema Errors
rammehta1899 profile

vLLM Constrained Decoding: Fix Tool Call Schema Errors

#vllm #llmserving #agenticworkflows #constraineddecoding
Comments
1 min read
Your Local LLM Isn't Bad at Tool Calling, Your Pipeline Is: Fixing Tool-Call Accuracy in vLLM
eme_gug_0821b41b948be6516 profile

Your Local LLM Isn't Bad at Tool Calling, Your Pipeline Is: Fixing Tool-Call Accuracy in vLLM

#llm #vllm #python #ai
Comments
7 min read
llm-d vs Plain vLLM: A Practical Kubernetes Inference Quickstart
jamilxt profile

llm-d vs Plain vLLM: A Practical Kubernetes Inference Quickstart

#llmd #vllm #kubernetes #ai
1
Comments
8 min read
KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off
wolfsea2357 profile

KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off

#ai #machinelearning #vllm #llm
Comments
14 min read
Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax
wolfsea2357 profile

Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax

#ai #machinelearning #vllm #performance
Comments
12 min read
Why LLM Inference Engines Are Fast: Continuous Batching, PagedAttention and Prefix Caching Explained with vLLM
eme_gug_0821b41b948be6516 profile

Why LLM Inference Engines Are Fast: Continuous Batching, PagedAttention and Prefix Caching Explained with vLLM

#llm #vllm #inference #performance
Comments
7 min read
A 3090 vs an L40S: 1.09x at batch 1, 0.46x at batch 128
qaisermehdi profile

A 3090 vs an L40S: 1.09x at batch 1, 0.46x at batch 128

#llm #gpu #vllm #benchmark
Comments
2 min read
vLLM on 4 GPUs / 64GB: A Multi-Model Benchmark — For a Home Multi-Host Setup, I'd Go with vLLM
faliye profile

vLLM on 4 GPUs / 64GB: A Multi-Model Benchmark — For a Home Multi-Host Setup, I'd Go with vLLM

#llm #ai #vllm #gpu
2
Comments 1
8 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.