DEV Community

#vllm

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
vLLM's --api-key leaves /invocations, /tokenize and /pause open. Lock it down with an allowlist proxy
munimv2 profile

vLLM's --api-key leaves /invocations, /tokenize and /pause open. Lock it down with an allowlist proxy

#vllm #llm #security #abotwrotethis
Comments
3 min read
Model Distillation & Quantization: Running 8B Parameter Models at a Fraction of Cloud Cost
mtahir27 profile

Model Distillation & Quantization: Running 8B Parameter Models at a Fraction of Cloud Cost

#modeldistillation #quantization #vllm #awq
Comments
10 min read
vLLM 0.30 and older: 30+ new advisories, most with no CVE ID, and your scanner can't see them
munimv2 profile

vLLM 0.30 and older: 30+ new advisories, most with no CVE ID, and your scanner can't see them

#vllm #security #selfhosted #abotwrotethis
Comments 2
4 min read
Running vLLM natively on Windows
aivrar profile
aivrar

Running vLLM natively on Windows

#vllm #windows #cuda #llm
Comments
6 min read
Small Apache-2.0 Models Against Gemma 4 E2B on One TPU v5e: Which Ones Keep Up?

Rigorous 14 GiB TPU v5e memory budget constraints
xbill profile

Small Apache-2.0 Models Against Gemma 4 E2B on One TPU v5e: Which Ones Keep Up?

#tpu #vllm #gemma #llm
21
Comments 3
10 min read
KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly
wolfsea2357 profile

KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly

#llm #vllm #performance #devops
Comments
7 min read
What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You
wolfsea2357 profile

What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You

#llm #vllm #performance #devops
Comments
6 min read
Five Ways Your LLM Serving Benchmark Is Lying to You (and How to Catch Each One)
wolfsea2357 profile

Five Ways Your LLM Serving Benchmark Is Lying to You (and How to Catch Each One)

#llm #vllm #performance #devops
Comments 1
15 min read
vLLM ignores LoRA rank_pattern and alpha_pattern and serves the adapter at the wrong scale
homelabpm profile

vLLM ignores LoRA rank_pattern and alpha_pattern and serves the adapter at the wrong scale

#llm #vllm #lora #selfhosted
Comments
5 min read
vLLM's weight cache can serve another checkpoint's weights when the tensor layout matches
homelabpm profile

vLLM's weight cache can serve another checkpoint's weights when the tensor layout matches

#llm #vllm #selfhosted #debugging
Comments
4 min read
Qwen3.8-27B on one RTX 3090, and the setting its README says chat clients should leave at default
renolu profile

Qwen3.8-27B on one RTX 3090, and the setting its README says chat clients should leave at default

#vllm #localllm #speculativedecoding #quantization
Comments
4 min read
Evaluating Speculative Decoding in vLLM on AMD MI300X GPUs
rammehta1899 profile

Evaluating Speculative Decoding in vLLM on AMD MI300X GPUs

#vllm #amd #mi300x #rocm
Comments
1 min read
Gemma 4 on Amazon SageMaker: 4-Bit Embeddings Decode up to 1.39x Faster on One L4
xbill profile

Gemma 4 on Amazon SageMaker: 4-Bit Embeddings Decode up to 1.39x Faster on One L4

#aws #sagemaker #gemma #vllm
1
Comments
11 min read
Can You Run vLLM on Windows? Yes — via WSL2
mrsaynothing profile

Can You Run vLLM on Windows? Yes — via WSL2

#ai #vllm #windows #wsl2
5
Comments 2
4 min read
PCIe 4.0 x8/x8 vs x16/x16 for Dual GPUs: Does Bandwidth Choke Local LLM Tensor Parallelism?
abdulrahman_maslmany profile

PCIe 4.0 x8/x8 vs x16/x16 for Dual GPUs: Does Bandwidth Choke Local LLM Tensor Parallelism?

#ai #machinelearning #hardware #vllm
1
Comments 1
8 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.