High-throughput LLM serving
The standard open-source inference engine: PagedAttention serving models at production scale.
UC Berkeley's LLM serving engine: PagedAttention turned KV-cache into an OS problem and unlocked massive throughput. The open-source standard for self-hosted inference
Who it's for
ML engineers serving LLMs at the highest throughput for batch and production inference.
Official links
Keep exploring
Concepts in the glossary
1 in catalog
vLLM is written in
1 in catalog
vLLM implements
1 in catalog