vLLM

by vLLM project

The go-to high-throughput inference server for serving LLMs to many users.

93.6 k stars GitHub Apache-2.0 Free when localLocalOpen sourceOpen weights

View on GitHub →

What it does

PagedAttention and continuous batching for high throughput
Production-ready OpenAI-compatible API
Supports most open models on Hugging Face

Install it

Requirements: Linux, Python and a supported GPU (NVIDIA CUDA recommended).

Install

uv pip install vllm

Quickstart guide

https://docs.vllm.ai/en/latest/

The prompt for your agent

Paste it into your coding agent: it checks the requirements and installs it for you.

Claude CodeCodexCursorGemini CLIopencode

Prompt

Install vLLM (https://vllm.ai) on my machine.
1. Check the requirements: Linux, Python and a supported GPU (NVIDIA CUDA recommended). If anything is missing, tell me before installing anything.
2. Follow the official method:
- Install: uv pip install vllm
- see https://docs.vllm.ai/en/latest/
3. Check that it works, then explain how to launch it and where to start.

Always review what your agent is about to run, especially install commands.