ExLlamaV3

by turboderp

A fast inference library to fit large LLMs on consumer graphics cards.

1,643 stars GitHub MIT Free when localLocalOpen sourceOpen weights

View on GitHub →

What it does

Memory-efficient EXL3 quantization (based on QTIP)
Multi-GPU parallel inference and CPU offloading for MoE
OpenAI-compatible server through TabbyAPI

Install it

Requirements: NVIDIA GPU, Python and PyTorch with CUDA 12.4+ already installed.

Pick a prebuilt wheel (recommended)

https://github.com/turboderp-org/exllamav3/releases

Or from PyPI (builds the extension)

pip install exllamav3

Recommended server: TabbyAPI

https://github.com/theroyallab/tabbyAPI

The prompt for your agent

Paste it into your coding agent: it checks the requirements and installs it for you.

Claude CodeCodexCursorGemini CLIopencode

Prompt

Install ExLlamaV3 (https://github.com/turboderp-org/exllamav3) on my machine.
1. Check the requirements: NVIDIA GPU, Python and PyTorch with CUDA 12.4+ already installed. If anything is missing, tell me before installing anything.
2. Follow the official method:
- see https://github.com/turboderp-org/exllamav3/releases
- Or from PyPI (builds the extension): pip install exllamav3
- see https://github.com/theroyallab/tabbyAPI
3. Check that it works, then explain how to launch it and where to start.

Always review what your agent is about to run, especially install commands.