Directory › Local AI › AirLLM
> GEM LOCAL AI · DEVELOPMENT
AirLLM
by Gavin Li (lyogavin)
“A 70B on 4 GB of VRAM”: true, but counted in seconds per word, not for chatting.
View on GitHub →What it does
- Loads one layer at a time: the full model stays on disk and streams through, with no quality loss
- Speed: 5–35 seconds per word on a good SSD, over 8 hours for 100 words in the worst reported cases
- Useful for overnight batch jobs or testing a model at full precision; useless for a chat
! DO NOT use it to chat with a model or for volume: every word re-reads the whole model from disk (SSD wear, machine unusable meanwhile). For normal use on 12 GB of VRAM, pick a quantised 12–14B model with llama.cpp or Ollama. Speed figures come from the repo's release notes and an independent test, not measured by PixDig.
Install it
Requirements: Python and PyTorch; a Hugging Face account for gated models (Llama). Start with a small model (8B) to test.
pip install airllmpip install -U bitsandbytes
The prompt for your agent
Paste it into your coding agent: it checks the requirements and installs it for you.
Install AirLLM (https://github.com/lyogavin/airllm) on my machine.
1. Check the requirements: Python and PyTorch; a Hugging Face account for gated models (Llama). Start with a small model (8B) to test. If anything is missing, tell me before installing anything.
2. Follow the official method:
- Install: pip install airllm
- 4-bit compression (faster): pip install -U bitsandbytes
3. Check that it works, then explain how to launch it and where to start.
Always review what your agent is about to run, especially install commands.