Directory › Data & analytics › Crawl4AI

> GEM DATA & ANALYTICS · RESEARCH & MONITORING · AGENTS & AUTOMATION · DEVELOPMENT

Crawl4AI

by UncleCode

Turns any website into clean Markdown, ready to feed your LLMs and RAG pipelines.

85.3 k stars GitHub Apache-2.0 FreeOpen sourceLocal

View on GitHub →

What it does

Async crawler driven by a real browser (dynamic pages included)
Markdown output or structured extraction for AI
Python library or Docker server with a REST API

Install it

Requirements: Python 3

Install

pip install -U crawl4ai

Install the browser (once)

crawl4ai-setup

Check the installation

crawl4ai-doctor

The prompt for your agent

Paste it into your coding agent: it checks the requirements and installs it for you.

Claude CodeCodexCursorGemini CLIopencode

Prompt

Install Crawl4AI (https://github.com/unclecode/crawl4ai) on my machine.
1. Check the requirements: Python 3. If anything is missing, tell me before installing anything.
2. Follow the official method:
- Install: pip install -U crawl4ai
- Install the browser (once): crawl4ai-setup
- Check the installation: crawl4ai-doctor
3. Check that it works, then explain how to launch it and where to start.

Always review what your agent is about to run, especially install commands.