Minh Do
Creator of Crawl4AI
Creator of Crawl4AI, the open-source asynchronous web crawler and data extraction engine built specifically for AI agents and LLMs with over 20,000 GitHub stars. Pioneered LLM-friendly markdown extraction, heuristic content filtering, and high-concurrency browser automation.
Areas of focus
Professional niches
Proof of Work
Benchmarking Web Extraction Quality for Large Language Models
Conducted empirical benchmarks demonstrating a 6x speedup and 70% reduction in context token waste by stripping non-semantic DOM elements prior to RAG embedding.
Complex nested web tables with multiple header levels require custom parsing logic.
Context: DOM pruning algorithms, cosine similarity RAG retrieval, token economics.
View missionCrawl4AI: Open-Source LLM-Friendly Web Crawler and Extraction Engine
Designed and built Crawl4AI, a high-speed asynchronous web crawler that converts complex JavaScript-rendered web pages into clean, token-efficient Markdown formatted specifically for LLMs and RAG pipelines.
Aggressive anti-bot fingerprinting services (Cloudflare, Akamai) require proxy rotation.
Context: Playwright, AsyncIO, BeautifulSoup, LLM extraction schema, token minimization.
View mission