ai-scraping

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

Python scraper based on AI

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

Lightweight library for scraping web-sites with LLMs

AI Scraper is a powerful scraping tool and scrape agent built to automate data extraction with unmatched precision. Ideal for scalable AI scraping tasks across diverse web sources, this tool simplifies complex scraping operations into efficient, intelligent workflows.

🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.

➖ Stripped down, stable version of firecrawl optimized for self-hosting and ease of contribution. Billing logic and AI features are completely removed. Crawl and convert any website into LLM-ready markdown.

High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using remote browsers. Firecrawl alternative

[Mirror] Self-hosted abuse detection and rule enforcement against low-effort mass AI scraping and bots.

⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...

Continuously updated catalogue of open-source OSINT tools, skills, plugins, MCP servers, and agentic integrations.

AI web scraper built with Crawl4AI for extracting structured leads data from websites.

Extract Google Maps business leads and enrich contact details using AI & web scraping

How to guides on web-crawling or scraping

Python, Javascript, and Rust libraries for the Spider Cloud API.

Automate web workflows on real browser tabs without hosting a browser farm.