selfhost.directory

The index of self-hostable projects — ranked, categorized and served with live updates.
Own your data and stop giving it to big tech.

Best self-hosted Web Scraping

Self-hosted web scraping software runs the crawling and extraction on your own machines instead of a third-party service. More about Web Scraping

--sort
33 projects· by score

Projects

30 of 33

About self-hosted Web Scraping software

You control where the requests go, what data is stored, and how the output is used. Crawl4AI is an open-source web crawler and scraper for LLMs and AI agents that converts any website into clean, LLM-ready Markdown, and it can be run yourself. changedetection.io focuses on monitoring: it tracks website changes, price drops, restock alerts and content updates. Maxun turns any website into a structured API so you can extract, automate, search and monitor the web. Browserless lets you deploy headless browsers in Docker on your own infrastructure.

33 self-hosted Web Scraping projects are listed here, ranked by popularity, maintenance and documentation.

Highest rated
firecrawl (199 668 GitHub stars, AGPL-3.0 license, written in Typescript), followed by Crawl4AI and Lightpanda
Docker
11 of the top 12 ship an official Docker image
Easiest to set up
Lightpanda, ArchiveBox and Obscura
Recently updated
browserless v2.60.0 on 2026-11-06

Top self-hosted Web Scraping software compared

ProjectLicenseLanguageStarsDockerSetupLatest releaseScore
firecrawlAGPL-3.0Typescript199 668YesAdvancedv2.11.091
Crawl4AIApache-2.0Python87 085YesModeratev0.9.587
LightpandaAGPL-3.0Zig41 589YesEasy1.0.384
HuginnMITRuby50 410YesModeratev2026.11.05.282
gpt-researcherApache-2.0Python30 560YesModeratev3.7.080
ArchiveBoxMITPython28 974YesEasyv0.9.7779
ObscuraApache-2.0Rust30 543YesEasyv0.2.676
camofox-browserMITJavaScript12 201—Easyv1.18.174
RSS-BridgeUnlicensePHP9 338YesModerate2025-08-0573
maxunAGPL-3.0Typescript18 168YesModeratev0.1.073
flaresolverrMITPython16 162YesEasyv3.5.273
browserlessOtherTypescript13 915YesEasyv2.60.073