# crawler-buddy

> Self-hosted HTTP crawling server with unified JSON API for web scraping, RSS discovery, and metadata extraction from YouTube, Reddit, GitHub, HackerNews.

- **Category:** Web Scraping
- **Licence:** GPL-3.0
- **Latest version:** 6.2.11
- **Open source:** oss
- **Languages:** Python, HTML
- **Platforms:** Docker Compose
- **Tags:** RSS, Feed Reader, Playwright, Json Api, Docker, Python, Flask, Selenium, Rss Reader, Scraping, Web Crawler, Rss Feed, Metadata extraction, Rss client, Webscraping, Webcrawling, Scrapy, Feed Discovery

## Description

Crawler Buddy is a self-hosted HTTP crawling server that delivers web content and metadata in a consistent JSON format. It provides a unified interface for extracting titles, descriptions, thumbnails, publication dates, RSS feed URLs, social media data, and link metadata from web pages. The server supports multiple crawling methods including Requests, Selenium (headless and full), curl-cffi, httpx, stealth-requests, Crawlee (BeautifulSoup and Playwright), and Scrapy, with automatic method selection based on URL patterns. It can discover RSS feeds for websites and YouTube channels, extract social media metrics from platforms like YouTube, HackerNews, GitHub, and Reddit, and parse RSS/Atom, OPML, and HTML files. The server runs as a Flask application on port 3000 and is deployed via Docker Compose, which includes Chrome and Chromedriver for browser-based crawling. All responses are UTF-8 encoded. The project is licensed under GPL-3.0 and written in Python.

## Links

- [Homepage](https://github.com/rumca-js/crawler-buddy)
- [Docker](rozbujnik/crawler-buddy)
- [GitHub Repository](https://github.com/rumca-js/crawler-buddy)
- [Docker Hub](https://hub.docker.com/r/rozbujnik/crawler-buddy)

---
Source: https://selfhost.directory/project/crawler-buddy