selfhost.directory

The index of self-hostable projects — ranked, categorized and served with live updates.
Own your data and stop giving it to big tech.

GitHubWebsite
TRAWL logo

Self-host TRAWL

v1.9.3

Self-hosted scraping engine with browser challenge handling

ReleaseOpen SourceAGPL-3.0
GitHubWebsite
Install
Updated8 hours ago
LanguageTypescript
Setup difficultyModerate
Stars
1 078
Ports
8191 (API and FlareSolverr-compatible /v1 endpoint), 8192 (MITM forward proxy (when MITM_ENABLED=true))
Docker
ghcr.io/germondai/trawl
Install with Docker
TRAWL metrics dashboard showing request history, tier outcomes and failure details

What is TRAWL?

TRAWL is a self-hosted web scraping engine designed for applications and AI agents. It starts with HTTP requests and only opens a browser (Camoufox Firefox) when needed to solve challenges like Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest, and others. It features an adaptive tier engine that reuses accepted clearance sessions for fast repeat requests, supports FlareSolverr-compatible API for *arr stack integration (Prowlarr, Jackett, Sonarr, Radarr, etc.), provides MCP (Model Context Protocol) tools for AI agents, isolated browser sessions, a local metrics dashboard, and a challenge-aware HTTP/HTTPS forward proxy. It runs via Docker with optional Redis for session caching.

Should you self-host TRAWL?

TRAWL scores 56/100 on selfhost.directory (how scoring works) and ranks #11 of 16 in Web Scraping. It has 1 078 GitHub stars.

It shipped 10 stable releases in the past year. The latest, v1.9.3, came out on 24 October 2026. New versions land roughly every 4 days.

Setup is moderate: expect some configuration beyond a single command. An official Docker image is available. It is AGPL-3.0-licensed and written mostly in Typescript.

Its closest alternative on selfhost.directory is firecrawl (91/100, 196 246 stars). It scores higher overall, so compare the two before you pick. See TRAWL vs firecrawl side by side.

Features

  • Camoufox Firefox browser
  • Challenge-aware HTTP/HTTPS forward proxy
  • Challenge handling
  • Fast repeat requests
  • FlareSolverr-compatible API
  • Isolated browser sessions
  • Local metrics dashboard
  • Local solving by defaultOptional 2Captcha fallback for additional challenge types
  • MCP for AI agents
  • Resource-aware lifecycle

TRAWL README

How to install TRAWL

Install TRAWL with Docker

Environment variables
SESSION_CACHE_DRIVER, BROWSER_POOL_SIZE, MCP_ENABLED, METRICS_DASHBOARD_ENABLED

docker run -d --name trawl
-p 127.0.0.1:8191:8191
--shm-size=1gb
-e SESSION_CACHE_DRIVER=memory
-e BROWSER_POOL_SIZE=1
-e MCP_ENABLED=true
-e METRICS_DASHBOARD_ENABLED=true
-v trawl_metrics:/data/metrics
ghcr.io/germondai/trawl:1.9.2

Requires --shm-size=1gb for browser operation

How to configure TRAWL

  • BROWSER_ACQUIRE_TIMEOUT_MS15000Time to wait for a free browser before rejecting a request

    Time to wait for a free browser before rejecting a request

  • BROWSER_BLOCK_ADStrueSet false to omit uBlock Origin and reduce extension overhead; ads/trackers will load

    Set false to omit uBlock Origin and reduce extension overhead; ads/trackers will load

    values: true,false

  • BROWSER_CLOSE_TIMEOUT_MS10000Maximum time to wait while closing a wedged browser or context

    Maximum time to wait while closing a wedged browser or context

  • BROWSER_HARDWARE_CONCURRENCYOptional: lower the browser-reported CPU count to reduce PoW workers

    Optional: lower the browser-reported CPU count to reduce PoW workers

  • BROWSER_HEADFUL_POOL_SIZE0Browsers in the headful sub-pool, used only for DataDome escalations. Warmed on first use and additional to BROWSER_POOL_SIZE (~380 MB each). Set 1 to scrape DataDome targets.

    Browsers in the headful sub-pool, used only for DataDome escalations. Warmed on first use and additional to BROWSER_POOL_SIZE (~380 MB each). Set 1 to scrape DataDome targets.

  • BROWSER_IDLE_TIMEOUT_MS0Optional idle retirement; 0 keeps browsers warm. Wake adds a cold start.

    Optional idle retirement; 0 keeps browsers warm. Wake adds a cold start.

  • BROWSER_LAUNCH_TIMEOUT_MS90000Maximum time to wait for a browser launch

    Maximum time to wait for a browser launch

  • BROWSER_MAX_CONTENT_PROCESSES2Maximum Firefox content processes per browser

    Maximum Firefox content processes per browser

  • BROWSER_POOL_SIZE1Number of warm browser instances

    Number of warm browser instances

  • BROWSER_RECYCLE_AFTER_CONTEXTS8Recycle a browser after this many temporary challenge contexts. Set 0 to disable count-based recycling; memory-pressure recovery remains.

    Recycle a browser after this many temporary challenge contexts. Set 0 to disable count-based recycling; memory-pressure recovery remains.

  • BROWSER_SESSION_MAX_ENTRIES4Named browser contexts: idle TTL and per-instance capacity (use 1 on small containers)

    Named browser contexts: idle TTL and per-instance capacity (use 1 on small containers)

  • BROWSER_SESSION_TTL_SECONDS3600Named browser contexts: idle TTL and per-instance capacity (use 1 on small containers)

    Named browser contexts: idle TTL and per-instance capacity (use 1 on small containers)

  • BROWSER_SHUTDOWN_DRAIN_MS0Maximum time shutdown waits for accepted API requests. 0 closes them at once.

    Maximum time shutdown waits for accepted API requests. 0 closes them at once.

  • BROWSER_STALL_TIMEOUT_MS180000Grace period after a request timeout before a checked-out browser is reclaimed

    Grace period after a request timeout before a checked-out browser is reclaimed

  • LOG_LEVELinfoLog level

    Log level

  • MCP_ALLOWED_ORIGINSComma-separated browser origins allowed to call /mcp. Requests without Origin (normal server-to-server clients) are accepted.

    Comma-separated browser origins allowed to call /mcp. Requests without Origin (normal server-to-server clients) are accepted.

  • MCP_ENABLEDfalseModel Context Protocol (MCP) scraping tool. Keep disabled on public networks.

    Model Context Protocol (MCP) scraping tool. Keep disabled on public networks.

    values: true,false

  • MEMORY_SESSION_CACHE_MAX_ENTRIES1000Maximum number of sessions retained by the memory driver (least recently used entries are evicted)

    Maximum number of sessions retained by the memory driver (least recently used entries are evicted)

  • METRICS_DASHBOARD_ENABLEDfalseOptional dashboard. Set ENABLED=true only when its port is bound to 127.0.0.1.

    Optional dashboard. Set ENABLED=true only when its port is bound to 127.0.0.1.

    values: true,false

  • METRICS_DASHBOARD_TOKENA token of at least 32 characters protects data if the port is reachable elsewhere.

    A token of at least 32 characters protects data if the port is reachable elsewhere.

  • METRICS_DB_PATH/data/metrics/trawl.sqliteSQLite history path. Compose mounts a persistent named volume at /data/metrics.

    SQLite history path. Compose mounts a persistent named volume at /data/metrics.

  • MITM_CA_DIR/data/proxy-caPersistent directory for the generated CA certificate and key

    Persistent directory for the generated CA certificate and key

  • MITM_ENABLEDfalseEnables the general proxy listener

    Enables the general proxy listener

    values: true,false

  • MITM_HOST0.0.0.0Proxy address and port. Use 127.0.0.1 for local-only access.

    Proxy address and port. Use 127.0.0.1 for local-only access.

  • MITM_PORT8192Proxy address and port

    Proxy address and port

  • PORT8191HTTP API port

    HTTP API port

  • PROXY_URLTier 3 datacenter proxy: HTTP, authenticated HTTP, or SOCKS5

    Tier 3 datacenter proxy: HTTP, authenticated HTTP, or SOCKS5

  • REDIS_SESSION_TTL_SECONDS3600Lifetime of Redis or in-memory sessions in seconds

    Lifetime of Redis or in-memory sessions in seconds

  • REDIS_URLredis://localhost:6379Redis stores solved cookies and browser sessions. Leave empty to disable the Redis cache.

    Redis stores solved cookies and browser sessions. Leave empty to disable the Redis cache.

  • RESIDENTIAL_PROXY_URLTier 4 residential proxy. Uses the same URL and pool formats.

    Tier 4 residential proxy. Uses the same URL and pool formats.

  • SCRAPE_MAX_TIMEOUT_MS60000Default scrape deadline in milliseconds; per-request maxTimeout takes precedence

    Default scrape deadline in milliseconds; per-request maxTimeout takes precedence

  • SCRAPE_MIN_TIER1Lowest scraper tier allowed for every endpoint (1=HTTP, 2=cached browser, 3=fresh browser, 4=residential proxy). Values above 1 increase browser load.

    Lowest scraper tier allowed for every endpoint (1=HTTP, 2=cached browser, 3=fresh browser, 4=residential proxy). Values above 1 increase browser load.

    values: 1,2,3,4

  • SCREENSHOT_JPEG_QUALITY60Optional viewport screenshots (browser tiers only)

    Optional viewport screenshots (browser tiers only)

  • SCREENSHOT_MAX_BYTES4000000Optional viewport screenshots (browser tiers only)

    Optional viewport screenshots (browser tiers only)

  • SCREENSHOT_SETTLE_MS3000Optional viewport screenshots (browser tiers only)

    Optional viewport screenshots (browser tiers only)

  • SCREENSHOT_TIMEOUT_MS10000Optional viewport screenshots (browser tiers only)

    Optional viewport screenshots (browser tiers only)

  • SESSION_CACHE_DRIVERredisChoose Redis or single-process memory caching

    Choose Redis or single-process memory caching

    values: redis,memory

  • TIER1_TRANSPORTbunOptional Firefox TLS/HTTP2 impersonation in Tier 1: bun (default) or wreq

    Optional Firefox TLS/HTTP2 impersonation in Tier 1: bun (default) or wreq

    values: bun,wreq

  • USER_PREFSOptional Firefox prefs applied to every launched browser, as a JSON object

    Optional Firefox prefs applied to every launched browser, as a JSON object

TRAWL releases & version history

Even more releases RSS

More in Web Scraping

Rate this project

Click to rate — no account needed.

0 comments

Posting as guest
Loading comments…