Self-host MuckScraper

MuckScraper README

How to install MuckScraper

OLLAMA_HOST, OLLAMA_MODEL, EMBEDDING_MODEL, ADMIN_PASSWORD, POSTGRES_PASSWORD, NEWS_API_KEY, GNEWS_API_KEY, SECRET_KEY, MEILI_MASTER_KEY, DISPLAY_TIMEZONE, PUBLIC_READ_ACCESS, LLM_PROVIDER, EMBEDDING_PROVIDER, GEMINI_API_KEY, GROQ_API_KEY, OPENROUTER_HOST, OLLAMA_FAST_MODEL, OLLAMA_THINK, MCP_TOKEN, N8N_WEBHOOK_URL, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST

How to configure MuckScraper

• ADMIN_EMAIL
• ADMIN_PASSWORD
• ADMIN_USERNAME=admin
• DISPLAY_TIMEZONE=UTCTime zone the aggregator shows times in. Use an IANA name for where you are, e.g. America/New_York, Europe/London, Australia/Sydney (full list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). Defaults to UTC.

  Time zone the aggregator shows times in. Use an IANA name for where you are, e.g. America/New_York, Europe/London, Australia/Sydney (full list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). Defaults to UTC.

• EMBEDDING_MODEL
• EMBEDDING_PROVIDER=ollama
• GEMINI_API_KEYRequired only when LLM_PROVIDER or EMBEDDING_PROVIDER is "gemini":

  Required only when LLM_PROVIDER or EMBEDDING_PROVIDER is "gemini":

• GEMINI_EMBEDDING_MODEL=gemini-embedding-001
• GEMINI_MODEL=gemini-2.5-flash-lite
• GNEWS_API_KEY
• GROQ_API_KEYRequired only when LLM_PROVIDER is "groq". GROQ_MODEL has no default -- Groq retires models periodically, so a baked-in name goes stale and 404s on every call. Check `curl -H "Authorization: Bearer $GROQ_API_KEY" \ https://api.groq.com/openai/v1/models` for what is currently served; openai/gpt-oss-20b was a reasonable general-purpose choice as of 2026-08.

  Required only when LLM_PROVIDER is "groq". GROQ_MODEL has no default -- Groq retires models periodically, so a baked-in name goes stale and 404s on every call. Check `curl -H "Authorization: Bearer $GROQ_API_KEY" \ https://api.groq.com/openai/v1/models` for what is currently served; openai/gpt-oss-20b was a reasonable general-purpose choice as of 2026-08.

• GROQ_MODEL
• GUNICORN_WORKERS=2Number of gunicorn worker processes for the admin app (default 2). Each is a separate process, so a long-running admin action in one worker doesn't freeze requests being served by the others.

  Number of gunicorn worker processes for the admin app (default 2). Each is a separate process, so a long-running admin action in one worker doesn't freeze requests being served by the others.

• LLM_FAST_PROVIDEROptional: send the fast tier to a different provider than LLM_PROVIDER. A run makes ~1,140 fast calls (story grouping, topic classification, outlet bias) against ~65 quality ones (summaries, deep reports), so this is how you run heavy summarization in the cloud while keeping the high-volume mechanical work on a local box: LLM_PROVIDER=openrouter <- summaries and deep reports LLM_FAST_PROVIDER

  Optional: send the fast tier to a different provider than LLM_PROVIDER. A run makes ~1,140 fast calls (story grouping, topic classification, outlet bias) against ~65 quality ones (summaries, deep reports), so this is how you run heavy summarization in the cloud while keeping the high-volume mechanical work on a local box: LLM_PROVIDER=openrouter <- summaries and deep reports LLM_FAST_PROVIDER

• LLM_PROVIDERProvider toggle: "ollama" (default, home machine), "gemini", "groq", or "openrouter" (e.g. running on a VPS while Ollama is unreachable). Flip both back to "ollama" once Ollama is reachable again -- no code changes needed. EMBEDDING_PROVIDER can only be "ollama" or "gemini". Groq serves no embedding models and OpenRouter routes chat completions only, so setting either here quietly falls back to Ol

  Provider toggle: "ollama" (default, home machine), "gemini", "groq", or "openrouter" (e.g. running on a VPS while Ollama is unreachable). Flip both back to "ollama" once Ollama is reachable again -- no code changes needed. EMBEDDING_PROVIDER can only be "ollama" or "gemini". Groq serves no embedding models and OpenRouter routes chat completions only, so setting either here quietly falls back to Ol

• MCP_ALLOWED_HOSTS
• MCP_BIND=127.0.0.1
• MCP_EXPECTED_OLLAMA_CONTEXTOptional: the context size your Ollama server is configured with (OLLAMA_CONTEXT_LENGTH there). ollama_health then reports when another client has loaded a model at a different size, which slows or stalls pipeline runs.

  Optional: the context size your Ollama server is configured with (OLLAMA_CONTEXT_LENGTH there). ollama_health then reports when another client has loaded a model at a different size, which slows or stalls pipeline runs.

• MCP_PORT=8765
• MCP_TOKENOptional read-only MCP server for health checks (docs/MCP.md). Started only with `docker compose --profile mcp up -d mcp`. MCP_TOKEN is required (16+ characters), e.g. `python3 -c "import secrets; print(secrets.token_urlsafe(32))"`. MCP_BIND is the host address the port is published on: 127.0.0.1 keeps it local; 0.0.0.0 opens it to the network (then rely on the token and a firewall or reverse prox

  Optional read-only MCP server for health checks (docs/MCP.md). Started only with `docker compose --profile mcp up -d mcp`. MCP_TOKEN is required (16+ characters), e.g. `python3 -c "import secrets; print(secrets.token_urlsafe(32))"`. MCP_BIND is the host address the port is published on: 127.0.0.1 keeps it local; 0.0.0.0 opens it to the network (then rely on the token and a firewall or reverse prox

• MEILI_MASTER_KEYProtects the search index. install.sh fills this in when it creates .env, so you only need it if you set up .env by hand (any random string of 16+ characters works): python3 -c "import secrets; print(secrets.token_hex(32))"

  Protects the search index. install.sh fills this in when it creates .env, so you only need it if you set up .env by hand (any random string of 16+ characters works): python3 -c "import secrets; print(secrets.token_hex(32))"

• MEILI_URL=http://meilisearch:7700
• NEWS_API_KEYmuckscraperHeadlinesGoogleNEW/.env.sample Optional: NewsAPI (newsapi.org) and GNews (gnews.io) keys. Leave blank to skip that source; the default RSS feeds work without either.

  muckscraperHeadlinesGoogleNEW/.env.sample Optional: NewsAPI (newsapi.org) and GNews (gnews.io) keys. Leave blank to skip that source; the default RSS feeds work without either.

• OLLAMA_FALLBACK_HOSTOptional: a second Ollama host used automatically when OLLAMA_HOST is unreachable (e.g. an always-on CPU box covering for a Wake-on-LAN GPU box). Must serve the same OLLAMA_MODEL / EMBEDDING_MODEL. Leave blank to disable.

  Optional: a second Ollama host used automatically when OLLAMA_HOST is unreachable (e.g. an always-on CPU box covering for a Wake-on-LAN GPU box). Must serve the same OLLAMA_MODEL / EMBEDDING_MODEL. Leave blank to disable.

• OLLAMA_FAST_MODELOptional: a smaller/faster model for the pipeline's high-volume mechanical calls -- story-grouping confirmation, topic classification, headline generation, outlet bias. A full run makes ~1,100 of those against ~65 summary/deep-report calls, which stay on OLLAMA_MODEL regardless. Biggest win when OLLAMA_MODEL is too large to fit your GPU's VRAM (check `size_vram` vs `size` in /api/ps -- a partially

  Optional: a smaller/faster model for the pipeline's high-volume mechanical calls -- story-grouping confirmation, topic classification, headline generation, outlet bias. A full run makes ~1,100 of those against ~65 summary/deep-report calls, which stay on OLLAMA_MODEL regardless. Biggest win when OLLAMA_MODEL is too large to fit your GPU's VRAM (check `size_vram` vs `size` in /api/ps -- a partially

• OLLAMA_HOST
• OLLAMA_MACOptional: set this if Ollama is on another machine and you use wake-on-LAN.

  Optional: set this if Ollama is on another machine and you use wake-on-LAN.

• OLLAMA_MODELThe main chat model, pulled on your Ollama box first (`ollama pull <name>`). Starting points: gemma4:e4b-it-qat (about 3 GB of VRAM) or, on a 10 GB+ GPU, gemma4:12b-it-qat with OLLAMA_THINK=false below. See README "Hardware".

  The main chat model, pulled on your Ollama box first (`ollama pull <name>`). Starting points: gemma4:e4b-it-qat (about 3 GB of VRAM) or, on a 10 GB+ GPU, gemma4:12b-it-qat with OLLAMA_THINK=false below. See README "Hardware".

• OLLAMA_PRIMARY_RECHECK_SECONDS=60Optional: seconds to serve from the fallback before re-probing the primary so it is picked back up automatically once it comes online (default 60).

  Optional: seconds to serve from the fallback before re-probing the primary so it is picked back up automatically once it comes online (default 60).

• OLLAMA_THINKOptional: turn a thinking model's hidden reasoning off (false) or on (true). Models such as gemma4, qwen3 and deepseek-r1 think before every answer by default, which can make each call 5-50x longer. Blank leaves Ollama's default. Only set it for a model that supports thinking; Ollama rejects the flag otherwise. Check quality on your own prompts before turning it off: on gemma4 e4b it made story gr

  Optional: turn a thinking model's hidden reasoning off (false) or on (true). Models such as gemma4, qwen3 and deepseek-r1 think before every answer by default, which can make each call 5-50x longer. Blank leaves Ollama's default. Only set it for a model that supports thinking; Ollama rejects the flag otherwise. Check quality on your own prompts before turning it off: on gemma4 e4b it made story gr

• OPENROUTER_API_KEYRequired only when LLM_PROVIDER is "openrouter". OPENROUTER_MODEL has no default -- leaving it blank reads as "not configured" rather than sending an empty model name, so set it to something like "meta-llama/llama-3.1-8b-instruct".

  Required only when LLM_PROVIDER is "openrouter". OPENROUTER_MODEL has no default -- leaving it blank reads as "not configured" rather than sending an empty model name, so set it to something like "meta-llama/llama-3.1-8b-instruct".

• OPENROUTER_APP_NAME
• OPENROUTER_HOSTOPENROUTER_HOST is the OpenAI-compatible base URL. Point it at any other provider speaking that API -- DeepSeek, OpenAI, Together, a local vLLM -- and the OPENROUTER_* settings drive it with no code changes.

  OPENROUTER_HOST is the OpenAI-compatible base URL. Point it at any other provider speaking that API -- DeepSeek, OpenAI, Together, a local vLLM -- and the OPENROUTER_* settings drive it with no code changes.

• OPENROUTER_MODEL
• OPENROUTER_SITE_URLOptional, attribution only -- OpenRouter shows these on its public leaderboards.

  Optional, attribution only -- OpenRouter shows these on its public leaderboards.

• POSTGRES_DB=aggregator
• POSTGRES_PASSWORD
• POSTGRES_USER=postgres
• PUBLIC_READ_ACCESS=falseLet signed-out visitors read Headlines, story pages and article summaries (never the scraped full text, and no buttons that run anything). Off by default, so every page needs an account. Accounts are managed at /admin/users.

  Let signed-out visitors read Headlines, story pages and article summaries (never the scraped full text, and no buttons that run anything). Off by default, so every page needs an account. Accounts are managed at /admin/users.

• SECRET_KEYSigns login sessions; keep it secret. install.sh fills this in when it creates .env, so you only need it if you set up .env by hand: python3 -c "import secrets; print(secrets.token_hex(32))"

  Signs login sessions; keep it secret. install.sh fills this in when it creates .env, so you only need it if you set up .env by hand: python3 -c "import secrets; print(secrets.token_hex(32))"