germondai/trawl

Self-hosted scraping engine — bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *a...

82 stars · 2 forks · 1 watchers · TypeScript Topics: docker, scraper, typescript, scraping, selfhosted, self-hosted, web-scraping, arr, radarr, jackett, bun, captcha-bypass, bypass-cloudflare, flaresolverr, prowlarr, camoufox, captcha-slover, byparr

README

TRAWL

Welcome to TRAWL! 👋 Self-hosted web scraping engine with adaptive tier execution. Solves Cloudflare challenges natively using Camoufox Firefox. Returns cached results in under 500ms. Drop-in FlareSolverr v2 replacement. Features

4-tier execution — plain HTTP → cached session → fresh CF solve → residential proxy Native captcha solving — CF Turnstile, reCAPTCHA v2 (free STT), hCaptcha, GeeTest v4 Slide Camoufox Firefox — fingerprint-patched at the C++/Juggler level; indistinguishable from a real browser Session cache — cf_clearance cookies stored in Redis; repeat requests to the same domain return in ~500ms FlareSolverr v2 compatible — works with Prowlarr, Jackett, Sonarr, Radarr, and the full *arr ecosystem out of the box No external APIs — reCAPTCHA audio transcription uses Google's free STT endpoint by default; everything else is local

Quick start

Clone and configure

Start scraper + Redis

Verify

First boot takes 15–30s while the browser pool warms up. Subsequent starts are fast. API FlareSolverr-compatible (/v1) curl -X POST http://localhost:8191/v1 \ -H 'Content-Type: application/json' \ -d '{"cmd":"request.get","url":"https://nowsecure.nl","maxTimeout":60000}'

Native API (/scrape) Returns richer metadata: tier, timings, sessionCached, full cookie list. curl -X POST http://localhost:8191/scrape \ -H 'Content-Type: application/json' \ -d '{"url":"https://nowsecure.nl","maxTimeout":60000}'

Connect Prowlarr / Jackett Set the FlareSolverr URL to: http://localhost:8191 # running on the same host http://trawl:8191 # running via Docker Compose on the same network

Tiers Request │ ▼ Tier 1: Plain HTTP fetch ────── success ──→ return (< 100ms) │ blocked ▼ Tier 2: Cached session ─────── success ──→ return (~500ms) │ cache miss / expired ▼ Tier 3: Fresh CF solve ─────── success ──→ cache + return (4–15s) │ IP flagged ▼ Tier 4: Residential proxy ──── success ──→ cache + return (15–45s) │ failed ▼ error

Docker Compose files

File Description

docker-compose.yml Scraper + Redis (default)

docker-compose.minimal.yml Scraper only, no Redis

docker-compose.prod.yml Production: restart: always, memory limit, healthcheck

docker-compose.full.yml Full stack: scraper + web + docs

Configuration

Variable Default Description

BROWSER_POOL_SIZE 3 Warm Camoufox Firefox instances

BROWSER_ACQUIRE_TIMEOUT_MS 15000 How long acquire() polls for a free browser before HTTP 429 is returned

SESSION_TTL_SECONDS 3600 Redis session cache TTL (seconds)

REDIS_URL redis://localhost:6379 Redis connection string

RESIDENTIAL_PROXY_URL — Enables Tier 4 proxy escalation

STT_URL — Local Whisper endpoint for reCAPTCHA (optional)

PORT_API 8191 API listen port

Stack

Layer Technology

Runtime Bun

API Elysia

Browser Camoufox Firefox (via camoufox-js)

Session cache Redis

Landing page Nuxt 4

Documentation VitePress

License AGPL-3.0

Made with ❤️ by @germondai

source