nodemaven/proxy-benchmark
Measurement harness for proxy providers, browser engines and scraping targets. Every claim carries the run it came from. - nodemaven/proxy-benchmark
15 stars · 0 forks
README
proxy-benchmark Find out what is blocking your requests - the proxy, the browser, the host, or the target itself. Built and maintained by NodeMaven, who sell proxies. Every number here is generated from the run files in data/runs/, which are committed, so a reader can recompute any of them rather than take them.
What this is A failed request tells you that something failed. It does not tell you what. The target may be refusing the exit address. It may be reading the browser. The handshake may be the discriminator. The machine you are running on may be the variable. Or the proxy may be failing before the request reaches the target at all. proxy-benchmark separates those and measures them in one time window. It runs the same targets across controlled combinations of:
engines - plain HTTP clients, stock Chromium, patched browsers, CDP drivers and anti-detect frameworks; proxy paths - gateways, providers, countries, sticky sessions, and a direct arm with no proxy at all; browser configuration - resource blocking, headful, timezone and locale alignment, humanized input where the engine has it; hosts - when the machine itself turns out to be part of the experiment, which here it did.
Every attempt is one JSONL row under data/runs/ carrying the full parameter set, and the tables below are generated from those rows rather than typed in. Bring your own proxy. No NodeMaven account is needed. A gateway is a .toml file and an engine is a module plus one registry line, so neither ever edits the runner - point it at a proxy you already own. Why this exists The usual debugging loop is: request failed, so the proxy must be bad, so change the proxy, so it failed again, so the browser must be detected. None of those steps is justified by a single request. This asks the question with a control instead:
Question The comparison that answers it
Is the address the problem? the same engine through the gateway and direct, in one window
Is the browser the problem? several engines through the same pool, same hour
Does the exit country matter? one engine, --countries us,any
Does the host matter? the same configuration on two machines, overlapping hours
Does an anti-detect browser actually help? it and the unmodified control in the same matrix
Is the handshake the discriminator? ClientHello read against the target's answer
Does warming an exit help? the warm-up ladder, all rungs interleaved
The output that matters is not "this request passed". It is: this variable changed, these did not, and the outcome moved with it. Four of the answers that came back, each with its denominator and its run file:
Warming an exit took Google from 20% to 84%. Six pages of warm-up before the query, interleaved with a cold arm inside the same runs: 20.0% (33/165) cold against 84.4% (152/180) at the deepest rung, pooled over three separate days, z = 12.0. One page of warm-up moved nothing. The ladder The unmodified browser finished in the leading group on Amazon. Stock Chromium 96% (419/436) against 63% (288/457) for the lowest anti-detect engine, over 3530 judged attempts. Six engines sit within four points at the top, no test separates them, and the control is one of the six. Full table The same code, gateway and target scored 39% on one machine and 0% on another. 24/61 from a Windows workstation against 0/84 from a Linux VPS in overlapping hours, Fisher p = 3.7e-11. The client machine is a variable a proxy comparison usually holds fixed without saying so. The split Chrome spends 43 MB per fresh profile talking to Google before you ask it for anything. 43.2 MB of a 43.4 MB idle window, on a browser parked on about:blank. On a metered residential exit that is billed traffic for a file no target ever sees. How it was counted
The rest of them. Contents
Sections
Start here What this is · Why this exists · Quickstart · Documentation
What is measured What it measures · How the experiment is structured · What is under test · What the rows carry
What came out Results at a glance · Research findings
Running it Setup · Running it · Command reference · Operational safety
The experiments Front-page entry · The warm-up ladder · Bringing your own proxy · Adding a provider · The axes
Checking the work Reproduce these numbers · Repository layout · Contributing
All 22 sections are listed, and every anchor above is checked by test_every_link_inside_the_repository_resolves. Quickstart A real measurement, no proxy account, no browser download. Three packages, and about 30 s to run on a fresh clone. git clone https://github.com/nodemaven/proxy-benchmark && cd proxy-benchmark python -m venv .venv .venv\Scripts\Activate.ps1 # macOS, Linux: . .venv/bin/activate pip install -r requirements-core.txt python scripts/benchmark.py --engines http --targets ddg_serp \ --queries 5 --direct --preset none
It prints the plan and what it will cost before it sends anything, then one line per attempt and a summary: engine target exit n pass verdicts http-direct ddg_serp direct 5 100% {'ok': 5}
The first run is deliberately the bare HTTP client with no proxy. There is no browser to download, no proxy account to configure, and nothing to spend, and it establishes that the harness works before a second variable is introduced. Every attempt is also a JSONL row under data/runs/, which is the only thing this repository treats as evidence. From here: the same run through a proxy, which needs no account with anybody; real browsers instead of a bare client; or check a number here rather than take it. Documentation This file is the map. The depth is deliberately split by purpose, so nothing has to be read before the first run:
Document What is in it
README.md what the harness does, how the experiment is structured, the headline findings, and every command
RESULTS.md the full tables: denominators, Wilson intervals, Fisher tests, run ids, engine by engine and day by day. Generated from the rows
NOTEBOOK.md how each result was arrived at: what was varied, what was held fixed, and the run behind every number
docs/quickstart.md an empty machine to a first measurement, step by step: Python, git, venv, dry-run, browsers, proxies, reading verdicts
CONTRIBUTING.md adding an engine, a target or a provider, and the rules that are not obvious from the code
Want the numbers - RESULTS.md. Want to know how they were arrived at - NOTEBOOK.md. Never used a terminal - docs/quickstart.md assumes nothing; this file assumes you know what a ClientHello is. What it measures Three layers can reject a request, and debugging fails when you inspect the one above whichever is actually refusing you. That is the whole reason the axes are separate.
Layer What gives you away Read with
IP reputation datacenter range, a burnt residential exit, a country the target treats harshly --countries, probe-and-hold, gateway-health
Browser signals navigator.webdriver, empty plugin list, a SwiftShader renderer, a HeadlessChrome User-Agent engine-fingerprint, detect-page
TLS handshake ClientHello shape: cipher and extension counts, absence of GREASE tls_ja4 on the row, tls-clienthello, tls-echo
A different handshake does not by itself explain a different pass rate. Measure the two together - here they did not agree. The handshake row has three entries because there are three routes to a JA4 and they answer different questions. Pick by what you are asking:
Route Covers Costs What the value is
tls_ja4 column the 3 engines that need the relay nothing, it comes with the matrix the handshake on the connection that produced this row's verdict
tls-clienthello 10 of the 11 engines, not obscura nothing: a listener on this machine, no live host, no traffic what the engine's handshake is, off the run
tls-echo all 11, obscura included a live host, one request per engine the same, plus what only a server that answers can derive
Only the first is per-attempt; the other two are per-engine and are the ones to re-run after a browser upgrade. tls-echo is the only route to obscura, which refuses to navigate to localhost at all, and the only route to the HTTP/2 SETTINGS hash and header order, which a listener that answers nothing cannot provoke. Cross-checked on one host: the eight engines both probes reach agree character for character. How the experiment is structured A run is a matrix, and every axis is independent: target, engine, provider, country, gateway parameters, resource blocking, headful, geo alignment, entry shape, warm-up rung. They do not nest inside one another, and that is the point - any one can be varied while the rest are held, and all of them are on every row. Two properties do most of the work:
Cells interleave at batch granularity, never run in sequence. Finishing one engine before starting the next would measure the afternoon as well. The hour is the largest confound this repository has found: the same gateway, country and browser moved 69 points to 52 between two windows of one afternoon. A matrix that could not be controlled is refused, not quietly adjusted. If an option applies to some engines and not others, the run would compare a humanized Camoufox against an unhumanized everything else and read it as an engine difference. Every engine declares what it supports and --dry-run refuses the matrix before it starts.
What is under test
11 frameworks, one registry line each. Anything missing from the machine reports itself unavailable and names the install command, and the rest of the matrix still runs - --dry-run prints that list.
--engines what it is needs
http Plain HTTP client. No browser, no JavaScript. The cheap baseline nothing beyond requests
chromium Stock Playwright Chromium. The unmodified control every other engine is measured against playwright install chromium
camoufox Camoufox: a patched Firefox driven through Playwright camoufox fetch
patchright Patchright: Playwright with the automation tells patched out patchright install chromium
obscura Obscura: a from-scratch browser in Rust, driven over CDP a built Obscura binary
cloak CloakBrowser: a patched Chromium that hands back a Playwright browser cloakbrowser.ensure_binary()
curlcffi A scriptless client wearing Chrome's handshake. The control for the control curl_cffi, no browser
seleniumbase SeleniumBase UC mode: the WebDriver family, which the matrix did not have the host's installed Chrome
zendriver Zendriver: Chrome over raw CDP, with no WebDriver and no Playwright the host's installed Chrome
rebrowser Rebrowser: Playwright with the Runtime.enable leak patched out rebrowser_playwright install chromium
botasaurus Botasaurus: the host's Chrome over raw CDP, driven by a scraping framework the host's installed Chrome
Any of them takes a :direct suffix, which runs that engine around the gateway inside the same matrix, so the proxy and the no-proxy arm are measured in one window rather than an hour apart.
Four targets carry the published results, chosen because they fail differently rather than because they are popular: google_serp, amazon_search, bing_serp and ddg_serp. ipinfo sits beside them and is not a target but an echo service, used to prove the path works before anything is concluded from a refusal. The registry in nmbench/targets.py holds a few more that do not yet have enough rows to say anything. What the rows carry Every attempt writes one JSONL row: engine and engine version, target, provider and gateway parameters, country, preset, headful, geo, entry shape, warm-up rung, session and query, the machine it ran on, the verdict and the marker counts behind it, the failure reason, timing, and bytes. ROW_FIELDS in nmbench/engines/base.py is the schema. Those files are the source of truth, and the tables in this README and in RESULTS.md are generated from them - a number nobody has to remember to update is a number that cannot drift. Verdicts come from page content, not HTTP status. The same Google reCAPTCHA page arrived once as 429 and once as 200, so a run judged by status scores the second as a success. There is no boolean success column:
verdict what it means
ok the target returned the page that was asked for
captcha a challenge stood in front of the result
consent a consent or cookie wall stood in front of the result and could not be cleared
block the target refused the address outright
throttle Amazon's throttle page: the address is refused rather than the browser challenged
empty a body arrived and the result was not in it
error the attempt never completed - the harness, never the target
The last two are the ones that decide whether a benchmark measures anything:
empty is not block. Google hands a scriptless client a 92 KB "enable JavaScript" scaffold that stays on /search and rejects nothing, so scoring it as a block credits Google with a refusal it never made. 14 of 14 such rows carried enablejs and none carried recaptcha. error is ours. A timed-out selector, a browser that would not launch and a query that never reached the box produced no evidence, so they produce no verdict. A harness that counts its own crashes as target refusals can manufacture a very convincing result while measuring almost nothing.
throttle is kept apart from block because on Amazon it is most of what a refusal is: 476 of the 704 refused Amazon rows are the throttle page (ref=cs_503). It arrives with status 503 on 397 of them and 200 on 59, so it is read from the body like every other verdict. A consent page is cleared where the target declares how, by pressing its reject button, and consent_dismissed on the row says it happened. consent is what is left when it could not be. Nine more columns are easy to misread:
An absent host means "not recorded", not "the machine was unknown". Older rows carry no host and are attributed to a machine by their timestamp, so on those rows host and date are one variable under two names. It matters more than a provenance column usually would: the largest unexplained result in this repository is a difference between two computers, 39% (24/61) against 0% (0/84) at p = 3.7e-11 on the same target, engine, entry shape and gateway parameters. host_os and host_cpus sit beside it because a label groups rows and does not explain them. Set NMBENCH_HOST to the machine's name in the notes; unset, the column holds a hash of the hostname, because these files are public and a hostname names somebody's infrastructure.
tls_ja4 is empty on most rows for a structural reason, not a missing one. It is the engine's JA4, read off the ClientHello as it passes through the local CONNECT relay, so only the engines that need that relay have one - zendriver, seleniumbase and botasaurus. The Playwright-driven engines take proxy credentials directly and never send a handshake through this process, so their rows carry null and the relayed column beside it says why. The column postdates every committed benchmark run, so there it is empty; the marketplace recon rows carry it. The fingerprint is computed in nmbench/tlsfp.py rather than asked of an echo service, and it was checked against one: on the same client, tls.peet.ws and this repository agree character for character. Every engine can still be fingerprinted, off the run, with python scripts/probes/tls_clienthello.py. It points each engine in turn at a listener on this machine that answers nothing, so it needs no live host and spends no traffic, and it reaches the Playwright-driven engines the relay route cannot.
A JA4 is a property of the browser build at least as much as of the engine. Measured with the chromium engine and nothing varied but the binary: Playwright's bundled Chromium 151.0.7922.34 gives t13d1516h2_8daaf6152771_806a8c22fdea, and the installed Chrome 149.0.7827.201, reached with --channel chrome, gives t13d1516h2_8daaf6152771_d8a2da3f94cd. The extension lists are byte for byte the same and so is the cipher hash; the whole difference is three signature algorithms, 0904,0905,0906, that the newer build offers and the older one does not. That is why the same probe puts chromium and patchright in one group and zendriver, seleniumbase, botasaurus, cloak and curl_cffi in another - it is the Chrome version each happened to launch, not anything the libraries do differently. Read engine_version beside this column before treating a split as a property of the engine.
A refused address diverts the request, and the status does not say so. Google answers a refusal by sending the request to /sorry/, with a 200 about a quarter of the time. report.was_served - a 200 whose final URL keeps the host and path asked for - is the test, and it is a property of the exchange, so nothing has to know a target's name.
A batch is one session, and a session is the unit. Ten queries through one browser is one identity doing ten searches; ten browsers doing one query each is a different experiment. session-continuity is the offline probe that checks every engine keeps one cookie jar across a batch.
bytes is two measurements and relayed says which. Playwright engines count through page.route and see page resources; the relay counts sockets and sees request headers and TLS overhead as well. Never pool them. The relay figure is what a provider bills, and it adds a loopback hop, so elapsed_ms is not comparable across relayed.
The matrix carries an unmodified control. chromium is Playwright's Chromium with no arguments, no user agent override and no patches; navigator.webdriver is true and stays that way, because without it a pass rate cannot be told apart from the target letting everything through. tests/test_engines.py reads the source of ChromiumEngine.open and fails if args= or user_agent appear.
Whether the exit held is on the session_closed row, not on the attempts. The exit is read before the first query, before the middle one and after the last; identity_stable and identity_changed_at say whether it moved, and (cell, batch_index) joins the answer back to the attempts.
speed_* is sampled, and connect_ms is not echo_ms. The first 20 sessions of a run time a fixed 1 MB download (--speed-sessions) and their rows carry it; the rest carry null. tcp_ms is the distance to the gateway and connect_ms what the gateway took to reach an exit, and both exist only where the gateway reports the exit in its CONNECT reply. Where the exit is read through an echo service the row has echo_ms, a longer path kept under its own name.
Response bodies are kept, gzipped. A verdict is one word about 92 KB of markup, and the question that decides a report is usually asked after the run. Non-ok bodies plus a sample of the passes, controlled by --no-bodies and --sample-ok. The archive is gitignored, unlike data/runs/, because exit addresses appear in embedded links. Re-reading 250 stored Amazon bodies offline found an Akamai interstitial and an AWS WAF challenge filed as refusals, and moved 21 historical rows at no traffic cost. Results at a glance
What the current evidence supports, engine by engine and target by target, from the 10432 attempt rows in data/runs/benchmark_*.jsonl. pass is ok over judged attempts - harness and path failures are counted separately and excluded from the denominator, because an engine that crashes is not an engine the target refused.
target best worst rows
amazon_search chromium/none 96% (419/436) patchright/none 63% (288/457) 3816
google_serp every engine below 5%, best is 1.1% (5/460) - not an engine comparison - the whole column is one browser on one host that this target refuses - 3811
bing_serp chromium/light 100% (45/45) http-direct 76% (34/45) 372
ddg_serp camoufox/light 100% (44/44) chromium-direct/light 22% (10/45) 339
walmart_search no cell reaches 30 judged attempts, so no engine is named - 35
On the one target with enough evidence to rank engines, the top of the table