@levelsio: ✨ Okay with lots of help from [@javilopen](https://x.com/javilopen) and his Spanish scraping friends I've managed to vib…

x.com · 2026-09-09 · by Long Nguyen ·xcom

57.2K

@levelsio

✨ Okay with lots of help from @javilopen and his Spanish scraping friends I've managed to vibe code my own @Scrapingbee alternative and replace my $249/mo Scrapingbee plan with my own $1/mo scraper running on my VPS!

First I tried OpenSERP, an open source project that seems freshly maintained but it didn't work on my VPS IP or laptop IP and got blocked fast

Then I tried to use residential IP proxies from IPRoyal but they were already quite tainted IPs meaning Google already knew they were scrapers and just blocked about 60% of them already, not so great, also didn't work!

Then I discovered OpenSERP uses a vanilla headless Chrome, Google instantly detects that and blocks that, so literally the entire repo is useless cause it won't even work? I don't get why they don't say that in the docs? OpenSERP is even worse than just basic CURL, as my AI said: "plain curl got 7% blocked but OpenSERP's headless browser got 100% blocked"

So then I just built my own with Playwright and a real Google Chrome on my VPS, headful and persistent context, and slow pacing like every 30 seconds or so

And I got some better residential IP recommendations (which I can't tell you because thenhey t stop working!)

Now I'm at about 90% success rate, which is similar or better than Scrapingbee

And it costs me ~$1/mo (measured by GBs, obviously for text scraping I barely use a lot) instead of the $249/mo I paid Scrapingbee!

My only remaining SaaS now are:
- Cloudflare Domains
- Cloudflare Email Sending
- Cloudflare R2 Storage
- Backblaze B2 for backups
- Hetzner for all my servers for all my sites (VPS)
- xAI for all LLMs for all my sites (so many things like moderation, writing emails, anything)

💬 63 · 🔁 14 · ❤️ 348 · 👁 57.2K

Sep 55:48 PM · Sep 9, 2026 · View on X

Replies (61)

Javi Lopez ⛩️ @javilopen

"Javi Lopez and his Spanish scraping friends"

Karl Heinz Frankeser @karlfrankeser

I use @firecrawl and it works pretty well imho. @levelsio do you use your own scraper for hotellist?

1K @levelsio author

Yep

Paul Bertrand @pbertrand_dev

super curious how long this will stay working because the "maintenance" people always tell me this would take lots of work to stay up and running long term

1K @levelsio author

Same, it Telegrams me once it stops working

Gregor @krekr@ioc.exchange @krekr.bsky.social @krekr

“Residential proxies” 😬

719 @levelsio author

That's what they are right or no? I have no clue

Niklas ⚡️ @Niklas_Sikorra

What are you using it for?

https:// @levelsio author

For https://nomads.com some live cost data, for https://hotelist.com some hotel ratings and for https://photoai.com popular photo shoots and then it reverse engineers prompts for community tab

Serg @serglotz

Dope, will you OSS it?

496 @levelsio author

If you copy this tweet your clanker can build it, all the info is there!

djcows @djcows

"vibe coded my own scraper" is insane😂

208 @levelsio author

This is a DJ COWS Production 

...production.....pro....ction.....ro...ion......nnn

Walid B. | Stop the Cap @WalidBou07

so what database you are using now?
and where you are hosting it?
all in hetzner?

mura @mura1566595

did you try dataforseo?

Massi @MaxMalloc

I created https://github.com/0xMassi/webclaw to scrape any website and avoid bot protection

angular2react @angular2react

Good luck. That's something they'll block eventually too. Did that already 6 years ago. Frustrating process

kdk @kdkdoteth

when are you building a server farm to replace Hetzner?

Mlili @mli__li

Is this the type of residential IP that uses my TV as a relay? 😂

KlaasEngels @Xinirnix

Will you post the prompts etc on https://canivibeit.com or somewhere?

Louis DCK @LouisDeconinck

Any reason why you pick xAI?

Venkat Dinavahi @venkatdinavahi

What's your main motivation for removing SaaS? I'm assuming it's not cost since it's not a lot for you.

Ismael ‏️‏️ @ismaVQ

proxies with IPv6 are decent+cheap for SERP scraping (at least when I needed them a year ago)

不俱戴天 // Jose @400_yen

I do something similar with https://github.com/D4Vinci/Scrapling… and a stealth chrome driver, as some anti-scrappers detect if Chrome is being automated.

Aditya Tripathi @AdityaTripathiD

Scraping is against Hetzner's ToS. @Hetzner_Online can you confirm if this is still the case?

Ilya Pivto @ipivto

thanks for sharing

i’ve recently researched email sending providers and cloudflare doesn’t support marketing emails. Do you somehow do it anyway?

Mischa van den Burg @mischavdburg

Sick , nice work !

Alex 🔔 | updatify.io @_avdept

you basically pay for residental IPs in case of scrapingbee + some middleware they provide if you need to inrerat with page

we use scrapingbee too, our bill magnitudes higher, but you gave me good idea to think about optimization

will DM you with a question

Jake Mcmahon @JakeMcm36986672

Right, I was surprised when I saw you had to resort to scraping bee. Because I've been able to do a lot with VPS and generic free tools. With a little nodemaven etc. where needed (those guys are great, and super cheap because you don't need to use it a lot).

Ritvij Sharma @ritvij14_

oooh nice, I am gonna try this out. this sounds fun

Adrian @morion4000

It looks like bot farms are what everyone is hinting at these days if you want to get ahead, especially with the dead internet theory. 

Clean IPs are like a commodity these days, sort of like anti-sybil receipts or proof of human.

Zam @i_am_za_man

This is the part people skip. Ship a real alternative on Playwright + a VPS instead of waiting for a platform to make it cheap.

Revo Laition @revolaition

Brilliant, this is another beast than "look! i just vibed my own $19 saas replacement!" $249/mo is serious money.

will you share the sauce?

Hugo Patti @hugopatti13

Honnêtement, remplacer un outil à 249$/mois par le tien à 1$, c’est le vrai flex indie. Tu contrôles le coût et le code.

AJawhar @alaajawhareng

playright will fail soon!
I tried the same approach, 2 days later it stopped.
I recommend you to use AgentReach project. it is been a month and it is still working for me

Bartek Rutkowski @RutkowskiHQ

Wait, hold on. You said it was impossible to replace with a self built solution! ;) There is no moat anymore, right?

Yoichi @Yoichi14a

Residential ips with headless chrome is financial suicide.

Matet | 👉 Takamoe.com @matetdev

Nice build. Whats been the hardest part to make reliable?

VastPlan @yandt888

$249/月 Scrapingbee 换成 VPS 上 $1 自建——账单一砍，依赖也断了。能 vibe 出替代品，订阅税就不该交。

Krish @krishnalpanara

Kuddos on saving yourself from IP Royal.

As someone who runs a unbiased proxy provider platform.

Trust me when I say, Proxy Providers are the most boatsful when it comes to their reliability of "Residential Proxies"

They have a whole team populating reddit and other forums

NodeZero @honeybadger1742

How do you handle captchas. Do you use any service for it. Or do simple retries

Michiel @3Michiel

Any plans to replace Hetzner with your  own hardware?

Etorobong Isong ™ @EtorobongI

Classic levelsio... turned a recurring SaaS cost into a one-time engineering problem.

 The headful Chrome + slow pacing detail is the part most people would skip and then wonder why their scraper dies.

Silver @sgraphics8

Did you solve captchas?

David Arnal @davidarngar

The interesting part is the reliability curve: at 90% success, retries + caching could make the effective hit rate much higher without adding much bandwidth. Measuring success by query type would reveal where the remaining 10% actually comes from.

Fırat Bayram Bakır @firatbb

0xAyush @ayush152002

Finally this comment is open for all .. 😅😅

Simon Novospes @novospes

most of the trouble is the ip rotation

Ivan Kuznetsov @ikuznetsov_com

I just use my residential devices as exit nodes for scrapers when needed, so together with proper browser headers they are indistinguishable from the real devices. Also flaresolverr helps me when I face cloudflare captcha. 

There are also some alternative approaches like using

Michael Shynar @shmichael_eth

You mean /y ?

Nifal Adam @nifal_adam

Wth man because you said scrapingbee is impossible to build I paid them 2 days ago to scrape stuff for my news site.

Jean @JohnStrongHodl

replacing domains will be the hardest one - harder than replacing xAI

Mike Gyi @mikegyi

Love it.

Wilmer Terrero @wilterrero

you will come back to Scrapingbee after the pages you scrape adds Cloudflare protection lol

Robin @xdNiBoR

Very cool project. You're really starting to run on 99.99999% margins lol

Jeremiah Agthe @Serial_Builder

Better than cloudflare’s scraper?

Daniel @dgrcode

wait am I reading correctly that I can send 100k emails per day, for FREE, with Cloudflare Email Sending??

and 33 million per month for $5???

that would be $29,675 with Resend. what am I missing?

KlaasEngels @Xinirnix

Impressive!

Ronald Chumaceiro @ronchumaceiro

sept 4: "ai told me it can't do it". sept 9: done, with help from a human who'd done scraping before. that's the part of vibe coding nobody puts in the meme. the ai does the work, the person who's been blocked by google before unblocks the last 20%. the $248 a month is the

Alex 💪 @awpthorp

Lmao fuck

stemonte @stemonteduro

The next logical step is to build a server farm

Comments (0)

No comments yet