Anti-Bot Bypass

Most "this site keeps blocking me" problems come down to anti-bot systems. OmniScrape handles them for you — you don't configure the bypass, you just request the page. This page explains what's happening so you know which knobs to turn when something is stubborn.

What OmniScrape handles automatically

When enable_solver is on (the default), OmniScrape detects and works through the major protection systems:

System	Typical signs
Cloudflare	"Checking your browser", Turnstile, cf_clearance cookie, 403/503 interstitials
DataDome	CAPTCHA walls, datadome cookie, 403 with a device-check page
Akamai	_abck cookie, sensor challenges
PerimeterX / HUMAN	_px cookies, press-and-hold challenges
AWS WAF	JS challenges, aws-waf-token
Imperva / Incapsula	incap_ses / visid_incap cookies

Behind the scenes this combines residential IPs, realistic TLS and browser fingerprints, a real headless browser when needed, and challenge solving.

The escalation ladder

If a page blocks you, climb this ladder:

1. Use auto (default). It already escalates from a fast request to a full browser when it detects a block.
2. Force js_rendering. Some sites need a real browser from the first byte. See Modes.
3. Add a matching proxy country. Geo-protected sites behave better with a local IP — proxy: "residential:us". For stubborn sites, use smart rotation mode (proxy: "residential:us:smart") so if one IP gets blocked, the engine swaps it automatically. See Proxies and Proxy Rotation Modes.
4. Wait for real content. Add js_wait_selector so you don't capture the challenge page instead of the result. See JS rendering.
5. Use a session. A stable IP and cookie jar across requests looks more human for multi-step or rate-sensitive sites. See Sessions.

{
  "url": "https://protected.example.com",
  "mode": "js_rendering",
  "proxy": "residential:us",
  "enable_solver": true,
  "js_wait_selector": ".main-content"
}

When you still get blocked

A 502 from OmniScrape means we couldn't unlock the target after trying. Things that help:

• Switch proxy country — some sites block certain regions.
• Confirm the URL is the real content page, not a redirect to a login or consent wall.
• Increase timeout for slow challenge pages.
• For interactive walls (press-and-hold, multi-step logins), use Browser-as-a-Service to drive the flow yourself.

Remember: failed unlocks aren't billed, so retries on 502 don't cost you. See Errors.

A note on responsible use

Bypassing anti-bot protection doesn't override a site's Terms of Service or the law. Scrape data you're permitted to access, avoid collecting personal data you don't need, and pace your requests. See Best practices.