Web Unlocker

Send a URL, get the page content back. Anti-bot bypass, proxies, and JS rendering all happen on our side.


Endpoint	`POST https://api.omniscrape.io/v1/scrape`
Auth	`X-API-Key: YOUR_KEY` header
Billing	$0.0035 per successful request — failed unlocks are free

1. Make a request

Only url is required:

curl -X POST https://api.omniscrape.io/v1/scrape \
  -H "X-API-Key: $OMNISCRAPE_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com" }'

import os, requests

resp = requests.post(
    "https://api.omniscrape.io/v1/scrape",
    headers={"X-API-Key": os.environ["OMNISCRAPE_KEY"]},
    json={"url": "https://example.com"},
    timeout=120,
)
print(resp.json()["data"]["content"])

const resp = await fetch("https://api.omniscrape.io/v1/scrape", {
  method: "POST",
  headers: {
    "X-API-Key": process.env.OMNISCRAPE_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com" }),
});
const { data } = await resp.json();
console.log(data.content);

2. Understand the response

Every response has the same shape:

{
  "success": true,
  "data": {
    "content": "<html>...</html>",
    "status_code": 200,
    "final_url": "https://example.com",
    "headers": { "content-type": "text/html" },
    "cookies": { "cf_clearance": "..." }
  },
  "metadata": {
    "method_used": "fast",
    "elapsed_time": 0.42,
    "solver_used": false
  },
  "billing": {
    "charged": 0.0035,
    "balance_after": 49.97
  }
}

Check both status layers before trusting the content:

r = resp.json()
if not r["success"]:
    raise Exception(r["error"])          # API error (auth, balance, unlock failed)
if r["data"]["status_code"] >= 400:
    raise Exception(r["data"]["status_code"])  # target returned 4xx/5xx
content = r["data"]["content"]           # you're good

3. Choose a mode

mode controls whether a real browser is used:

Mode	Browser?	Use when
`auto` (default)	Only if needed	You're not sure — starts fast, auto-escalates to browser if the page is protected or JS-heavy
`fast`	No	Static HTML, JSON APIs, sitemaps — you want lowest cost
`js_rendering`	Always	You know the site needs JS, or `auto` keeps returning empty/blocked content

{ "url": "https://protected.example.com", "mode": "js_rendering" }

Start with auto. Only pin fast or js_rendering when you have a specific reason.

4. Choose an output format

output_format controls what comes back in data.content:

Format	Returns	Use for
`html` (default)	Raw HTML	Custom parsing, full DOM
`markdown`	Clean Markdown	LLM input, content pipelines
`plain_text`	Text only	Search indexing, NLP
`autoparse`	Auto-detected JSON	Quick structured data without selectors
`screenshot`	Base64 PNG	Visual capture

{ "url": "https://blog.example.com/post", "output_format": "markdown" }

5. Extract specific fields

Don't want to parse the whole HTML yourself? Pass css_selectors — OmniScrape runs the selectors server-side and returns only the values you need:

{
  "url": "https://shop.example.com/product/1",
  "mode": "js_rendering",
  "css_selectors": {
    "title": "h1.product-title",
    "price": ".price-amount",
    "stock": ".stock-status"
  },
  "js_wait_selector": ".price-amount"
}

Response includes data.css_extracted:

{
  "success": true,
  "data": {
    "css_extracted": {
      "title": "Wireless Headphones X200",
      "price": "$129.00",
      "stock": "In stock"
    },
    "status_code": 200
  }
}

For common patterns (links, images, emails, tables) you can use built-in templates instead of writing selectors:

{ "url": "https://example.com", "templates": ["links", "images", "emails"] }

Results appear in data.template_extracted.

6. Use proxies and geo-targeting

Add proxy to route through a residential IP from a specific country:

{ "url": "https://shop.example.com", "proxy": "residential:de" }

217 countries supported — use ISO 3166-1 alpha-2 codes (us, gb, de, fr, jp, br, id, sg, etc.).

Control IP rotation

Add a third segment to change how IPs rotate:

Value	What happens
`residential:de`	Same IP reused for the session (default — sticky)
`residential:de:rotate`	Fresh IP on every single request
`residential:de:smart`	Reuses same IP, auto-swaps if it gets blocked

smart is ideal for large Cloudflare-protected catalogues — it keeps the cached clearance cookie as long as possible, then switches IPs automatically when one gets blocked.

7. Keep the same IP across requests (sessions)

session_id pins one residential IP and cookie jar across multiple requests. Use this for:

Pagination — site ties page state to a cookie or IP
Logged-in scraping — auth once, reuse the cookie
Cloudflare-protected sites — reuse the solved clearance cookie instead of re-solving every page

{
  "url": "https://shop.example.com/de/page/2",
  "proxy": "residential:de",
  "session_id": "de-shop-crawl"
}

Sessions expire after ~10 minutes of inactivity. Omit session_id for independent parallel requests — fresh IPs spread load better.

8. Wait for JS content to load

For JS-heavy pages, tell OmniScrape to wait for a specific element before returning:

{
  "url": "https://spa.example.com/results",
  "mode": "js_rendering",
  "js_wait_selector": ".results-list",
  "js_wait_timeout": 8000
}

Returns the moment the selector appears. If it never appears, waits up to js_wait_timeout ms (default: 5000).

Capture background API calls instead of parsing HTML

Many SPAs load their data via XHR/fetch — capture those raw payloads directly:

{
  "url": "https://spa.example.com/products",
  "mode": "js_rendering",
  "capture_xhr": true
}

Results appear in data.xhr_requests. Often cleaner than parsing the DOM.

9. Run browser actions before returning

Use js_actions to click, fill, or scroll before OmniScrape captures the page — useful for search forms, popups, or "load more" buttons:

{
  "url": "https://example.com/search",
  "mode": "js_rendering",
  "js_actions": [
    { "action": "fill",     "selector": "#search-input", "value": "laptop" },
    { "action": "click",    "selector": "button[type=submit]" },
    { "action": "wait_for", "selector": ".search-results" }
  ]
}

Available actions: click, fill, select, check, uncheck, wait, wait_for, scroll_y, scroll_x, scroll_depth, load_more, next_page.

For full interactive flows (real login, OTP, checkout), use Browser-as-a-Service.

10. Take a screenshot

{
  "url": "https://example.com",
  "screenshot": true,
  "screenshot_type": "fullpage"
}

Returns a base64 PNG in data.screenshot. Types: viewport (default), fullpage, element. For element: add screenshot_selector: ".my-chart".

Or use "output_format": "screenshot" to get only the image with no other data.

All parameters

Field	Type	Default	Description
`url`	string	required	Target URL including scheme (`https://`).
`method`	string	`GET`	`GET` or `POST`.
`body`	object/string	—	Body sent to target when `method` is `POST`.
`mode`	string	`auto`	`auto`, `fast`, or `js_rendering`.
`output_format`	string	`html`	`html`, `markdown`, `plain_text`, `autoparse`, `screenshot`.
`timeout`	integer	`30`	Max seconds to wait (10–90).
`enable_solver`	boolean	`true`	Auto-solve anti-bot challenges (Cloudflare, DataDome, etc.).
`proxy`	string	—	`residential:<cc>` or `residential:<cc>:<mode>`. Modes: `sticky` / `rotate` / `smart`.
`session_id`	string	—	Pins same IP and cookie jar across requests.
`custom_headers`	object	`{}`	Extra headers sent to the target.
`custom_cookies`	object	`{}`	Cookies sent with the request.
`css_selectors`	object	—	`{ "key": "css selector" }` → results in `data.css_extracted`.
`templates`	array	`[]`	Built-ins: `links`, `images`, `emails`, `tables`, `headings`, `metadata`, `phone_numbers`. Results in `data.template_extracted`.
`js_wait_selector`	string	—	Wait for this CSS selector before returning. Forces browser rendering.
`js_wait_timeout`	integer	`5000`	Max ms to wait for `js_wait_selector` (1000–60000).
`capture_xhr`	boolean	`false`	Capture XHR/fetch calls into `data.xhr_requests`.
`js_actions`	array	`[]`	Browser steps before returning. Forces `js_rendering`.
`screenshot`	boolean	`false`	Return base64 PNG in `data.screenshot`.
`screenshot_type`	string	`viewport`	`viewport`, `fullpage`, or `element`.
`screenshot_selector`	string	—	CSS selector when `screenshot_type` is `element`.

Full response reference

{
  "success": true | false,
  "error": "...",                     // present when success is false
  "data": {
    "content": "...",                 // page in output_format
    "css_extracted": { ... },         // present when css_selectors used
    "template_extracted": { ... },    // present when templates used
    "xhr_requests": [ ... ],          // present when capture_xhr: true
    "screenshot": "base64...",        // present when screenshot: true
    "status_code": 200,               // target's HTTP status
    "final_url": "https://...",       // URL after redirects
    "headers": { ... },               // target response headers
    "cookies": { ... }                // cookies set by target
  },
  "metadata": {
    "method_used": "fast",            // "fast" or "js_rendering"
    "elapsed_time": 0.42,             // seconds
    "solver_used": false,             // true if anti-bot solver ran
    "challenge_solved": false         // true if challenge was solved
  },
  "billing": {
    "charged": 0.0035,                // 0 if request failed
    "balance_after": 49.97
  }
}

Error handling

HTTP	Meaning	Action
`200`	Success	Read `data` — still check `data.status_code`
`400`	Bad parameter	Fix the request body — check `error` message
`401`	Invalid API key	Check `X-API-Key` header
`402`	Balance / trial issue	Top up, or read `code` field (`TRIAL_EXPIRED`, `PROXY_LOCKED`)
`429`	Too many concurrent requests	Back off, retry — see Rate limits
`502`	Unlock failed — not billed	Try `js_rendering`, different proxy country, or increase `timeout`
`500/503/504`	Server error	Retry with exponential backoff (1s → 2s → 4s → 8s)

Never retry 400, 401, or 402 — they will keep failing until you fix the underlying issue.

Large batches — async mode

POST /v1/scrape holds the HTTP connection open until done. For large batches or slow targets, submit async instead:

# Submit
curl -X POST https://api.omniscrape.io/v1/scrape/async \
  -H "X-API-Key: $OMNISCRAPE_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com" }'
# → { "job_id": "3f9a2c8d-..." }

# Poll
curl https://api.omniscrape.io/v1/jobs/3f9a2c8d-... \
  -H "X-API-Key: $OMNISCRAPE_KEY"
# → { "status": "completed", "result": { ... } }

Same request body. Results kept 24 hours. Poll every 2–5 seconds.

Need a real interactive browser?

Web Unlocker handles most scraping. Use Browser-as-a-Service when you need to:

Drive a real login flow (email + password + OTP)
Keep a live browser open across many pages in sequence
Run automation that goes beyond what js_actions supports

1. Make a request​

2. Understand the response​

3. Choose a mode​

4. Choose an output format​

5. Extract specific fields​

6. Use proxies and geo-targeting​

Control IP rotation​

7. Keep the same IP across requests (sessions)​

8. Wait for JS content to load​

Capture background API calls instead of parsing HTML​

9. Run browser actions before returning​

10. Take a screenshot​

All parameters​

Full response reference​

Error handling​

Large batches — async mode​

Need a real interactive browser?​