Rate Limits & Concurrency

OmniScrape enforces two limits:

1. Concurrency — how many scrape jobs can be in flight at once (by plan).
2. Request rate — a per-account token bucket (~20 requests/second, burst 10) to protect the gateway. Exceeding it returns 429 with "Rate limit exceeded. Please slow down." (no code field).

Stay under both limits for smooth throughput.

Concurrency by plan

Plan	Max concurrent requests
Free trial	1
Pay-As-You-Go	5
Startup	10
Growth	25

A "concurrent request" is any scrape (sync or async) that is currently being processed. When a request finishes, its slot frees up immediately.

What happens at the limit

If you exceed your concurrency limit, the request returns 429 with a structured code and a Retry-After header:

{
  "success": false,
  "error": "Concurrent request limit reached for your plan. Retry shortly.",
  "code": "CONCURRENCY_LIMIT"
}

Wait the suggested time (or a couple of seconds) and retry.

Request rate

In addition to concurrency, each account has a token-bucket limit of roughly 20 requests per second (burst 10). Sending faster returns 429 with the message "Rate limit exceeded. Please slow down." — there is no code field on this response. Space out submissions or use the async jobs API for large batches.

Monthly quotas

Subscription plans include a monthly allowance of successful Web Unlocker requests. Those included requests do not draw down your prepaid balance. After the allowance is used, additional requests bill from balance at PAYG rates ($0.0035 each). Pay-As-You-Go has no monthly allowance — every success draws from balance.

Plan	Included successful requests / month
Pay-As-You-Go	Unlimited (balance-based)
Startup	10,000
Growth	40,000

If included requests are exhausted and balance cannot cover overage, the API returns 402 (insufficient balance). Top up to continue, upgrade the plan, or wait for the monthly reset. See Pricing.

Staying within limits

The simplest pattern is a bounded worker pool sized to your plan's concurrency:

import os, requests
from concurrent.futures import ThreadPoolExecutor

KEY = os.environ["OMNISCRAPE_KEY"]
MAX_CONCURRENCY = 5  # match your plan

def scrape(url):
    return requests.post(
        "https://api.omniscrape.io/v1/scrape",
        headers={"X-API-Key": KEY},
        json={"url": url, "mode": "auto"},
        timeout=120,
    ).json()

urls = [f"https://example.com/page/{i}" for i in range(1, 101)]
with ThreadPoolExecutor(max_workers=MAX_CONCURRENCY) as pool:
    results = list(pool.map(scrape, urls))

For very large batches, prefer the async jobs API so you submit work without holding connections open, and still cap how many jobs are unresolved at once.

Retry guidance

Retry 429, 500, 502, 503, and 504 with exponential backoff. Do not retry 400, 401, or 402. See Errors for the full strategy.