Skip to main content

Sessions

A session keeps the same IP address and cookie jar across multiple requests. Use one whenever a sequence of requests needs to look like a single, continuous visitor — paginated crawls, multi-step flows, or anything behind a login.

How sessions work​

Pick any string as a session_id and reuse it. The first request that uses a given ID establishes the session (pins an IP, starts a fresh cookie jar). Every later request with the same ID is routed through the same IP and carries the accumulated cookies.

{
  "url": "https://example.com/page/1",
  "session_id": "my-crawl"
}
{
  "url": "https://example.com/page/2",
  "session_id": "my-crawl"
}

Both requests share one IP and accumulate cookies, so pagination and "remember me" state work as expected.

When to use a session​

  • Pagination — sites that tie page state to a cookie or IP.
  • Logged-in scraping — authenticate once, reuse the cookie.
  • Multi-step flows — add to cart, then check out.
  • Cloudflare-protected sites — a solved clearance cookie is cached on the worker; reusing the same session avoids re-solving on every page.
  • Rate-sensitive sites — a stable identity looks more human than a new IP each time.

When NOT to use a session​

For independent, parallel requests across many unrelated URLs, omit session_id. Fresh IPs spread load and reduce the chance of one IP getting flagged. Use proxy: "residential:<country>:rotate" explicitly if you want guaranteed fresh IPs.

Session lifetime​

Sessions expire after roughly 10 minutes of inactivity. After expiry, the same session_id starts a new session with a new IP. For long crawls, keep requests flowing or accept that a new IP may be assigned.

Combining with geo-targeting​

Pin a country and a session to crawl a regional site as one consistent visitor:

{
  "url": "https://shop.example.com/de/page/1",
  "proxy": "residential:de",
  "session_id": "de-shop-crawl"
}

Smart rotation with a session​

smart proxy mode starts sticky (reuses the same IP as a normal session) but automatically mints a new proxy session if the IP gets blocked, then continues without interrupting your session_id logic:

{
  "url": "https://protected.example.com/data",
  "proxy": "residential:us:smart",
  "session_id": "batch-crawl-01"
}

This is useful for large catalogues where most pages benefit from IP reuse, but occasional blocks would otherwise force you to handle rotation yourself. See Proxy Rotation Modes for the full picture.