Production Traffic & Bot Detection

Some websites serve a bot-verification challenge or an access-denied page instead of their content. This page explains exactly what WebUplink does when that happens, what it deliberately does not do, and what kinds of sites work well in production today.

What happens on a blocked page

When a site serves a challenge or denial page instead of real content, WebUplink returns a typed error — it does not hand your agent a PageMap of the interstitial:

{
  "error": "SITE_BLOCKED",
  "message": "The site presented a bot-verification challenge instead of the page. WebUplink does not solve CAPTCHAs or bypass bot detection.",
  "request_id": "req-abc-123"
}

• HTTP status: 502
• Not billed. A blocked request charges zero actions.
• Not retryable. There is no retry_after — a retry from the same infrastructure hits the same wall, so the SDKs never auto-retry it.
• The message tells you which kind of block it was: a bot-verification challenge (CAPTCHA, "verify you are human") or an access denied page.

Before classifying a page as blocked, WebUplink waits a moment (~2.5 seconds) and re-checks — many JavaScript challenges resolve on their own, and those pages come back as normal PageMaps. Only a persistent block becomes SITE_BLOCKED.

Detection covers the common signature families: Cloudflare challenge and block pages, PerimeterX, DataDome, Akamai-style access-denied pages, and bare CAPTCHA interstitials. It is deliberately conservative: a page about CAPTCHAs, or a login form with an embedded CAPTCHA widget, is never classified as blocked. The trade-off is that an unusual block page can occasionally slip through as a regular (billed) PageMap.

What WebUplink does not do

WebUplink does not attempt to defeat bot detection:

• No CAPTCHA solving. Automated CAPTCHA solving is explicitly disabled when every browser session is created.
• No residential proxies. Requests come from datacenter infrastructure.
• No stealth browser fingerprinting.

This is a deliberate stance, not a gap we paper over: our Terms of Service and Acceptable Use Policy prohibit circumventing access controls, CAPTCHAs, and bot-detection mechanisms, and the product holds itself to the same rule.

What works well today

Qualitatively, WebUplink is reliable on:

• Public content sites — news, blogs, documentation, reference sites
• Search and browse flows — search a site, filter results, open items, read pages
• Form workflows on sites without bot walls — fill fields, submit, follow the result
• Multi-step navigation within a session (see Sessions)

What will commonly fail

• Aggressively bot-protected retail and ticketing sites. (Ticketing — along with banking and streaming — is on the restricted list and returns DOMAIN_BLOCKED on every plan regardless.)
• Sites that require a CAPTCHA at login. WebUplink will not solve it; the request returns SITE_BLOCKED.
• Sites that block datacenter IP ranges outright.

We don't publish a coverage percentage — which sites block automated traffic changes week to week, and a single number would be stale the day it shipped. Test against the sites your agent actually needs; blocked requests cost you nothing.

Sessions are ephemeral

Browser sessions expire after 2 minutes idle and approximately 15 minutes total, so login state does not persist across sessions. For multi-step workflows and re-authentication patterns, see Sessions.

Roadmap

Persistent authenticated profiles are the top item on our roadmap — keeping a logged-in state available across sessions so agents don't re-authenticate on every run. We are not promising proxy or stealth capabilities.

Handling SITE_BLOCKED in your agent

import { WebUplink, WebUplinkError } from 'webuplink';

const client = new WebUplink();

try {
  const page = await client.browse('https://example.com');
} catch (err) {
  if (err instanceof WebUplinkError && err.code === 'SITE_BLOCKED') {
    // Not billed, not retryable — route around it:
    // try an alternative source, or surface the block to the user.
    console.log(err.message); // names the block: challenge vs. access denied
  }
}

See the Error Reference for the full error taxonomy.