blocked-page-recovery

1---2name: blocked-page-recovery3description: "Use when a fetch fails: 403/429, paywall, WAF, bot wall."4version: 1.0.05author: Hermes Agent6license: MIT7platforms: [linux, macos, windows]8metadata:9  hermes:10    tags: [Research, Archives, Wayback, Paywall, WAF, Fallback]11    related_skills: [grounded-citations]12---13 14# Blocked-Page Recovery15 16When a page won't fetch — 403/429, Cloudflare "Just a moment...", a paywall,17or a bot-detection interstitial — don't give up and don't loop on the same18URL. Third-party services often hold a **copy** of the page. Work down this19ladder, cheapest first.20 21## The ladder22 23```241. Wayback Machine  — archive.org "available" API  (snapshot + timestamp)252. archive.today    — domain rotation: archive.ph → .md → .li → .is263. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)274. API-first pivot  — look for /api/, /graphql, .json, or RSS on the same host285. Real browser     — browser tool as the last, most expensive resort29```30 31Run it in one shot with the bundled script:32 33```bash34python3 scripts/recover_page.py "https://example.com/blocked-article" --json35```36 37The script tries each route in order, validates every body (see "Fake38successes" below), and prints the first genuine hit with its provenance.39 40## Provenance discipline (non-negotiable)41 42Every recovered copy carries a provenance you MUST preserve when citing:43 44| Route | Provenance | How to cite |45|-------|-----------|-------------|46| Wayback / archive.today | `snapshot` | Cite WITH the snapshot date: "as archived 2026-08-06". Never present a snapshot as the live page — it may be stale. |47| Jina Reader | `live` | Server-side re-render of the live page; cite normally. |48| Live fetch / browser | `live` | Cite normally. |49 50If the user needs *current* data (prices, availability, breaking news), a51snapshot is context, not an answer — say so explicitly and note its age.52 53## Manual routes54 55### 1. Wayback Machine (best provenance, try first)56 57```bash58# Discovery: returns closest snapshot URL + timestamp as JSON59curl -sL "https://archive.org/wayback/available?url={URL}"60# Then fetch archived_snapshots.closest.url61```62 63For enumerating many snapshots (or recovering deleted pages), the CDX index:64 65```bash66curl -sL "https://web.archive.org/cdx/search/cdx?url={URL}&output=json&limit=10"67```68 69CDX intermittently returns 503 under load — if it does, fall back to the70`available` API; don't retry-hammer it.71 72Works for: any publicly crawled URL. Fails for: robots-blocked sites,73never-crawled URLs, JS-only SPAs (snapshots don't render).74 75### 2. archive.today (paywalls, deleted content)76 77User-submitted archives — often has paywalled news articles Wayback lacks.78Rate-limits aggressively (429) and rotates domains, so iterate:79 80```bash81for d in archive.ph archive.md archive.li archive.is; do82  curl -sL --max-time 20 "https://$d/newest/{URL}" -o /tmp/page.html \83    -w "%{http_code}" && break84done85```86 87**Validate the body, not the status code** — a 429 still ships several KB of88rate-limit HTML that looks like a success to a size check alone.89 90### 3. Jina Reader (requires JINA_API_KEY)91 92`r.jina.ai` re-renders the live page in a real browser server-side and93returns markdown. Anonymous access is dead (401 → Turnstile); a key is94required:95 96```bash97curl -s -H "Authorization: Bearer $JINA_API_KEY" "https://r.jina.ai/{URL}"98```99 100Handles JS SPAs that archives can't. Skip this route entirely when the env101var is unset.102 103### 4. API-first pivot104 105WAFs protect the HTML surface far more aggressively than the data endpoints106behind it. After 2-3 blocked attempts on a site, stop fighting the HTML and107look for:108 109- `/api/...`, `/graphql`, or `.json` variants of the page URL110- An RSS/Atom feed (`/feed`, `/rss`, `<link rel="alternate">` in any copy111  you did recover)112- A sitemap (`/sitemap.xml`) revealing canonical URLs that may not be gated113 114## Fake successes — routes that LIE115 116These return HTTP 200 with a plausible body that is NOT the page. The script117rejects them automatically; reject them manually too:118 119- **Google Cache is dead** (since mid-2024). `webcache.googleusercontent.com`120  returns 200 + tens of KB, but it's a Google Search interstitial with a JS121  redirect, not a cache. Never use it.122- **AMP caches** (`*.cdn.ampproject.org`) mostly return a ~300-byte123  `<title>Redirecting</title>` meta-refresh stub pointing back at the124  original (blocked) URL. Treating that as success creates a fetch loop.125- **Rate-limit bodies**: archive.today 429 pages are multi-KB HTML. Check for126  the target's actual content (title words, expected strings), not just size.127 128Detection heuristics the script applies: body under a per-route byte floor;129meta-refresh/JS-redirect stubs whose target is the original host; interstitial130titles ("Just a moment", "Redirecting", "Google Search", "Attention Required").131 132## Proxy relays: don't133 134Generic "web proxy" relays are man-in-the-middle by construction. Never send135cookies or Authorization headers through one, and don't use them for anything136the user will rely on — provenance is unverifiable. Prefer archives, which at137least timestamp their copies.138