# Search & Scrape API

Two HTTP services behind one domain:

| Base URL | Service | Use it to |
|---|---|---|
| `https://api.georgeafield.com/serp` | **SERP** (OpenSERP) | Search Google, Bing, DuckDuckGo, Yandex, Baidu, Ecosia; get ranked results as JSON or markdown; optionally pull page content for the top results |
| `https://api.georgeafield.com/scrape` | **Scrape** (novelscrape) | Fetch one URL past bot protection and get clean HTML, agent-ready markdown, text, or a PNG screenshot |

Typical agent flow: search with `/serp/google/search`, pick URLs, then read them with `/scrape/scrape?format=markdown`.

## Authentication

Every `/serp/*` and `/scrape/*` request needs the API key. These docs (`/`, `/llms.txt`) are public.

```sh
curl -H "Authorization: Bearer $API_KEY" "https://api.georgeafield.com/serp/google/search?text=golang"
```

The key is accepted as any of:

| Where | Example |
|---|---|
| `Authorization` header (preferred) | `Authorization: Bearer YOUR_KEY` |
| `X-API-Key` header | `X-API-Key: YOUR_KEY` |
| `api_key` query parameter (for opening URLs in a browser) | `/scrape/view?url=https://example.com&api_key=YOUR_KEY` |

A missing or wrong key returns `401` with `{"error": "unauthorized", "message": "send Authorization: Bearer <API_KEY>"}`.

## Conventions

- All endpoints are `GET` with query parameters unless noted. URL-encode parameter values.
- JSON responses are UTF-8. Timestamps are RFC 3339.
- Requests can be slow: a search takes 1-10 s; a scrape that needs a real browser or the proxy can take up to ~90 s. Use a client timeout of at least 120 s.

---

# SERP

## GET /serp/{engine}/search

Web search on one engine.

`engine`: `google`, `bing`, `duck` (DuckDuckGo), `yandex`, `baidu`, `ecosia`.

```sh
curl -H "Authorization: Bearer $API_KEY" "https://api.georgeafield.com/serp/google/search?text=how+do+mcp+servers+work&limit=10"
```

### Parameters

| Name | Type | Default | Description |
|---|---|---|---|
| `text` | string | required | Search query |
| `limit` | int 1-100 | `10` | Organic results to return. `<=10` parses only the first results page; more may paginate. Ads can be returned on top of this. |
| `start` | int >= 0 | `0` | Pagination offset. Use `pagination.next_start` from the previous response. |
| `lang` | string | | Language code, e.g. `EN`, `DE` |
| `region` | string | | Market hint: country (`DE`), locale (`en-GB`); Google also accepts a city (`Berlin`), Yandex a numeric region (`213`) |
| `date` | string | | Date range `YYYYMMDD..YYYYMMDD`, e.g. `20250101..20251231` |
| `site` | string | | Restrict to a domain, e.g. `github.com` |
| `file` | string | | File type, e.g. `pdf` |
| `format` | enum | `json` | `json`, `markdown`, `text`, or `ndjson` (one result per line) |
| `extract` | bool or int 0-5 | `false` | Fetch and embed page content for organic results: `true`/`1` = top result, `N` = top N. Adds latency. |
| `extract_mode` | enum | `auto` | `auto` (raw HTTP, browser if thin), `fast` (raw HTTP only), `rendered` (browser) |
| `min_runes` | int | built-in | In `auto` mode, escalate to the browser if raw extraction yields fewer characters than this |
| `filter` | bool | `true` | Google only: hide near-duplicate results |
| `features` | bool | `true` | Fill `serp_features` (AI overview, featured snippet, people also ask, ...) |
| `resolve_urls` | bool or int | `true` | Google only: turn redirect links into destination URLs (`true`, `false`, or top `N`) |

Bing localizes by exit IP: `region` is ignored for `US`, `CN`, `RU`.

### Response (`format=json`)

```json
{
  "query": { "text": "golang", "lang": "EN", "region": "US", "engines_requested": ["google"] },
  "meta": {
    "request_id": "01HXYZ...",
    "requested_at": "2026-04-24T12:00:00Z",
    "took_ms": 842,
    "engines_failed": [],
    "version": "2.1"
  },
  "results": [
    {
      "id": "s_a1b2c3d4e5f6a1b2",
      "rank": 1,
      "type": "organic",
      "title": "The Go Programming Language",
      "url": "https://go.dev/",
      "display_url": "go.dev",
      "snippet": "Go is an open source programming language...",
      "domain": "go.dev",
      "favicon": "https://go.dev/favicon.ico",
      "position": { "absolute": 1 },
      "engine": "google",
      "domain_info": { "tld": "dev", "sld": "go", "category": "" },
      "extracted": {
        "title": "The Go Programming Language",
        "format": "markdown",
        "content": "# Build simple, secure, scalable systems with Go\n...",
        "mode_used": "fast",
        "fetched_at": "2026-04-24T12:00:01Z"
      }
    }
  ],
  "serp_features": [
    {
      "id": "f_a1b2c3d4e5f6a1b2",
      "engine": "google",
      "type": "ai_summary",
      "text": "Go is an open source programming language used for fast, reliable services.",
      "links": [{ "title": "The Go Programming Language", "url": "https://go.dev/" }],
      "source_result_ids": ["s_a1b2c3d4e5f6a1b2"],
      "position": { "absolute": 1 },
      "confidence": 0.95,
      "extracted_at": "2026-04-24T12:00:00Z"
    }
  ],
  "pagination": { "page": 1, "has_more": true, "next_start": 10 }
}
```

- `results[].type`: `organic` or `ad` for normal rows; filter on `organic` for ranking.
- `results[].extracted` is present only when `extract` is set. On failure it has `error` instead of `content`.
- `serp_features[].type` is one of: `featured_snippet`, `knowledge_panel`, `people_also_ask`, `video`, `image`, `news`, `shopping`, `local`, `answer_box`, `ai_summary`, `related_questions`, `related_searches`, `sitelinks`, `videos`, `images_inline`, `calculator`, `weather`, `dictionary`.
- Useful response headers: `X-Request-ID`, `X-Cache` (`HIT`/`MISS`/`BYPASS`), `X-Proxy-Used` (`direct` or the masked proxy host).

## GET /serp/{engine}/image

Image search. Same parameters as search (no `extract`). Engines: `google`, `bing`, `duck`, `yandex`, `baidu`.

```json
{
  "results": [
    {
      "id": "i_9f8e7d6c5b4a3f2e",
      "rank": 1,
      "type": "image",
      "title": "Go gopher logo",
      "image": { "url": "https://.../gopher.png", "thumbnail": "https://.../thumb.jpg", "width": 1200, "height": 800 },
      "source": { "page_url": "https://go.dev/blog/gopher", "domain": "go.dev" },
      "engine": "bing"
    }
  ]
}
```

(Plus the same `query`, `meta`, `pagination` as search.)

## GET /serp/mega/search

Search several engines in one call.

```sh
curl -H "Authorization: Bearer $API_KEY" "https://api.georgeafield.com/serp/mega/search?text=golang&engines=google,bing&mode=balanced"
```

All `/serp/{engine}/search` parameters, plus:

| Name | Type | Default | Description |
|---|---|---|---|
| `engines` | csv | all enabled | e.g. `google,bing,duckduckgo` |
| `mode` | enum | `balanced` | `balanced`: all engines in parallel, merged. `any`: try engines in order until one succeeds. `fast`: only the currently fastest engine. |
| `dedupe` | bool | `true` | Balanced mode: drop duplicate URLs across engines |
| `merge` | bool | `true` | Balanced mode: merge duplicates into one ranked list |

Response: the search envelope plus `clusters`, one per unique URL:

```json
{
  "clusters": [
    {
      "id": "c_...",
      "canonical_url": "https://go.dev/",
      "domain": "go.dev",
      "title": "The Go Programming Language",
      "occurrences": [ { "engine": "google", "rank": 1, "result_id": "s_..." }, { "engine": "bing", "rank": 2, "result_id": "s_..." } ],
      "engines_count": 2,
      "best_rank": 1,
      "score": 1.83
    }
  ]
}
```

Partial failures are listed in `meta.engines_failed` and `meta.engine_errors`. If every engine fails, the status is `502`.

`GET /serp/mega/image` does the same for images. `GET /serp/mega/engines` lists engines and their current health.

## GET /serp/extract

Extract readable content from one URL (lightweight, no anti-bot tiers; use `/scrape/scrape` for protected sites).

| Name | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | Absolute URL |
| `mode` | enum | `auto` | `auto`, `fast`, `rendered` |
| `format` | enum | `json` | `json`, `markdown`, `text` |
| `clean` | bool | `true` | Article-only. `false` keeps the whole readable body (landing pages, nav-heavy sites). |
| `use_llms_txt` | bool | `false` | For a site root, return its `/llms-full.txt` or `/llms.txt` if present |
| `lang` | string | | Sent as `Accept-Language` |

```json
{
  "url": "https://example.com/post",
  "title": "Post title",
  "description": "Meta description",
  "markdown": "# Post title\n\nBody...",
  "text": "Post title\n\nBody...",
  "headings": [ { "level": 1, "text": "Post title" } ],
  "links": [ { "text": "Docs", "url": "https://example.com/docs" } ],
  "canonical": "https://example.com/post",
  "lang": "en",
  "meta": { "mode_used": "fast", "fetched_at": "2026-04-24T12:00:00Z", "bytes": 48213, "took_ms": 412 }
}
```

`POST /serp/extract` takes the same fields as a JSON body.

## POST /serp/extract/batch

Up to 20 URLs at once.

```sh
curl -H "Authorization: Bearer $API_KEY" -X POST "https://api.georgeafield.com/serp/extract/batch" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com", "https://go.dev"], "mode": "fast"}'
```

Returns a bare array, one item per unique URL. A failed URL has empty `page_content` and `metadata.error`; it does not fail the batch.

```json
[
  { "page_content": "# Example Domain\n...", "metadata": { "source": "https://example.com", "title": "Example Domain" } },
  { "page_content": "", "metadata": { "source": "https://go.dev", "error": "fetch failed: timeout" } }
]
```

## SERP errors

Errors are JSON with a stable `error` code:

```json
{ "error": "captcha_detected", "code": 502, "request_id": "01HXYZ...", "message": "google returned a captcha", "meta": { "engine": "google" } }
```

| Status | `error` values | What to do |
|---|---|---|
| 400 | `bad_request` | Fix the parameters |
| 429 | `rate_limited` | Back off and retry |
| 502 | `captcha_detected`, `blocked`, `parser_failure`, `all_engines_failed`, `proxy_*` | Retry later or use another engine / `mega/search?mode=any` |
| 503 | `circuit_open`, `service_unavailable` | Engine temporarily disabled after repeated failures; use another engine |
| 504 | `search_timeout`, `request_timeout` | Retry |

Full OpenSERP schema: `https://api.georgeafield.com/serp/openapi.yaml`.

---

# Scrape

Fetches a page with escalating methods, cheapest first, and stops at the first one that is not blocked:

| Tier | Method | Cost |
|---|---|---|
| `http` | HTTP client with Chrome TLS fingerprint | free |
| `stealth` | Real headless browser, solves Cloudflare | free (slower) |
| `stealth_proxy` | Real browser through a residential proxy | paid per GB |

A response counts as blocked on status 401/403/407/429/5xx, a challenge page (Cloudflare, captcha, PerimeterX, ...), or an empty body. `404` and `410` are real answers and are returned as-is.

## GET or POST /scrape/scrape

```sh
curl -H "Authorization: Bearer $API_KEY" "https://api.georgeafield.com/scrape/scrape?url=https://example.com&format=markdown"

curl -H "Authorization: Bearer $API_KEY" -X POST "https://api.georgeafield.com/scrape/scrape" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": "markdown"}'
```

| Name | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | Absolute `http(s)` URL |
| `format` | enum | `html` | `markdown`: main content only (no nav/footer/sidebars), hidden text removed, absolute links. Best for LLMs. `html`: full page with scripts, styles, iframes and event handlers removed. `text`: plain text. `raw`: the page exactly as fetched. |

### Response

```json
{
  "url": "https://example.com",
  "final_url": "https://example.com/",
  "status": 200,
  "ok": true,
  "tier": "http",
  "proxy_used": false,
  "cost": { "free": true, "proxy_bytes_est": 0, "usd_est": 0.0 },
  "attempts": [
    { "tier": "http", "status": 200, "blocked": null, "took_ms": 81 }
  ],
  "content": "# Example Domain\n\nThis domain is for use in documentation examples...\n"
}
```

| Field | Description |
|---|---|
| `ok` | `true` if the final attempt was not blocked. Check this before trusting `content`. |
| `status` | HTTP status of the final page (after redirects) |
| `final_url` | URL after redirects |
| `tier` | Tier that produced `content` |
| `proxy_used` | `true` if the paid proxy was needed |
| `cost.free` | `true` if no proxy bandwidth was used |
| `cost.proxy_bytes_est` | Estimated proxy bytes (from page size; approximate) |
| `cost.usd_est` | Estimated proxy cost in USD; `null` if the proxy price is not configured |
| `attempts[]` | One entry per tier tried: `tier`, `status`, `blocked` (reason or `null`), `took_ms`, or `error` if it failed outright |
| `content` | The page in the requested `format` |

### Status codes

| Status | Meaning |
|---|---|
| 200 | Got real content (`ok: true`) |
| 400 | `{"error": "url must be http(s)"}` or invalid `format` |
| 502 | Every tier was blocked or failed. If any page came back, `content` holds the last one (often a block page) and `ok` is `false`. If nothing came back: `{"url", "ok": false, "error": "all tiers failed", "attempts"}` |

## GET /scrape/screenshot

PNG screenshot of a page, using the browser tiers only (`stealth`, then `stealth_proxy`).

```sh
curl -H "Authorization: Bearer $API_KEY" -o page.png "https://api.georgeafield.com/scrape/screenshot?url=https://example.com&full_page=1"
```

| Name | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | Absolute `http(s)` URL |
| `full_page` | `1`/`true` | off | Whole page instead of the first screen. Scrolls to the bottom first so lazy-loaded and scroll-animated content appears. |

Response body is `image/png` (1920x1080 viewport at 2x pixel density). Metadata is in headers:

| Header | Example | Description |
|---|---|---|
| `X-Tier` | `stealth` | Tier that took the screenshot |
| `X-Status` | `200` | HTTP status of the page |
| `X-Blocked` | `false` | `true` if the image shows a block/captcha page |
| `X-Proxy-Used` | `false` | `true` if the paid proxy was used |
| `X-Cost-USD-Est` | `0.0` | Estimated proxy cost; empty if unknown |

Status `200` = real page. `502` with a PNG = every tier was blocked and the image shows the block page. `502` with JSON = no screenshot could be taken. `400` = bad `url`.

## GET /scrape/view

Human-facing viewer: open `https://api.georgeafield.com/scrape/view?url=https://example.com&api_key=YOUR_KEY` in a browser (the key is then remembered in a cookie) to see the markdown rendered, its token estimate and per-model input cost, request cost, and a screenshot link. Not meant for programmatic use; call `/scrape/scrape` instead.

## GET /scrape/health

```json
{ "status": "ok", "proxy_configured": true }
```

---

# Which endpoint?

| Need | Use |
|---|---|
| Find pages about a topic | `/serp/google/search` (or `/serp/mega/search` for several engines) |
| Search + read the top hits in one call | `/serp/google/search?extract=3&format=markdown` |
| Read a specific page, especially behind Cloudflare/anti-bot | `/scrape/scrape?format=markdown` |
| Read many ordinary pages at once | `POST /serp/extract/batch` |
| See what a page looks like | `/scrape/screenshot?full_page=1` |