# Web Structured Data Extractor (Claude, JSON Schema) (`gochujang/web-structured-extractor`) Actor

Pass a URL + JSON schema (or natural-language goal). Claude reads the page and returns a strict JSON object matching your schema. Product / news / hotel / real-estate / job-board extraction. BYO Anthropic API key. $0.01 per page.

- **URL**: https://apify.com/gochujang/web-structured-extractor.md
- **Developed by:** [Hojun Lee](https://apify.com/gochujang) (community)
- **Categories:** Developer tools
- **Features:** Limited permissions
- **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 page extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Structured Data Extractor (Claude)

> Pass a URL + JSON schema (or natural-language goal). Claude reads the page and returns a strict JSON object matching your schema. **Product / news / hotel / real-estate / job-board extraction.** BYO Anthropic API key. **$0.01 per page.**

***

### Input Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `url` | string | `https://apify.com/gochujang/smart-money-tracker` | URL of the page to extract from. |
| `selectors` | array | page `title` + first `h1` | CSS selectors `[{name, selector, attr}]`; no API key needed. |
| `schema` | object | `{}` | If provided, Claude will fill an object matching this schema. Use 'goa |
| `goal` | string | ``| Natural-language description (e.g. 'price, in stock, top review'). Use |
| `model` | string | `claude-opus-4-7` | Claude model. |
| `anthropicApiKey` | string | `—` | Get one at console.anthropic.com. |
| `userAgent` | string |`` | Custom UA for page fetch. |

***

### Why this exists

You want to scrape structured fields out of arbitrary web pages — price, SKU, rating, hours, contact info, reviews. Building a per-site CSS-selector scraper is brittle (sites change every week). General-purpose LLM extraction is robust but requires prompting + parsing.

This actor wraps the whole pipeline:

1. URL → clean Markdown via trafilatura
2. Markdown + your schema/goal → Claude
3. Claude returns strict JSON → we parse and validate

Same idea as DiffBot's Article API ($299/mo) or Browse AI's extraction ($99/mo) — but with your own Claude API key and no monthly subscription.

***

### What you get

```json
{
  "url": "https://...",
  "model": "claude-opus-4-7",
  "goal": "Extract product info",
  "schema_used": true,
  "extracted_data": {
    "name": "Nintendo Switch 2",
    "price_usd": 499.99,
    "in_stock": true,
    "rating": 4.8,
    "reviews_count": 1247
  },
  "raw_output": "...",
  "input_chars": 5230,
  "usage": {"input_tokens": 1450, "output_tokens": 80}
}
```

***

### Two ways to specify what to extract

#### Option 1: JSON Schema (recommended)

```json
{
  "url": "https://www.amazon.com/dp/B07VPHN6CR",
  "schema": {
    "type": "object",
    "properties": {
      "name":         {"type": "string"},
      "price_usd":    {"type": "number"},
      "in_stock":     {"type": "boolean"},
      "rating":       {"type": "number"},
      "reviews_count":{"type": "integer"}
    }
  },
  "anthropicApiKey": "sk-ant-..."
}
```

#### Option 2: Natural language goal

```json
{
  "url": "https://...",
  "goal": "Extract product name, USD price, in-stock status, average rating",
  "anthropicApiKey": "sk-ant-..."
}
```

You can combine both — schema sets the shape, goal sets emphasis.

***

### Use cases

1. **E-commerce competitor monitoring** — Track price + availability across competitors
2. **Real estate listing aggregation** — Extract beds/baths/price/sqft from Zillow, Redfin, Realtor
3. **Job board scraping** — Title, company, salary, location, remote-flag from LinkedIn, Indeed
4. **News article fact extraction** — Get the same 5 fields from any news source
5. **Hotel / travel research** — Name, rating, price/night, amenities from any booking site

***

### Pricing

**Pay-Per-Event**: `$0.01 per page` (Apify-side).

Anthropic tokens charged separately. Typical:

| Page complexity | Input tokens | Anthropic | Total |
|---|---|---|---|
| Simple product page | ~1500 | ~$0.008 | $0.018 |
| Long article | ~4000 | ~$0.020 | $0.030 |
| Big e-commerce listing | ~8000 | ~$0.040 | $0.050 |

Use Haiku for batch / cheap extraction (~10x cheaper).

***

### Setting your Anthropic API key

See [Article Summarizer README](https://apify.com/gochujang/article-summarizer) for the full BYO API key guide. Short version:

1. Get key at console.anthropic.com
2. Paste in `anthropicApiKey` input (Apify saves it encrypted)
3. Or save as Apify Account-level Secret and reference as `@MY_KEY`

***

### Tips for reliable extraction

- **Better prompts → better results.** A `goal` string like "extract product name, USD price, in-stock boolean, rating 1-5" outperforms "extract product info".
- **Constrain types in the schema.** `"type": "number"` is stricter than `"type": ["string","number","null"]`.
- **Test with Haiku first.** Haiku 4.5 is fast and 10x cheaper for prototyping. Switch to Opus 4.7 when you need accuracy.

***

### Related actors (same author)

- [Article Summarizer](https://apify.com/gochujang/article-summarizer) — TL;DR instead of structured
- [Web Page → Markdown Converter](https://apify.com/gochujang/web-to-markdown) — Just the body, no LLM
- [HTML Metadata Extractor](https://apify.com/gochujang/html-metadata-extractor) — Cheaper for OG / Twitter / JSON-LD
- [JSON Schema Generator](https://apify.com/gochujang/json-schema-generator) — Bootstrap a schema from samples

***

### Feedback

A short review helps engineers find it: [Leave a review on Apify Store](https://apify.com/gochujang/web-structured-extractor#reviews)

**Keywords:** web data extraction, AI web scraper, structured data scraper, JSON extraction tool, Claude web scraper, LLM data extraction, webpage to JSON, no-code web scraper, schema-based extraction

# Actor input Schema

## `url` (type: `string`):

URL of the page to extract from.

## `schema` (type: `object`):

If provided, Claude will fill an object matching this schema. Use 'goal' instead/alongside for free-form requests.

## `goal` (type: `string`):

Natural-language description (e.g. 'price, in stock, top review'). Used if 'schema' is not provided.

## `selectors` (type: `array`):

Batch CSS selector extraction. Each item: {"name": "price", "selector": ".price-box", "attr": "text"}. attr can be 'text' or any HTML attribute (e.g. 'src', 'href'). Defaults to the page title and first <h1> so a run works without an API key; replace with your own selectors.

## `xpathSelectors` (type: `array`):

XPath extraction. Each item: {"name": "price", "xpath": "//span\[@class='price']/text()"}. Returns first match.

## `parseTables` (type: `boolean`):

If true, all <table> elements are parsed into {headers, rows} JSON arrays and returned in 'tables'.

## `extractList` (type: `boolean`):

If true, extract repeated elements matching 'listSelector'. Use 'fieldSelectors' to extract fields within each item.

## `listSelector` (type: `string`):

CSS selector for the repeating container element (e.g. '.product-card', 'article.news-item').

## `fieldSelectors` (type: `array`):

Fields to extract within each list item. E.g. \[{"name": "title", "selector": "h2", "attr": "text"}, {"name": "link", "selector": "a", "attr": "href"}].

## `waitForSelector` (type: `string`):

CSS selector to wait for before extracting. Triggers Playwright-based JS rendering (requires playwright installed in Dockerfile). Leave blank for static pages.

## `model` (type: `string`):

Claude model (only needed when using schema/goal).

## `anthropicApiKey` (type: `string`):

Get one at console.anthropic.com. Required only when using schema or goal.

## `userAgent` (type: `string`):

Custom UA for page fetch.

## Actor input object example

```json
{
  "url": "https://apify.com/gochujang/smart-money-tracker",
  "schema": {},
  "goal": "",
  "selectors": [
    {
      "name": "title",
      "selector": "title",
      "attr": "text"
    },
    {
      "name": "h1",
      "selector": "h1",
      "attr": "text"
    }
  ],
  "xpathSelectors": [],
  "parseTables": false,
  "extractList": false,
  "listSelector": "",
  "fieldSelectors": [],
  "waitForSelector": "",
  "model": "claude-opus-4-7",
  "userAgent": ""
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://apify.com/gochujang/smart-money-tracker",
    "selectors": [
        {
            "name": "title",
            "selector": "title",
            "attr": "text"
        },
        {
            "name": "h1",
            "selector": "h1",
            "attr": "text"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("gochujang/web-structured-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "url": "https://apify.com/gochujang/smart-money-tracker",
    "selectors": [
        {
            "name": "title",
            "selector": "title",
            "attr": "text",
        },
        {
            "name": "h1",
            "selector": "h1",
            "attr": "text",
        },
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("gochujang/web-structured-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://apify.com/gochujang/smart-money-tracker",
  "selectors": [
    {
      "name": "title",
      "selector": "title",
      "attr": "text"
    },
    {
      "name": "h1",
      "selector": "h1",
      "attr": "text"
    }
  ]
}' |
apify call gochujang/web-structured-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gochujang/web-structured-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/VvfPSprbITb7HwstS/builds/fMCVd1uDBH8CyvMMV/openapi.json