# AI Web Scraper - Extract Structured Data & URL to JSON (`tidytools/ai-web-data-extractor`) Actor

AI web scraper and AI extractor: URL to JSON. Name the fields or write a prompt, and an LLM reads each page and returns structured data. One row per list item, follows detail links, reads PDF and image documents. $0.01 per page with data. Failed pages are free.

- **URL**: https://apify.com/tidytools/ai-web-data-extractor.md
- **Developed by:** [Yukai Lin](https://apify.com/tidytools) (community)
- **Categories:** AI, Automation, Developer tools
- **Features:** Limited permissions
- **Stats:** 4 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $8.00 / 1,000 extracted pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### AI Web Scraper: extract structured data and turn a URL into JSON

#### What it does

Give it page URLs. Name the fields you want, or describe them in one sentence. An LLM reads each page and returns structured JSON in `data`. You do not need CSS selectors or code. Values that are not on the page come back as `null`.

| Input | Example |
|---|---|
| URL | `https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html` |
| Fields | `{"title":"string","price":"number","in_stock":"boolean"}` |
| JSON output (`data`) | `{"title":"A Light in the Attic","price":51.77,"in_stock":true}` |

The output above came from a real run on this page. It is not a live price check.

**Price:** $0.01 per page that returns data, on the Free plan. That is $10 per 1,000 pages. The price is lower on higher Apify plans (see below). Failed, blocked, timed-out and empty pages are free.

| Input | Example |
|---|---|
| URL | `https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html` |
| Fields | `{"title":"string","price":"number","in_stock":"boolean"}` |
| JSON output (`data`) | `{"title":"A Light in the Attic","price":51.77,"in_stock":true}` |

The `data` values above came from a real run on this Books to Scrape product page, documented in the 2026-10-28 README; they are not a live price check. On the Free plan, each page with data costs $0.01 and creates one `page-extracted` event. Failed, blocked, timed-out and empty pages are free. For a list page, choose *One row per item* instead. You can paste up to 10,000 URLs or use the URL field of another Actor's dataset.

With `reviewEvidence: true` on a public review page, reviewer names and review text are matched to individual star markup after HTML entity and whitespace normalization; ratings with no individual evidence or conflicting cards remain `null`. A card-header tail the page shows next to a reviewer's name (post count, Australian state) is removed from `author` when the page has a review card under the shorter name. Merchant replies the page marks as replies and text-less entries from the page's similar-shops list are not output as review rows, nor are other entries without review text on a page whose other rows have text; the page is still one `page-extracted` event. A page left with no review after that gets one free failure row (`errorType: "no_items"`, `failureStage: "list_items"`) and is not charged.

**Ready-made fields:** Skip writing field names: choose Property, Hotel, Menu, Recipe, Company directory, Product catalog or Job posting, then clear the form's sample *Fields*. Replace illustrative `.example` URLs with pages you own or may access:

- **Property** — input `{"urls":["https://your-agency.example/listings/123"],"preset":"property"}` → one row per page by default; add `"outputMode":"items"` for one row per visible listing on a list page (one event per successful page). Fields include `name`, `listingUrl`, `address`, `price`, `currency`, `bedrooms`, `bathrooms`, `area`, `areaUnit`, `area_sqm`, `status`, `agentName`, `agentPhone`.
- **Hotel** — input `{"urls":["https://your-hotel.example/rooms/garden"],"preset":"hotel"}` → output `data` fields `name`, `address`, `nightly_rate`, `check_in_time`, `check_out_time`, `amenities` (one row per page).
- **Menu** — input `{"urls":["https://your-restaurant.example/menu"],"preset":"menu"}` → output `data` fields `name`, `description`, `price`, `currency`, `category` (one row per menu item; charged once per page).
- **Recipe** — input `{"urls":["https://your-food-blog.example/banana-bread/"],"preset":"recipe"}` → output `data` fields `name`, `description`, `servings`, `prepTime`, `cookTime`, `totalTime` (ISO 8601, e.g. `PT1H15M`), `ingredients` (quantity, unit, item, raw), `steps`, `cuisine`, `category`, `calories`, `author`, `datePublished`, `imageUrl` (one row per page). A page's schema.org Recipe data is used first: with a name, ingredients and steps there is no AI call; other pages are read by AI. Same price per successful page; a page with no ingredients and no steps (not a recipe) is a free `no_data` row.
- **Company directory** — input `{"urls":["https://www.ycombinator.com/companies/location/boston"],"preset":"lead_directory"}` → output `data` fields `company_name`, `website`, `category`, `location`, `description` (one row per visible company; one charge per page with results). This is a public example directory, not a verified sales lead list.
- **Product catalog** — input `{"urls":["https://books.toscrape.com/"],"preset":"product_catalog"}` → output `data` fields `title`, `price`, `currency`, `availability`, `product_url` (one row per visible product; one charge per page with results). This is a public scraping sandbox.
- **Job posting** — input `{"urls":["https://ph.indeed.com/viewjob?jk=326367db14f11b31"],"preset":"job_detail","respectRobotsTxt":false}` → output `data` fields `title`, `employer`, `location`, `employmentType`, `salaryText`, `description`, `jobUrl` (one row per page). For public job detail pages you already have (e.g. Indeed `viewjob` links); it is not a job search. Only the main job is read: a company, salary or description shown only under similar or related jobs is not used. Short fields must be printed in the posting itself, so pay the posting does not show is `null`, never an estimate; `jobUrl` is the URL that was read. A page with no job title for its job is a free `no_data` row; a blocked page is a free row. Indeed's robots.txt disallows `/viewjob` for general readers, so the example explicitly turns off the default robots check. Use that setting only for pages you are permitted to read; access can still fail. With `respectRobotsTxt` on, those pages are free `robots_disallowed` rows; the Indeed Job Details Actor turns it off by default and reads the job URLs you give it.

For `product_catalog`, a price visibly attached to a product such as `1,280円`, `12,000원` or `₩8,000` stays in `data.price` as displayed; `data.currency` is `JPY` or `KRW`. A missing price, range or discounted amount is `null`. Currency and `product_url` must be supported by that product's own evidence; a `.jp` or `.kr` address alone does not set currency. Multiple products on one successful page still incur one `page-extracted` event; an empty or failed page is free. These are illustrative input strings, not verified live catalog listings.

You can also call the [REST API version on RapidAPI](https://rapidapi.com/tidytools/api/ai-web-data-extractor-api-any-page-to-structured-json).

### Extract a complete list across pages

Set `followPagination: true` to extract the starting list page and follow its same-domain **Next** or next page-number links. It is off by default. `maxPages` includes the starting page: default **5**, maximum **50**, per starting URL. **Each successful page creates one `page-extracted` event**, even when one page produces many item rows. Pagination stops when the page has no next link; it does not guess a URL from `?page=N` alone. Repeated URLs and identical page content also stop the chain; failed and empty pages are free. This follows list pages, whereas `followLinks` reads a start page for free and extracts its linked detail pages.

```json
{"urls":["https://books.toscrape.com/catalogue/page-1.html"],"preset":"product_catalog","fields":{},"followPagination":true,"maxPages":2}
```

This example can extract at most two catalog pages and charge at most two `page-extracted` events. A maximum run charge can further cap the run.

- **Short list, one successful page:** multiple item rows, just one `page-extracted` event for that page.
- **Very long list, only some items returned:** result rows show `itemsIncomplete: true` and `chunksFailed` (the failed-part count, which can be 0 if the page was cut off). A page with returned items still incurs one `page-extracted` event at the existing per-page price. Try a narrower `contentSelector`, or split the input into actual paginated URLs; `followPagination: true` can follow the site's real next-page links.
- **No matching items, failure, or no fields found:** a free error row with `charged: false`; no `page-extracted` event.

`followPagination` only follows actual next-page links on the site. It cannot guarantee recovery of items cut off within a single page, and content inside an iframe may not be visible.

#### One de-duplicated list across pages and directories (CSV)

The same clinic, shop or company often appears on several directory pages. Set `dedupeKeyFields` to the fields that identify one entity, e.g. `["name","address"]`, to also get `DEDUPED_ITEMS.csv` in the run's key-value store: each entity once, ready to import into a CRM without duplicates. It is off by default.

```json
{"urls":["https://your-directory.example/clinics?page=1","https://other-directory.example/clinics"],"fields":{"name":"string","address":"string","phone":"string"},"outputMode":"items","dedupeKeyFields":["name","address"]}
```

- **Matching is exact** on all key fields together, after trimming, ignoring upper/lower case and repeated spaces: `Acme Clinic, 1 Main St` and `acme  clinic, 1 main st` are one row. Same name with a different address stays two rows; no fuzzy matching.
- **A row missing any key field is never merged**: it stays in the CSV with `dedupeStatus: "missing_key_review"` and `missingKeyFields`, for you to check.
- **Columns**: your fields, the same with or without *Flat columns* (a nested field is named `parent_child`, e.g. `price_amount`; if two fields get the same name, such as a field `a_b` and a nested `a.b`, they become `a_b_1`, `a_b_2`…, skipping any name another field already has, and those are the names to use as key fields), then `sourceUrls` (every page the entity was found on, joined with `|`), `sourceCount`, `rowCount` (dataset rows combined), `dedupeStatus` (`unique`, `merged` or `missing_key_review`) and `missingKeyFields` (one of these names already used by your fields gets `dedupe_` in front, so every header is unique). The first row found is kept. Cells that start with `=`, `+`, `-` or `@` get a leading `'` so a spreadsheet does not run them as formulas.
- **The dataset keeps every row** of every page, with its source URL; failed pages are not in the CSV. `SUMMARY.dedupe` has `originalItems`, `dedupedItems`, `duplicatesRemoved`, `missingKeyItems` and `csvKey`. If Apify restarts the run, the CSV is rebuilt from all saved rows.
- **Price**: no extra AI calls and no extra events; each successful page is still one `page-extracted` event, however many of its rows were duplicates. If the CSV cannot be written, `SUMMARY.dedupe.error` says why; rows and charges are not changed. A key field that is not one of your fields is a free input error before any page is read (with fields designed from a prompt, it is reported in `SUMMARY.dedupe.error`).

### Extract JSON from PDF invoices, receipt photos and bank statements

Put a direct `.pdf`, `.jpg`, `.jpeg` or `.png` file URL in `urls` or a dataset URL field. A PDF text layer is converted to Markdown; pages without readable text use OCR. JPG and PNG files use OCR. The normal HTML page path is unchanged. Choose a preset and clear the sample `fields`, or provide your own `jsonSchema` for a different layout.

| Preset | `data` fields |
|---|---|
| `invoice` | `supplier`, `invoice_number`, `date`, `currency`, `subtotal`, `tax`, `total`, `line_items` (description, quantity, unit_price, amount) |
| `receipt` | `merchant`, `date`, `total`, `currency`, `items` (name, quantity, amount) |
| `bank_statement` | One Dataset row per transaction: `date`, `description`, `debit`, `credit`, signed `amount`, `balance`, `currency`, `opening_balance`, `closing_balance`, `transactionSum`, `calculatedClosingBalance`, `directionSource`, `balanceCheck`, `account_last_four`, `period`. With `flatOutput:false`, these fields are inside each row's `data`. |

```json
{"urls":["https://filedone.ai/sample-files/sample-invoice.pdf"],"preset":"invoice","fields":{},"maxPdfPages":20}
```

For a phone photo, use `https://filedone.ai/sample-files/sample-receipt-photo.jpg` with `preset: "receipt"`. For a two-page statement, use `https://filedone.ai/sample-files/sample-bank-statement.pdf` with `preset: "bank_statement"`. These are fictional CC0 test documents; their download pages provide answer keys.

Set `"lineItemRows": true` with `invoice` or `receipt` to emit one Dataset row per line item; the default is `false` and keeps the original one-row document with `line_items` or `items` in `data`. With `flatOutput: true`, the repeated header and `item_description`, `item_quantity`, `item_unit_price`, `item_amount` are top-level CSV/Excel columns; the item array is omitted. For receipts, `item_description` comes from `name` and `item_unit_price` is `null`. With flat output off, those fields are inside `data`. Every expanded row also has top-level `documentUrl` (final file URL) and zero-based `lineItemIndex`; an itemless document emits one header row with `lineItemIndex: null` and null item fields. `reviewWarnings` and `documentTruncated` are repeated on every row. Rows do not add charges: events still count readable pages.
**Bank balance review:** Adjacent printed balances set debit/credit direction and the sign of `amount` (`directionSource:balance`); otherwise the model direction is retained (`directionSource:model`). Each row has `balanceCheck:ok` when its printed balance matches the previous printed balance plus its signed extracted amount within one cent, `mismatch` when it does not, or `n/a` if either balance or amount is missing. This does not block output or add charges. A bank statement with no readable transactions returns a free `no_items` row. Export flat rows as CSV to open in Excel; no separate XLSX is generated.

**Invoice amount review:** for a complete PDF or image using `preset: "invoice"`, a success row may include `reviewWarnings: [{"code":"invoice_amount_mismatch","message":"..."}]` when the printed `subtotal` plus `tax` differs from the printed `total` by more than 0.01. The warning asks you to check for discounts, shipping or other fees. It does not change the extracted amounts, `success`, `charged` or `page-extracted` events. Missing amounts, unsupported source evidence and truncated documents produce no warning; no amount is inferred. This is a review cue, not a financial audit conclusion.

**Billing:** the existing `page-extracted` event and plan prices apply. Each PDF page with readable text that is sent to extraction incurs one event, up to `maxPdfPages` (default 20, maximum 500). A JPG or PNG incurs one event. Empty, unreadable, failed, or skipped pages are not charged. A result row shows `documentPagesRead`, `documentPageCount` and `documentTruncated`; a PDF beyond the page or model-content limit has only the pages actually sent to extraction counted. The run's maximum-cost setting still caps charges.

#### Optional batch quality report

Set `"qualityReport": true` to save `DATA_QUALITY` (JSON) and `DATA_QUALITY.csv` in this run's key-value store. The output links appear as **Data quality report** and **Field quality counts**. The option defaults to `false`; with it off, neither report is written. It makes no extra AI calls and does not change Dataset rows or `page-extracted` charges.

The JSON report contains `records` (successfully saved data rows, including each item in item mode), `missingByField` and `typeErrorsByField` (rows per top-level field; a row with several bad line items counts once under `line_items`), `reviewSuggested` (rows with a missing field, type error, review warning or field evidence that is not supported), and `averageCompleteness` across those saved data rows. `failedPages` counts saved free failure rows separately; those rows are **not** in the completeness or field-count denominator. The CSV lists one field per line with its missing and type-error counts. Formula-leading field names are escaped for spreadsheets. `SUMMARY` only adds `qualityReportKey`, `qualityReportCsvKey`, `qualityRecords`, `qualityReviewSuggested`, `qualityFailedPages`, and, if writing a report fails, `qualityReportError`. A report write error does not undo saved rows or charges. These are automated review hints, not a human check that the extracted values are true.

**Large batches in three steps:**

1. **Test a small sample first**: run 5–10 of your URLs and check the fields and the *Output* setting (`outputMode`: one row per page or one row per list item) before the full run.
2. **Run the batch**: paste the whole URL list, or set *Dataset ID* to a crawler's dataset ([chain it after a crawler](#chain-it-after-a-crawler)). Each page that returned data is one `page-extracted` charge, also a list page that was only partly read; set *Maximum cost per run* to cap the total.
3. **Re-run only the failed pages**: in a large batch some pages can fail (blocked, slow, timed out). Those rows are free and say `charged: false`; filter them in the export (or open the run's `RETRY_URLS` record, up to 1,000 URLs) and start a new run with just those URLs.

- 🧠 **Describe, don't code**: list field names and types, or just write a prompt such as *"Get every product with its name, price and rating"*
- 📋 **List pages → one row per item**: products, jobs, search results, team members; long lists are read in parts and successful pages cost one page, but a partial result may omit items (`itemsIncomplete: true`)
- If a long list initially returns no usable items, the Actor can retry once with smaller parts when time remains. Parts that still fail or time out remain free unless at least one real item is returned; a recovered page is charged once.
- 🔗 **Follow links to detail pages**: start from a category or search page and extract every linked product, job or story page that matches your pattern
- ✅ **Completeness check on every row (free)**: `completeness` (0–1) and `missingFields` tell you which values were not found; numbers, integers and yes/no values are converted to real types ("$1,299.00" → 1299, "In stock" → true)
- 🔒 **Stable columns from a prompt**: in prompt-only mode the AI designs the fields once, keeps them for the whole run and saves them for reuse
- 🌐 **Any website**: product pages, job posts, real estate listings, company "About" pages, articles, events, docs
- 🧾 **Honest output**: values that are not on the page come back as `null`; the AI is told never to invent data
- 🧩 **Nested data**: supply a full JSON Schema for arrays and objects
- 🛡️ **Pay only for success**: blocked pages (403, bot checks), errors, empty pages and pages where none of the fields were found are **free**
- ⚡ **Gets past more blocks**: plain HTTP from our servers, then from Apify's network (optionally through your proxy), then a real browser. A thin JavaScript shell can go straight to browser rendering; our Chrome pool is tried when Cloudflare browser capacity or its monthly usage limit is reached.

#### How it compares

| | This Actor | Apify's AI Web Scraper | Calling an LLM API yourself |
|---|---|---|---|
| Price per 1,000 pages (Free plan) | **$10** | $30 | Token cost, plus your own fetching, proxies and code |
| List pages | Charged **per page**, one row per item | Charged per page | Your code splits the list |
| Failed, blocked or empty pages | **Free** (`charged: false` on the row) | Not stated in its listing | You pay for the tokens used |
| Long lists | Read in parts: 250 of 250 rows in our test | Not stated | Limited by the model's context |
| Typed values (`"£51.77"` → 51.77) | Built in | Not stated | Your code |
| Per-row `completeness` and `missingFields` | Built in, free | Not stated | Your code |
| Finds pages from the sitemap | No: list the URLs or follow links from a list page | Yes (Scout and Agentic modes) | Your code |

Prices checked October 2026 on the Apify Store ($20 vs our $8 per 1,000 pages on Gold and higher plans).

### How much does it cost?

| Event | Price |
|---|---|
| Page extracted | **$10 per 1,000 pages** ($0.01 per page; $0.009 on Silver, $0.008 on Gold and higher Apify plans) |

**Example:** extracting 500 product pages costs 500 × $0.01 = **$5.00** on the Free plan, whether a page gives one item or 50.

**No start fee**, no extra charge for browser rendering, and item mode is still charged **per page**, however many items the page lists. Your maximum charge limit is always respected.

#### Control your cost

- **What is charged**: one `page-extracted` event per page (or text) that returned data. In item mode that is one charge per page, also when a long list was only partly read (`itemsIncomplete: true`).
- **What is free**: lines that are not a URL, duplicates of a URL already in the list, pages disallowed by robots.txt, blocked pages, errors, time-outs, pages with no matching items and pages where none of the fields were found. A disallowed start page, listed URL or redirect destination returns one `errorType: "robots_disallowed"` row with `charged: false`; no AI extraction is run for it. robots.txt is checked unless you turn off `respectRobotsTxt` (on by default). Every row has `charged: true/false`, and failed rows say "(not charged)".
- **Before the run starts**, the log and the status message show the plan: how many pages, the worst-case cost and your max charge per run. The same numbers are saved as `costPlan` in the `SUMMARY` record.
- **Max charge per run**: set *Maximum cost per run* in the run options (or `maxTotalChargeUsd` in the API). When it is reached, the run stops starting new pages and finishes cleanly: `SUMMARY.status` is `LIMIT_REACHED` and `SUMMARY.notProcessed` lists the pages that were not started (count and up to 100 inputs), so you can run just those next time.
- **Follow links** reads start pages for free and charges only the linked pages it extracts; *Max pages per start URL* caps them.
- **If Apify restarts the run** (server migration or Resurrect), items already finished are skipped and not charged again (`SUMMARY.resumedSkipped`).
- **Retry failed pages**: the run's private key-value store has `RETRY_URLS` with a JSON `urls` array of free failed page URLs in input order. It excludes successful pages and plain-text failures; with *Follow links*, it uses the failed detail page URL. Duplicates are removed and the list holds at most 1,000 URLs. `SUMMARY.failedUrlCount`, `SUMMARY.retryUrlsTruncated`, and `SUMMARY.retryUrlsKey` show the unique total, omitted count, and key name. Copy the URLs into a new run yourself with your desired fields and settings; no run starts automatically.

Compared with Apify's own AI Web Scraper, which costs $30 per 1,000 pages on the Free plan and $20 per 1,000 on Gold and higher plans (checked October 2026), this Actor is **2–3× cheaper** at every plan level.

### Switching from Apify's AI Web Scraper

Paste the input JSON you already use. `startUrls` and `prompt` are read as they are, so this runs unchanged:

```json
{
    "startUrls": [{ "url": "https://books.toscrape.com/" }],
    "prompt": "Every book with its title, price and whether it is in stock"
}
```

The same pages cost about a third: $10 instead of $30 per 1,000 pages on the Free plan. Good to know when you switch:

- **`prompt`** is the same as our *Instructions* (used when Instructions is empty). With no *Fields*, the AI designs the columns from your prompt once and keeps them for the whole run.
- **List pages**: add `"outputMode": "items"` to get one row per book, product or job. The page is still charged once.
- **Crawl options** (`extractionMode` Scout or Agentic, `maxPagesToVisit`, `maxCrawlDepth`) are ignored and the run log says so: only the start URLs are read. To extract the pages a list links to, add `"followLinks": true` with a `linkPattern`.
- **Typed columns**: for numbers and yes/no values, list *Fields*, e.g. `{"title": "string", "price": "number", "in_stock": "boolean"}`.
- **Failed pages are free**, and every row says `charged: true` or `false`.

### How to use it

When Workers AI returns no items or leaves every required field empty, paid Actor extraction makes one Claude Haiku retry through the backend. A recovered page is charged once under the existing page price. If Claude is unavailable, its daily limit is reached, or the requested JSON Schema has constraints Claude cannot accept, the original empty result remains a free failure row. The backend `/v1/extract` response identifies the model path with `engine`; Dataset row fields stay the same.

1. Paste your **URLs or domains** (one per line; `stripe.com` works too).
2. Choose a **Ready-made fields** preset, or describe what you want in one of three ways:
   - **Fields to extract**: `{"company_name": "string", "founded_year": "integer", "headquarters": "string"}`
   - **Instructions only** (leave Fields empty): *"Get every job with its title, location, salary and whether it is remote"*
   - **JSON Schema** for nested results
3. For listing pages, set **Output** to *One row per item*, or turn on **Follow links to detail pages** to extract each linked page.
4. Click **Start**. Export as JSON, CSV or Excel.

A line that is not a URL (a blank line, a company name, a typo) does not stop the run: it becomes one free row with `errorType: "invalid_input"` and the other lines are processed. If the whole input cannot be used (no valid URL, no fields or instructions, a JSON Schema that is not an object schema, an unknown field type), the run still ends normally with one free row that says what to fix (`errorType: "invalid_input"`, `SUMMARY.status: "INPUT_ERROR"`) instead of failing. The *URLs (request list)* field from earlier versions still works; for API calls and new integrations use `urls`, a plain list of strings.

#### Input example: one row per page

This is the example the input form opens with:

```json
{
    "urls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
    "fields": { "title": "string", "price": "number", "in_stock": "boolean" }
}
```

Result (values from a real run on this page, $0.01):

```json
{
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "success": true,
    "data": { "title": "A Light in the Attic", "price": 51.77, "in_stock": true },
    "completeness": 1,
    "charged": true
}
```

The page says "£51.77" and "In stock (22 available)"; the row has a number and `true`, ready for a spreadsheet.

#### Ready-made fields: property listing

For a public property page you may access, select `property` and clear the sample *Fields*. It defaults to one row per page. For an agency list page, add `"outputMode":"items"` to return one row per visible listing, still charged once per successful page; a detail page in item mode returns only its main property, not related-listing cards. The fields are `name`, `listingUrl`, `address`, `price`, `currency`, `bedrooms`, `bathrooms`, `area`, `areaUnit`, `area_sqm`, `status`, `agentName`, and `agentPhone`. Missing details are `null`. `name` and `address` belong to the property, never the agency or office. `listingUrl` requires a visible link; a general office phone is not an agent phone.

```json
{ "urls": ["https://your-agency.example/listings/123"], "preset": "property" }
```

Illustrative output row (replace the `.example` URL with your permitted site):

```json
{ "success": true, "data": { "name": "Oak Cottage", "address": "12 Oak Road", "price": 320000, "bedrooms": 3, "bathrooms": 2, "area_sqm": 95 }, "charged": true }
```

#### Ready-made fields: hotel page

For your hotel's own website, select `hotel` and clear the sample *Fields*. It uses one row per page with `name`, `address`, `nightly_rate`, `check_in_time`, `check_out_time`, and `amenities`. `name` is the hotel or room the page is about (not a section heading like "Rooms & Suites"); `amenities` lists facilities and services only, not room types, menu links or booking perks.

```json
{ "urls": ["https://your-hotel.example/rooms/garden"], "preset": "hotel" }
```

Illustrative output row:

```json
{ "success": true, "data": { "name": "Garden Room", "address": "8 Harbor Lane", "nightly_rate": 120, "check_in_time": "15:00", "check_out_time": "11:00", "amenities": ["Wi-Fi", "parking"] }, "charged": true }
```

#### Ready-made fields: restaurant menu

For your restaurant's own menu page, select `menu` and clear the sample *Fields*. It uses one row per menu item with `name`, `description`, `price`, `currency`, and `category`. The page is charged once, regardless of the number of items. `currency` is an ISO code such as `EUR` when the page's symbol names one currency (€, £, ₹…); a symbol shared by several currencies (`$`, `kr`, `Rs`, `¥`) stays as shown unless the page says which.

```json
{ "urls": ["https://your-restaurant.example/menu"], "preset": "menu" }
```

Illustrative output row:

```json
{ "success": true, "itemIndex": 0, "data": { "name": "Tomato Pasta", "description": "Pasta with tomato sauce", "price": 12, "currency": "USD", "category": "Mains" }, "charged": true }
```

#### Ready-made fields: company directory

Use `lead_directory` on a public company list, such as [Y Combinator's Boston directory](https://www.ycombinator.com/companies/location/boston). Clear the sample *Fields*. It returns one row per company shown on the page. `website`, `category`, `location` and `description` are `null` when absent; a company website is kept only when it appears literally in the page text sent to the AI, including on long lists read in parts. A directory entry is not a verified lead, and AI generated websites or categories must not be treated as verified facts. This preset reads the list page only, not each company profile.

```json
{ "urls": ["https://www.ycombinator.com/companies/location/boston"], "preset": "lead_directory" }
```

#### Ready-made fields: product catalog

Use `product_catalog` on a public product list, such as the [Books to Scrape sandbox](https://books.toscrape.com/). Clear the sample *Fields*. It returns one row per visible product, with `title`, `price`, `currency`, `availability` and `product_url`. By default, `price` is the displayed text, so currency symbols and wording are preserved; it is not a confirmed checkout price. Prices that are not numeric amounts (including `+7` color counts and `4 variants`) become `null`; bare numeric amounts remain valid. A price not found in the text sent to the AI also becomes `null`. `currency` uses JSON-LD only for the matching product, then a symbol or ISO code beside its price (`$` defaults to `USD`); page-wide OG currency is used only when the price does not show a conflicting currency. Otherwise it is `null`. A missing `product_url` is filled only when an exact-title link shares a short product fragment with price or stock details; navigation-only, ambiguous or unmatched links remain `null`. If you override `price` to `number`, the numeric value must occur in that text (for example, `12` matches `£12.00`); otherwise it becomes `null`. Other missing values are also `null`. For a single product detail page, use manual *Fields* with one row per page instead.

```json
{ "urls": ["https://books.toscrape.com/"], "preset": "product_catalog" }
```

**Taiwan prices.** A price shown as `NT$1,280` or `TWD 1280` stays exactly as displayed and gets `currency: "TWD"`. A plain `$1,280` still follows the `$` rule above (`USD`, or the page's own `og:price:currency` such as `TWD`); a `.tw` address alone never makes it TWD. Each product keeps only a price from its own row: if a product's row shows no price (for example `乙 | 售完`), the neighbouring product's price is not copied and `price` is `null`. Price ranges (`NT$1,280 - NT$1,580`) and discount wording (`NT$900 (8折)`) are not one price and become `null`. Large Taiwan marketplaces such as Shopee, Ruten or momo often block automated readers or load prices with JavaScript, so those pages may come back as free `blocked` rows or with `null` prices; try a few pages first. If you can copy the visible list text, `texts` skips fetching the site and is still one `page-extracted` event per text with products:

**Japanese and Korean prices.** A product row such as `抹茶 | 1,280円` returns `price: "1,280円"`, `currency: "JPY"`; `라면 | 12,000원` and `차 | ₩8,000` return their displayed prices with `currency: "KRW"`. The amount and currency must be supported by the same product's text, and a product link must belong to that product. A `.jp` or `.kr` address is not currency evidence. A range, discount, or missing price returns `null` for price; successful pages with multiple products still incur only one `page-extracted` event, and empty pages are free. These example strings are illustrative, not verified live listings.

```json
{ "texts": ["甲 | NT$1,280 | 有貨\n乙 | NT$900 | 售完"], "preset": "product_catalog" }
```

You can add *Fields* to any preset or override a preset field's type. A full *JSON Schema* still takes priority over both. The run's private `SUMMARY.preset` records only `property`, `hotel`, `menu`, `lead_directory` or `product_catalog` for measuring use; it does not record the URL or page content there. Without a preset, output and charging follow the existing settings. Both new list presets keep the same `page-extracted` price: at most one event per page with results, regardless of item count; empty and failed pages are free.

#### Input example: list page, one row per item

```json
{
    "urls": ["https://books.toscrape.com/"],
    "fields": { "title": "string", "price": "number", "rating": "integer", "in_stock": "boolean" },
    "outputMode": "items",
    "instructions": "rating is the number of stars (1-5)."
}
```

Real result: **20 rows for the 20 books on the page**, charged as one page:

```json
[
    { "url": "https://books.toscrape.com/", "itemIndex": 0, "itemCount": 20, "data": { "title": "A Light in the Attic", "price": 51.77, "rating": null, "in_stock": true } },
    { "url": "https://books.toscrape.com/", "itemIndex": 1, "itemCount": 20, "data": { "title": "Tipping the Velvet", "price": 53.74, "rating": null, "in_stock": true } }
]
```

`rating` is `null` because this site shows stars only as an image style, not as text: the AI does not guess values it cannot read.

#### Long lists: read in parts

A list page longer than about 12,000 characters is split into parts of about 5,000 characters (one paragraph repeated at each boundary, so an item cut in two is seen whole), up to 16 parts (about 80,000 characters), 4 at a time. The items are merged in page order, exact repeats are removed and an item split at a boundary is joined back together. The model is told to use only values written in each part; if an item cut off at the end of a part still comes back with different numbers than its full copy at the start of the next part, the full copy is kept. Placeholder words such as "None" or "N/A" are returned as `null`. Real result for https://www.scrapethissite.com/pages/simple/ (250 countries; fields country, capital, population, area; October 2026): **250 rows from 5 parts in 89 seconds, no repeats, charged as one page**.

If some parts fail or run out of time, the items from the other parts are still returned and the page is charged once, because you got data: the rows then carry `itemsIncomplete: true` and `chunksFailed`, and `SUMMARY.status` is `PARTIAL_RESULTS`. A page is charged only when at least one item came back.

When no item is returned, the free failure row includes `failureStage` to help you retry: `start_links` means a `followLinks` start page queued no links; the row tells you whether none matched *Link pattern* (check the pattern and page links) or matching links were excluded by robots.txt (check the site's rules for those pages). `list_items` means the list extraction replied but returned no usable items (check whether the page shows the requested items and whether *Fields* match them); `list_chunks` means every list extraction call failed (check the row's chunk error). The first two keep `errorType: "no_items"`; an all-chunk timeout keeps `errorType: "timeout"`. An empty model reply does not prove the website has no items. These rows have `charged: false`.

#### Input example: just a prompt

```json
{
    "urls": ["https://news.ycombinator.com/", "https://news.ycombinator.com/news?p=2"],
    "instructions": "Get every story with its title, points, number of comments and the site domain",
    "outputMode": "items"
}
```

Real result: the AI chose the fields `title`, `points`, `num_comments` and `site_domain` from the first page, used them for both pages (60 rows) and saved them in the key-value store as `SCHEMA`:

```json
{ "url": "https://news.ycombinator.com/news?p=2", "itemIndex": 3, "data": { "title": "MongoDB CEO resigns to join Meta", "points": 329, "num_comments": 270, "site_domain": "reuters.com" }, "schemaInferred": true }
```

To keep exactly the same columns next time, copy `SCHEMA.fields` into **Fields to extract**.

#### Input example: follow links from a list page to detail pages

```json
{
    "urls": ["https://books.toscrape.com/"],
    "followLinks": true,
    "linkPattern": "https://books.toscrape.com/catalogue/*/index.html",
    "maxPagesPerStartUrl": 5,
    "fields": { "title": "string", "price": "number", "in_stock": "boolean", "stock_count": "integer", "star_rating": "integer", "upc": "string", "isbn": "string" }
}
```

The start page is read for free and only its links are used; the 5 linked book pages are extracted and charged ($0.05). Real result (one of 5 rows):

```json
{
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "foundOn": "https://books.toscrape.com/",
    "success": true,
    "data": { "title": "A Light in the Attic", "price": 51.77, "in_stock": true, "stock_count": 22, "star_rating": null, "upc": "a897fe39b1053632", "isbn": null },
    "completeness": 0.71,
    "missingFields": ["star_rating", "isbn"]
}
```

`missingFields` shows at a glance that the star rating (drawn with CSS icons) and the ISBN (not on the page) were not found; the other 5 of 7 fields were. `linkPattern` takes globs (`*` within one path segment, `**` across segments) or plain text the link must contain, e.g. `item?id=` for Hacker News stories: with the instructions *"Story title, points, author, number of comments, and the top comment text"*, 3 story pages from the front page came back with all 5 fields (`completeness: 1`). Leave it empty to follow every link to the same site (menus included, so a pattern is better).

#### Advanced: JSON Schema

For nested results, use **JSON Schema** instead of *Fields* (in item mode it describes one item):

```json
{
    "type": "object",
    "properties": {
        "jobs": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": { "title": { "type": "string" }, "location": { "type": "string" }, "remote": { "type": "boolean" } }
            }
        }
    },
    "required": ["jobs"]
}
```

Before the run, the schema is made safe for the AI: every property you list is asked for (so none is silently left out) and may be `null`, so a value that is not on the page comes back as `null` instead of an invented one, also where your schema says `required`. The top level must be an object with `properties` (up to 50, schema up to 40,000 characters); anything else ends the run with a free `invalid_input` row before any page is read. Fields can also be given as a JSON text.

#### Flat columns for Google Sheets, n8n and CSV

Turn on **Flat columns** (`"flatOutput": true`) to get every field in its own column instead of inside `data`. The number of rows and the price stay the same. The books example above with `"flatOutput": true` gives rows like this (the first book, and a failed page):

```json
[
    { "input": "https://books.toscrape.com/", "inputIndex": 0, "url": "https://books.toscrape.com/", "finalUrl": "https://books.toscrape.com/", "pageTitle": "All products | Books to Scrape - Sandbox", "success": true, "recordIndex": 0, "recordCount": 20, "title": "A Light in the Attic", "price": 51.77, "rating": null, "in_stock": true, "completeness": 0.75, "missingFields": "rating", "typeErrors": null, "charged": true, "error": null, "errorType": null, "mode": "fast", "via": "backend", "contentTruncated": false, "chunksRead": 1, "chunksFailed": 0, "itemsIncomplete": false, "extractedAt": "2026-10-01T00:00:00.000Z" },
    { "input": "https://example.com/blocked", "inputIndex": 1, "url": "https://example.com/blocked", "finalUrl": null, "pageTitle": null, "success": false, "recordIndex": null, "recordCount": 0, "title": null, "price": null, "rating": null, "in_stock": null, "completeness": null, "missingFields": null, "typeErrors": null, "charged": false, "error": "blocked (not charged)", "errorType": "blocked", "mode": null, "via": null, "contentTruncated": null, "chunksRead": null, "chunksFailed": null, "itemsIncomplete": null, "extractedAt": "2026-10-01T00:00:00.000Z" }
]
```

- **Same columns, same order, in every row**, failed pages and invalid input lines included (empty values, `recordCount: 0`), so a Google Sheets "append row" step or an n8n node maps the columns once. The columns come from your fields, not from the values found. `failureStage` is a fixed page column beside `errorType`; it is `null` on successful rows and failures without a stage.
- **Nested objects** (JSON Schema) become columns joined with `_`: `{"price": {"amount": 49, "currency": "EUR"}}` → `price_amount`, `price_currency`.
- **Lists of values** become one text cell joined with `, ` (`["S", "M", "L"]` → `"S, M, L"`); an empty list is `null`. **Lists of objects** (e.g. a `jobs` list in one-row-per-page mode) become one JSON text cell; for one row per job, use *One row per item* instead.
- `recordIndex` / `recordCount` replace `itemIndex` / `itemCount` (one row per page: `0` / `1`). The page title is in `pageTitle`, so a field called `title` keeps its name. A field whose name is already a page column (e.g. `url`) gets `data_` in front (`data_url`). An extracted field named `failureStage` becomes `data_failureStage`, leaving the page diagnostic in `failureStage`. `missingFields` is a comma-separated text, `typeErrors` a JSON text.

### Writing good prompts and field names

| Instead of | Write |
|---|---|
| `version` | `newest_stable_version` |
| `price` (when a page shows several) | `sale_price` and `regular_price`, plus the instruction "numbers only, without currency" |
| "Get the data" | "Get every job with its title, city, salary range and whether it is remote" |
| "Find the contact" | "The company's general contact email and phone number from the page; null if not shown" |
| "Summarize the product" | `key_features` (array) and a separate `one_sentence_summary` (string) |

- **Name each field for exactly one fact**, and say the unit or format in Instructions ("dates as YYYY-MM-DD", "prices in USD as numbers").
- **Say which item to pick** when a page shows several ("only the first job", "the main product, not recommendations"), or use item mode to get all of them.
- **Ask only for what is on the page.** The AI returns `null` for missing values instead of guessing, so `null` means "not on the page".
- **Long pages**: set a *Content selector* (e.g. `main`) so the AI reads the relevant part. With one row per page about 32,000 characters are read; in item mode up to about 80,000 characters, in parts (`contentTruncated: true` when more was cut).

### Chain it after a crawler

Set **Dataset ID** to the dataset of another run (for example our Website to Markdown Crawler or a search results scraper) and **Dataset field** to the field with the URLs (empty = the first URL field found). The Actor then extracts data from every URL in that dataset. To use an AI Page Classifier dataset, set **Dataset labels** to 1–100 labels, for example `{"datasetId":"YOUR_DATASET_ID","datasetLabels":["product"],"fields":{"title":"string"}}`. Labels are trimmed and matched exactly with case sensitivity. Only rows with `success: true`, a matching `label` and a usable URL are extracted. Omit `datasetLabels` to keep the original behavior. An empty or invalid label list, or labels without a Dataset ID, gives a free input error. When nothing matches, `SUMMARY.status` is `NO_RESULTS` and `SUMMARY.dataset.matched` is 0; no `page-extracted` event occurs. With matches, `SUMMARY.dataset.matched` counts eligible source URLs before deduplication. Output `datasetIndex` is the original zero-based source row index; a pasted URL adopted before the same Dataset URL has no Dataset index. Each successful selected page has one existing `page-extracted` event; skipped and failed pages are free.

### Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/ai-web-data-extractor) to Claude, Cursor or any MCP client, then ask e.g. "Get the name, price and rating of every product on these three category pages".

```json
{ "urls": ["https://books.toscrape.com/"], "instructions": "Every book with title and price", "outputMode": "items" }
```

Failed pages are not charged and carry an `errorType` (`robots_disallowed` when the site's robots.txt excludes the page, `blocked` for 403s and bot checks such as a "Client Challenge" page, `login_required` for login walls, `not_found`, `timeout`, `no_items`, `no_data` when none of the fields were found…). The key-value store record `SUMMARY` has an overall `status` (`SUCCESS`, `PARTIAL_RESULTS`, `FAILED`, `NO_RESULTS`, `LIMIT_REACHED`, `INPUT_ERROR` when the input itself could not be used).

`SUMMARY.timing` shows where the time went: how many pages were read (`read`) and extracted by the AI (`ai`) with their average and longest time in seconds, how many pages ran out of time while reading and while extracting (`timeoutByStage`), and the `maxConcurrency` and `itemTimeoutSecs` the run used. If many pages time out, a higher `itemTimeoutSecs` (up to 900) gives each page more time.

Via API:

```bash
curl -X POST "https://api.apify.com/v2/acts/tidytools~ai-web-data-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com"],"fields":{"title":"string","summary":"string"}}'
```

**Schedules and integrations:** run it daily or weekly with Apify Schedules, get a webhook when a run finishes, or send the results to Zapier, Make, n8n, Google Sheets, Slack and other apps with Apify integrations. Results can be exported as JSON, CSV, Excel or XML. For Google Sheets, n8n and CSV, turn on *Flat columns* (`flatOutput`) to get one plain column per field.

### Output fields

| Field | Meaning |
|---|---|
| `input`, `inputIndex` | The line you entered and its position in your list (after invalid lines and duplicates were removed), so every row maps back to your input |
| `data` | The extracted values (one item in item mode) |
| `itemIndex`, `itemCount` | Item mode: position of the item on the page and the number of items found |
| `recordIndex`, `recordCount`, `pageTitle` | With *Flat columns* only: the extracted fields are top-level columns instead of `data`, `recordIndex`/`recordCount` replace `itemIndex`/`itemCount` (`recordCount: 0` on failed rows) and the page title is `pageTitle` (see [Flat columns](#flat-columns-for-google-sheets-n8n-and-csv)) |
| `title`, `finalUrl` | Page title and URL after redirects |
| `mode`, `via` | `fast` or `browser`; fetched by `backend` (our servers), `direct` (Apify's network) or `browser` |
| `contentTruncated` | `true` when the page was longer than the part the AI reads |
| `chunksRead`, `chunksFailed`, `itemsIncomplete` | Item mode: parts of the page read, parts that failed or ran out of time, and `true` when some items may be missing (failed parts, or a page longer than 16 parts) |
| `schemaInferred` | `true` when the fields were designed by the AI from your instructions |
| `reviewWarnings` | On a complete invoice document only: an optional non-blocking `invoice_amount_mismatch` review cue when supported `subtotal` + `tax` differs from supported `total` by more than 0.01. No amount is recalculated. |
| `documentUrl`, `lineItemIndex` | With `lineItemRows: true` for invoice/receipt: final document URL and zero-based item position (`null` when no items). |
| `item_description`, `item_quantity`, `item_unit_price`, `item_amount` | With `lineItemRows: true`: one item's fields, inside `data` normally or top-level with Flat columns. Receipt unit price is `null`. |
| `completeness`, `missingFields` | Share of fields (0–1) that have a value, and the fields that are empty (`null`, `""`, `[]`, or a placeholder word such as "None" or "N/A", which is returned as `null`). Computed by the Actor, free |
| `typeErrors` | Values that could not be converted to the field's type (kept as returned), e.g. `{"field": "rating", "expected": "integer", "value": "three"}`. Declared nested values are checked too (invoice `line_items`, receipt `items`, statement `transactions`, lists of objects in your own schema, up to 8 levels and 1,000 entries per list): a number written as text such as `"2"` becomes `2`; a value that is not one number, or would lose its sign or debit/credit direction (`"oops"`, `"(10.00)"`, `"10.00 DR"`, `"10.00 debit"`, `"Withdrawal 25"`), is kept as returned and listed with its path, e.g. `{"field": "line_items", "path": "line_items[0].amount", "expected": "number", "value": "oops"}` (up to 200 per row). A bad nested value does not change `completeness`, does not fail the page and does not change the charge |
| `startUrl` (also `foundOn`) | With *Follow links*: the start page the link was found on; `input` is that start URL as you entered it |
| `datasetIndex` | With *Dataset labels*: original source row index; otherwise the old value index |
| `charged` | `true` when this page was charged (in item mode every row of a charged page says `true`; the page is charged once) |
| `error`, `errorType` | For failed pages and invalid input lines (not charged): `invalid_input`, `blocked`, `login_required`, `not_found`, `timeout`, `no_items`, `no_data`… |
| `failureStage` | On free no-item list rows: `start_links`, `list_items`, or `list_chunks` (see *Long lists*). |

### Limitations

- AI extraction is very good but not perfect; verify critical values.
- Only public `http`/`https` pages; no logins. For sites that block data centers, add a proxy under *Advanced settings* (Apify proxy usage is billed to your Apify account).
- One row per page reads about 32,000 characters of each page; item mode reads up to about 80,000 characters in 16 parts. Longer lists come back with `itemsIncomplete: true`: list the paginated URLs instead.
- *Time limit per page* (Advanced settings, default 180 seconds) stops a page that takes too long; it comes back as a free `timeout` row. In item mode the items from the parts that finished in time are kept. Raise it for very long lists with a slower model.
- *AI model* (Advanced settings, same price): Auto uses Llama 3.3 70B for one row per page and Qwen3 30B for item mode. On the 250-country page both found all 250 countries, and Qwen3 30B wrote the list about 3 times faster. With Auto, a call that fails or stalls is repeated once on Llama 4 Scout (same price), and a part of a list page that is only a menu (category links, breadcrumbs) is dropped instead of being returned as items.
- Values that exist only in images or styling (e.g. star icons) cannot be read.
- *Follow links* goes one level deep (start page → linked pages); it does not click "next page" buttons. List the paginated URLs as start URLs instead.
- Type conversion only touches top-level fields; values of nested JSON Schema objects are returned as the AI gave them.
- *Flat columns* joins lists of values with `, `, so a value that itself contains a comma cannot be told apart in that cell; use the normal output when that matters. The *Extracted data* table view in the Output tab is made for the normal output; open *All fields* or export the dataset to see every flat column (integrations and the API always get every column).

### Is it legal?

You are responsible for how you use extracted data. Respect the target site's terms of service, copyright and privacy laws, and don't collect personal data without a legal basis.

### FAQ

**Why does my run say "partial" / why did some pages come back as `timeout`?** In a bulk run of hundreds or thousands of URLs, some pages can be slower than the *Time limit per page* (Advanced settings, default 180 seconds): a very long page, a slow site or a busy AI model. Such a page ends as a free `errorType: "timeout"` row (`charged: false`); the other pages are unaffected and the run still finishes normally (it is only marked partial because some rows failed). To get those pages: copy the `url` of the `charged: false` rows in the dataset, or the URLs from the run's `RETRY_URLS` key-value store record, into a new run, raise *Time limit per page* (up to 900), set *AI model* to another model (same price), or set a *Content selector* (e.g. `main`) so less text is read. The `error` text of each failed row says which step ran out of time.

**Are failed pages charged?** No. Blocked pages, login walls, unreachable sites, errors, time-outs and pages where none of the fields were found are free and say `charged: false`.

**Can it extract every item from a list page?** Item mode returns one row per extracted product, job or result and reads long pages in parts (up to about 80,000 characters), still for one page price when items are returned. Items may be missing if parts fail or the page is too long; check `itemsIncomplete: true` and `chunksFailed`. Use actual paginated URLs or a narrower `contentSelector` for long lists.

**What if my fields or JSON Schema are wrong?** The run does not fail: it ends normally with one free row (`errorType: "invalid_input"`) that says what to change, and nothing is charged. The same goes for a list with no valid URL.

**What happens when a value is not on the page?** It comes back as `null`; the AI is told never to invent data. Every row has `completeness` and `missingFields`, and pages where none of the fields were found are free. Yes/no fields are `null` (not `false`) when the page does not say.

**Do I need an OpenAI or other LLM API key?** No. The AI model runs on our side and is included in the price per page. Under *AI model* (Advanced settings) you can pick another model at the same price.

**How is it different from a regular web scraper?** You describe the fields instead of writing CSS selectors or XPath, so the same input works on many sites and keeps working when a site's layout changes.

**Does the Actor collect usage data?** The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

### Support

Open an issue in the **Issues** tab with the URL and your input.

Found this useful? A short review on the Store helps other users find it.

### Product catalog Q1 inputs (live run after integration)

Each readable page with product rows should create one `page-extracted` event; a blocked or empty page creates zero. These are validation inputs, not claims that this build has been run live.

```q1
{"actor":"ai-extractor","cases":[{"name":"decathlon_hiking_boots","input":{"urls":["https://www.decathlon.com/collections/hiking-boots"],"preset":"product_catalog"},"expectedEvents":{"page-extracted":1},"expect":"Visible product prices with $ infer USD unless matching page metadata gives another currency; one event for a successful page."},{"name":"allbirds_mens","input":{"urls":["https://www.allbirds.com/collections/mens"],"preset":"product_catalog"},"expectedEvents":{"page-extracted":1},"expect":"Color counts such as +7 are null prices; product URLs resolve only from matching links inside product fragments; one event for a successful page."},{"name":"books_sandbox","input":{"urls":["https://books.toscrape.com/"],"preset":"product_catalog"},"expectedEvents":{"page-extracted":1},"expect":"Visible pound prices infer GBP and matching links inside product fragments give product URLs; one event for a successful page."}]}
```

# Actor input Schema

## `urls` (type: `array`):

Paste a page URL here, then name the Fields you want or write Instructions. One URL per line; the book URL is a sample you can run now. Bare domains such as stripe.com also work. A line that is not a URL becomes a free error row; the other URLs continue. You can also use startUrls or a datasetId. Direct PDF, JPG and PNG file URLs also work.

## `startUrls` (type: `array`):

Same as "URLs or domains", in Apify's request list format (also accepts a link to a text file with URLs). Kept for existing integrations; new integrations should use "urls".

## `preset` (type: `string`):

Choose a preset, then clear sample Fields. Presets supply columns and row mode; your Fields override types. Directory and catalog return one row per visible item. Catalog keeps NT$1,280, 1,280円, 12,000원 and ₩8,000 with own-product TWD, JPY or KRW evidence. A .tw, .jp or .kr domain alone does not set currency; ranges, discounts and missing prices are null. Blocked marketplaces return free rows. Use pages you own or may access.

## `fields` (type: `object`):

Name what to extract: {"title":"string","price":"number","in_stock":"boolean"}. The sample matches the book URL above; edit both for your page. Missing values return null. Clear this sample when using a preset or when you want Instructions alone to choose fields. Types: string, number, integer, boolean, array.

## `instructions` (type: `string`):

Or write one sentence, such as "Get each product's name, price and rating". Clear the sample Fields first if you want the AI to choose columns from this sentence. With no Fields or JSON Schema, it chooses once from the first page, reuses the columns for this run and saves them as SCHEMA. You can also use this for hints such as "Prices in USD".

## `prompt` (type: `string`):

Same as "Instructions"; used only when Instructions is empty. Lets you paste the input JSON of Apify's AI Web Scraper (`startUrls` + `prompt`) and run it unchanged. Its crawl options (extractionMode scout/agentic, maxPagesToVisit, maxCrawlDepth) are ignored: only the start URLs are read, or turn on "Follow links to detail pages".

## `outputMode` (type: `string`):

"One row per page": one data object per URL. "One row per item": for listing pages (products, jobs, search results, team members), every item on the page becomes its own row with url and itemIndex. Still charged per page. Values: page = One row per page; items = One row per item on the page (lists).

## `flatOutput` (type: `boolean`):

Put every extracted field in its own column instead of inside "data". Nested objects become columns joined with \_ (price.amount → price_amount), lists of values become one text cell ("red, blue"), lists of objects become a JSON text cell. Every row has the same columns in the same order, failed pages included (empty values, recordCount 0). recordIndex is the item's position on the page and the page title is in pageTitle. Same number of rows and same price as without it.

## `lineItemRows` (type: `boolean`):

For invoice or receipt presets, emit one Dataset row per line item with repeated header fields. Documents without items still emit one header row. Works with Flat columns for CSV and Excel; page-extracted is charged per readable page, not per item.

## `qualityReport` (type: `boolean`):

Write DATA_QUALITY.json and DATA_QUALITY.csv with field missing counts, type error counts (a row with bad nested values, such as an invoice line item amount, counts once under its top-level field), review suggestions and completeness across saved data rows. Failed or free rows are counted separately. Quality hints are not human verification. No extra AI calls or page-extracted charges.

## `dedupeKeyFields` (type: `array`):

Optional: field names that identify one entity, e.g. name and address. At the end of the run, DEDUPED_ITEMS.csv in the key-value store lists each entity once, with every source URL it was found on. Values are compared exactly after trimming, ignoring case and repeated spaces; a row missing any key field is never merged and is marked missing_key_review. The dataset keeps every row; failed pages are left out. Up to 5 fields. No extra AI calls or page-extracted charges.

## `followLinks` (type: `boolean`):

Treat the start URLs as list pages (category, search results, index): read each one for free, then extract data from the pages it links to (e.g. every product page). The start page itself is not extracted. Each extracted page is charged as usual.

## `followPagination` (type: `boolean`):

Extract the starting list page, then follow visible same-domain Next or page-number links. Stops when there is no next link; it does not guess from ?page=N alone. Each successful page is one page-extracted charge, even when Output is one row per item. Default off. Use Max pages to cap charges per starting URL.

## `maxPages` (type: `integer`):

Maximum pages including the starting page for each URL when Follow list pages automatically is on. Default 5; maximum 50. Each successful page is one page-extracted charge.

## `linkPattern` (type: `string`):

Which links to follow. A glob such as https://books.toscrape.com/catalogue/*/index.html (* = any text within one path segment, \*\* = across segments), or plain text the link must contain, such as /product/. Separate several with commas. Empty = every link to the same site (menus included, so a pattern is recommended).

## `maxPagesPerStartUrl` (type: `integer`):

At most this many linked pages are extracted from each start page (in page order). Caps the cost.

## `jsonSchema` (type: `object`):

Optional full JSON Schema (type: object) for nested output. When set, it replaces "Fields". In item mode it describes one item.

## `datasetId` (type: `string`):

Read the URLs from another Actor run's dataset, e.g. a crawler or a search results scraper. Values that are not URLs are extracted from as text.

## `datasetLabels` (type: `array`):

Only extract successful rows from a classifier Dataset whose label exactly matches one of these values (case-sensitive). Requires Dataset ID; empty URLs are skipped. Omit this field to read every Dataset URL as before.

## `datasetField` (type: `string`):

Field with the URLs, e.g. "url" or "website" (dot paths such as "company.url" work). Empty = the first URL field found.

## `mode` (type: `string`):

"Auto" reads pages over plain HTTP and switches to a real browser when a site blocks it or needs JavaScript. Values: auto = Auto (recommended); fast = Fast (HTTP only); browser = Browser (JavaScript rendering).

## `cssSelector` (type: `string`):

Optional. Only read this part of each page, e.g. "main". Helps with very long pages.

## `maxConcurrency` (type: `integer`):

How many pages are processed at the same time.

## `itemTimeoutSecs` (type: `integer`):

A page that takes longer (reading + AI) is stopped and returned as a free error row with errorType "timeout". In list mode, the items from the parts that finished in time are kept. 0 = no limit.

## `aiModel` (type: `string`):

Same price for every model. Auto uses Llama 3.3 70B for one row per page and Qwen3 30B for list pages (one row per item), where it writes long lists about 3 times faster with the same accuracy in our tests.

## `httpVia` (type: `string`):

Some sites block requests from data centers. Auto retries from a second network before falling back to a real browser. If both are refused, Auto also tries our second server (Oracle, different IP) before a real browser.

## `proxyConfiguration` (type: `object`):

Only used for requests sent from Apify's network. Apify proxy usage is billed to your Apify account.

## `respectRobotsTxt` (type: `boolean`):

On (default): pages the site's robots.txt disallows for automated readers are not read and return a free robots_disallowed row. Off: the URLs you give are read even where robots.txt disallows automated readers, as when you open them yourself. Either way we never log in and never solve CAPTCHAs; blocked pages are free.

## `maxPdfPages` (type: `integer`):

Read up to this many pages per PDF. Only pages with text that are sent to extraction incur one page-extracted event each.

## `reviewEvidence` (type: `boolean`):

For review pages with Output mode = items: read review card star markers and JSON-LD so each rating is tied to its own review (rating is null when no reviewer-specific score is found). The extra HTML read can take longer; it does not add page-extracted events.

## Actor input object example

```json
{
  "urls": [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
  ],
  "fields": {
    "title": "string",
    "price": "number",
    "in_stock": "boolean"
  },
  "outputMode": "page",
  "flatOutput": false,
  "lineItemRows": false,
  "qualityReport": false,
  "followLinks": false,
  "followPagination": false,
  "maxPages": 5,
  "maxPagesPerStartUrl": 20,
  "mode": "auto",
  "maxConcurrency": 5,
  "itemTimeoutSecs": 180,
  "aiModel": "auto",
  "httpVia": "auto",
  "respectRobotsTxt": true,
  "maxPdfPages": 20,
  "reviewEvidence": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `full` (type: `string`):

No description

## `qualityJson` (type: `string`):

No description

## `qualityCsv` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
    ],
    "fields": {
        "title": "string",
        "price": "number",
        "in_stock": "boolean"
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidytools/ai-web-data-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
    "fields": {
        "title": "string",
        "price": "number",
        "in_stock": "boolean",
    },
}

# Run the Actor and wait for it to finish
run = client.actor("tidytools/ai-web-data-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
  ],
  "fields": {
    "title": "string",
    "price": "number",
    "in_stock": "boolean"
  }
}' |
apify call tidytools/ai-web-data-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidytools/ai-web-data-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/zuXZ7jgllbXStKb3f/builds/WubhJnMaN7MQXCFPq/openapi.json