AI Web Scraper - Extract Structured Data & URL to JSON

AI Web Scraper: extract structured data and turn a URL into JSON

What it does

Give it page URLs. Name the fields you want, or describe them in one sentence. An LLM reads each page and returns structured JSON in data. You do not need CSS selectors or code. Values that are not on the page come back as null.

Input	Example
URL	https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Fields	{"title":"string","price":"number","in_stock":"boolean"}
JSON output (data)	{"title":"A Light in the Attic","price":51.77,"in_stock":true}

The output above came from a real run on this page. It is not a live price check.

Price: $0.01 per page that returns data, on the Free plan. That is $10 per 1,000 pages. The price is lower on higher Apify plans (see below). Failed, blocked, timed-out and empty pages are free.

Input	Example
URL	https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Fields	{"title":"string","price":"number","in_stock":"boolean"}
JSON output (data)	{"title":"A Light in the Attic","price":51.77,"in_stock":true}

The data values above came from a real run on this Books to Scrape product page, documented in the 2026-10-28 README; they are not a live price check. On the Free plan, each page with data costs $0.01 and creates one page-extracted event. Failed, blocked, timed-out and empty pages are free. For a list page, choose One row per item instead. You can paste up to 10,000 URLs or use the URL field of another Actor's dataset.

With reviewEvidence: true on a public review page, reviewer names and review text are matched to individual star markup after HTML entity and whitespace normalization; ratings with no individual evidence or conflicting cards remain null. A card-header tail the page shows next to a reviewer's name (post count, Australian state) is removed from author when the page has a review card under the shorter name. Merchant replies the page marks as replies and text-less entries from the page's similar-shops list are not output as review rows, nor are other entries without review text on a page whose other rows have text; the page is still one page-extracted event. A page left with no review after that gets one free failure row (errorType: "no_items", failureStage: "list_items") and is not charged.

Ready-made fields: Skip writing field names: choose Property, Hotel, Menu, Recipe, Company directory, Product catalog or Job posting, then clear the form's sample Fields. Replace illustrative .example URLs with pages you own or may access:

• Property — input {"urls":["https://your-agency.example/listings/123"],"preset":"property"} → one row per page by default; add "outputMode":"items" for one row per visible listing on a list page (one event per successful page). Fields include name, listingUrl, address, price, currency, bedrooms, bathrooms, area, areaUnit, area_sqm, status, agentName, agentPhone.
• Hotel — input {"urls":["https://your-hotel.example/rooms/garden"],"preset":"hotel"} → output data fields name, address, nightly_rate, check_in_time, check_out_time, amenities (one row per page).
• Menu — input {"urls":["https://your-restaurant.example/menu"],"preset":"menu"} → output data fields name, description, price, currency, category (one row per menu item; charged once per page).
• Recipe — input {"urls":["https://your-food-blog.example/banana-bread/"],"preset":"recipe"} → output data fields name, description, servings, prepTime, cookTime, totalTime (ISO 8601, e.g. PT1H15M), ingredients (quantity, unit, item, raw), steps, cuisine, category, calories, author, datePublished, imageUrl (one row per page). A page's schema.org Recipe data is used first: with a name, ingredients and steps there is no AI call; other pages are read by AI. Same price per successful page; a page with no ingredients and no steps (not a recipe) is a free no_data row.
• Company directory — input {"urls":["https://www.ycombinator.com/companies/location/boston"],"preset":"lead_directory"} → output data fields company_name, website, category, location, description (one row per visible company; one charge per page with results). This is a public example directory, not a verified sales lead list.
• Product catalog — input {"urls":["https://books.toscrape.com/"],"preset":"product_catalog"} → output data fields title, price, currency, availability, product_url (one row per visible product; one charge per page with results). This is a public scraping sandbox.
• Job posting — input {"urls":["https://ph.indeed.com/viewjob?jk=326367db14f11b31"],"preset":"job_detail","respectRobotsTxt":false} → output data fields title, employer, location, employmentType, salaryText, description, jobUrl (one row per page). For public job detail pages you already have (e.g. Indeed viewjob links); it is not a job search. Only the main job is read: a company, salary or description shown only under similar or related jobs is not used. Short fields must be printed in the posting itself, so pay the posting does not show is null, never an estimate; jobUrl is the URL that was read. A page with no job title for its job is a free no_data row; a blocked page is a free row. Indeed's robots.txt disallows /viewjob for general readers, so the example explicitly turns off the default robots check. Use that setting only for pages you are permitted to read; access can still fail. With respectRobotsTxt on, those pages are free robots_disallowed rows; the Indeed Job Details Actor turns it off by default and reads the job URLs you give it.

For product_catalog, a price visibly attached to a product such as 1,280円, 12,000원 or ₩8,000 stays in data.price as displayed; data.currency is JPY or KRW. A missing price, range or discounted amount is null. Currency and product_url must be supported by that product's own evidence; a .jp or .kr address alone does not set currency. Multiple products on one successful page still incur one page-extracted event; an empty or failed page is free. These are illustrative input strings, not verified live catalog listings.

You can also call the REST API version on RapidAPI.

Extract a complete list across pages

Set followPagination: true to extract the starting list page and follow its same-domain Next or next page-number links. It is off by default. maxPages includes the starting page: default 5, maximum 50, per starting URL. Each successful page creates one page-extracted event, even when one page produces many item rows. Pagination stops when the page has no next link; it does not guess a URL from ?page=N alone. Repeated URLs and identical page content also stop the chain; failed and empty pages are free. This follows list pages, whereas followLinks reads a start page for free and extracts its linked detail pages.

{"urls":["https://books.toscrape.com/catalogue/page-1.html"],"preset":"product_catalog","fields":{},"followPagination":true,"maxPages":2}

This example can extract at most two catalog pages and charge at most two page-extracted events. A maximum run charge can further cap the run.

• Short list, one successful page: multiple item rows, just one page-extracted event for that page.
• Very long list, only some items returned: result rows show itemsIncomplete: true and chunksFailed (the failed-part count, which can be 0 if the page was cut off). A page with returned items still incurs one page-extracted event at the existing per-page price. Try a narrower contentSelector, or split the input into actual paginated URLs; followPagination: true can follow the site's real next-page links.
• No matching items, failure, or no fields found: a free error row with charged: false; no page-extracted event.

followPagination only follows actual next-page links on the site. It cannot guarantee recovery of items cut off within a single page, and content inside an iframe may not be visible.

One de-duplicated list across pages and directories (CSV)

The same clinic, shop or company often appears on several directory pages. Set dedupeKeyFields to the fields that identify one entity, e.g. ["name","address"], to also get DEDUPED_ITEMS.csv in the run's key-value store: each entity once, ready to import into a CRM without duplicates. It is off by default.

{"urls":["https://your-directory.example/clinics?page=1","https://other-directory.example/clinics"],"fields":{"name":"string","address":"string","phone":"string"},"outputMode":"items","dedupeKeyFields":["name","address"]}

• Matching is exact on all key fields together, after trimming, ignoring upper/lower case and repeated spaces: Acme Clinic, 1 Main St and acme clinic, 1 main st are one row. Same name with a different address stays two rows; no fuzzy matching.
• A row missing any key field is never merged: it stays in the CSV with dedupeStatus: "missing_key_review" and missingKeyFields, for you to check.
• Columns: your fields, the same with or without Flat columns (a nested field is named parent_child, e.g. price_amount; if two fields get the same name, such as a field a_b and a nested a.b, they become a_b_1, a_b_2…, skipping any name another field already has, and those are the names to use as key fields), then sourceUrls (every page the entity was found on, joined with |), sourceCount, rowCount (dataset rows combined), dedupeStatus (unique, merged or missing_key_review) and missingKeyFields (one of these names already used by your fields gets dedupe_ in front, so every header is unique). The first row found is kept. Cells that start with =, +, - or @ get a leading ' so a spreadsheet does not run them as formulas.
• The dataset keeps every row of every page, with its source URL; failed pages are not in the CSV. SUMMARY.dedupe has originalItems, dedupedItems, duplicatesRemoved, missingKeyItems and csvKey. If Apify restarts the run, the CSV is rebuilt from all saved rows.
• Price: no extra AI calls and no extra events; each successful page is still one page-extracted event, however many of its rows were duplicates. If the CSV cannot be written, SUMMARY.dedupe.error says why; rows and charges are not changed. A key field that is not one of your fields is a free input error before any page is read (with fields designed from a prompt, it is reported in SUMMARY.dedupe.error).

Extract JSON from PDF invoices, receipt photos and bank statements

Put a direct .pdf, .jpg, .jpeg or .png file URL in urls or a dataset URL field. A PDF text layer is converted to Markdown; pages without readable text use OCR. JPG and PNG files use OCR. The normal HTML page path is unchanged. Choose a preset and clear the sample fields, or provide your own jsonSchema for a different layout.

Preset	data fields
invoice	supplier, invoice_number, date, currency, subtotal, tax, total, line_items (description, quantity, unit_price, amount)
receipt	merchant, date, total, currency, items (name, quantity, amount)
bank_statement	One Dataset row per transaction: date, description, debit, credit, signed amount, balance, currency, opening_balance, closing_balance, transactionSum, calculatedClosingBalance, directionSource, balanceCheck, account_last_four, period. With flatOutput:false, these fields are inside each row's data.

{"urls":["https://filedone.ai/sample-files/sample-invoice.pdf"],"preset":"invoice","fields":{},"maxPdfPages":20}

For a phone photo, use https://filedone.ai/sample-files/sample-receipt-photo.jpg with preset: "receipt". For a two-page statement, use https://filedone.ai/sample-files/sample-bank-statement.pdf with preset: "bank_statement". These are fictional CC0 test documents; their download pages provide answer keys.

Set "lineItemRows": true with invoice or receipt to emit one Dataset row per line item; the default is false and keeps the original one-row document with line_items or items in data. With flatOutput: true, the repeated header and item_description, item_quantity, item_unit_price, item_amount are top-level CSV/Excel columns; the item array is omitted. For receipts, item_description comes from name and item_unit_price is null. With flat output off, those fields are inside data. Every expanded row also has top-level documentUrl (final file URL) and zero-based lineItemIndex; an itemless document emits one header row with lineItemIndex: null and null item fields. reviewWarnings and documentTruncated are repeated on every row. Rows do not add charges: events still count readable pages. Bank balance review: Adjacent printed balances set debit/credit direction and the sign of amount (directionSource:balance); otherwise the model direction is retained (directionSource:model). Each row has balanceCheck:ok when its printed balance matches the previous printed balance plus its signed extracted amount within one cent, mismatch when it does not, or n/a if either balance or amount is missing. This does not block output or add charges. A bank statement with no readable transactions returns a free no_items row. Export flat rows as CSV to open in Excel; no separate XLSX is generated.

Invoice amount review: for a complete PDF or image using preset: "invoice", a success row may include reviewWarnings: [{"code":"invoice_amount_mismatch","message":"..."}] when the printed subtotal plus tax differs from the printed total by more than 0.01. The warning asks you to check for discounts, shipping or other fees. It does not change the extracted amounts, success, charged or page-extracted events. Missing amounts, unsupported source evidence and truncated documents produce no warning; no amount is inferred. This is a review cue, not a financial audit conclusion.

Billing: the existing page-extracted event and plan prices apply. Each PDF page with readable text that is sent to extraction incurs one event, up to maxPdfPages (default 20, maximum 500). A JPG or PNG incurs one event. Empty, unreadable, failed, or skipped pages are not charged. A result row shows documentPagesRead, documentPageCount and documentTruncated; a PDF beyond the page or model-content limit has only the pages actually sent to extraction counted. The run's maximum-cost setting still caps charges.

Optional batch quality report

Set "qualityReport": true to save DATA_QUALITY (JSON) and DATA_QUALITY.csv in this run's key-value store. The output links appear as Data quality report and Field quality counts. The option defaults to false; with it off, neither report is written. It makes no extra AI calls and does not change Dataset rows or page-extracted charges.

The JSON report contains records (successfully saved data rows, including each item in item mode), missingByField and typeErrorsByField (rows per top-level field; a row with several bad line items counts once under line_items), reviewSuggested (rows with a missing field, type error, review warning or field evidence that is not supported), and averageCompleteness across those saved data rows. failedPages counts saved free failure rows separately; those rows are not in the completeness or field-count denominator. The CSV lists one field per line with its missing and type-error counts. Formula-leading field names are escaped for spreadsheets. SUMMARY only adds qualityReportKey, qualityReportCsvKey, qualityRecords, qualityReviewSuggested, qualityFailedPages, and, if writing a report fails, qualityReportError. A report write error does not undo saved rows or charges. These are automated review hints, not a human check that the extracted values are true.

Large batches in three steps:

1. Test a small sample first: run 5–10 of your URLs and check the fields and the Output setting (outputMode: one row per page or one row per list item) before the full run.
2. Run the batch: paste the whole URL list, or set Dataset ID to a crawler's dataset (chain it after a crawler). Each page that returned data is one page-extracted charge, also a list page that was only partly read; set Maximum cost per run to cap the total.
3. Re-run only the failed pages: in a large batch some pages can fail (blocked, slow, timed out). Those rows are free and say charged: false; filter them in the export (or open the run's RETRY_URLS record, up to 1,000 URLs) and start a new run with just those URLs.

• 🧠 Describe, don't code: list field names and types, or just write a prompt such as "Get every product with its name, price and rating"
• 📋 List pages → one row per item: products, jobs, search results, team members; long lists are read in parts and successful pages cost one page, but a partial result may omit items (itemsIncomplete: true)
• If a long list initially returns no usable items, the Actor can retry once with smaller parts when time remains. Parts that still fail or time out remain free unless at least one real item is returned; a recovered page is charged once.
• 🔗 Follow links to detail pages: start from a category or search page and extract every linked product, job or story page that matches your pattern
• ✅ Completeness check on every row (free): completeness (0–1) and missingFields tell you which values were not found; numbers, integers and yes/no values are converted to real types ("$1,299.00" → 1299, "In stock" → true)
• 🔒 Stable columns from a prompt: in prompt-only mode the AI designs the fields once, keeps them for the whole run and saves them for reuse
• 🌐 Any website: product pages, job posts, real estate listings, company "About" pages, articles, events, docs
• 🧾 Honest output: values that are not on the page come back as null; the AI is told never to invent data
• 🧩 Nested data: supply a full JSON Schema for arrays and objects
• 🛡️ Pay only for success: blocked pages (403, bot checks), errors, empty pages and pages where none of the fields were found are free
• ⚡ Gets past more blocks: plain HTTP from our servers, then from Apify's network (optionally through your proxy), then a real browser. A thin JavaScript shell can go straight to browser rendering; our Chrome pool is tried when Cloudflare browser capacity or its monthly usage limit is reached.

How it compares

	This Actor	Apify's AI Web Scraper	Calling an LLM API yourself
Price per 1,000 pages (Free plan)	$10	$30	Token cost, plus your own fetching, proxies and code
List pages	Charged per page, one row per item	Charged per page	Your code splits the list
Failed, blocked or empty pages	Free (charged: false on the row)	Not stated in its listing	You pay for the tokens used
Long lists	Read in parts: 250 of 250 rows in our test	Not stated	Limited by the model's context
Typed values ("£51.77" → 51.77)	Built in	Not stated	Your code
Per-row completeness and missingFields	Built in, free	Not stated	Your code
Finds pages from the sitemap	No: list the URLs or follow links from a list page	Yes (Scout and Agentic modes)	Your code

Prices checked October 2026 on the Apify Store ($20 vs our $8 per 1,000 pages on Gold and higher plans).

How much does it cost?

Event	Price
Page extracted	$10 per 1,000 pages ($0.01 per page; $0.009 on Silver, $0.008 on Gold and higher Apify plans)

Example: extracting 500 product pages costs 500 × $0.01 = $5.00 on the Free plan, whether a page gives one item or 50.

No start fee, no extra charge for browser rendering, and item mode is still charged per page, however many items the page lists. Your maximum charge limit is always respected.

Control your cost

• What is charged: one page-extracted event per page (or text) that returned data. In item mode that is one charge per page, also when a long list was only partly read (itemsIncomplete: true).
• What is free: lines that are not a URL, duplicates of a URL already in the list, pages disallowed by robots.txt, blocked pages, errors, time-outs, pages with no matching items and pages where none of the fields were found. A disallowed start page, listed URL or redirect destination returns one errorType: "robots_disallowed" row with charged: false; no AI extraction is run for it. robots.txt is checked unless you turn off respectRobotsTxt (on by default). Every row has charged: true/false, and failed rows say "(not charged)".
• Before the run starts, the log and the status message show the plan: how many pages, the worst-case cost and your max charge per run. The same numbers are saved as costPlan in the SUMMARY record.
• Max charge per run: set Maximum cost per run in the run options (or maxTotalChargeUsd in the API). When it is reached, the run stops starting new pages and finishes cleanly: SUMMARY.status is LIMIT_REACHED and SUMMARY.notProcessed lists the pages that were not started (count and up to 100 inputs), so you can run just those next time.
• Follow links reads start pages for free and charges only the linked pages it extracts; Max pages per start URL caps them.
• If Apify restarts the run (server migration or Resurrect), items already finished are skipped and not charged again (SUMMARY.resumedSkipped).
• Retry failed pages: the run's private key-value store has RETRY_URLS with a JSON urls array of free failed page URLs in input order. It excludes successful pages and plain-text failures; with Follow links, it uses the failed detail page URL. Duplicates are removed and the list holds at most 1,000 URLs. SUMMARY.failedUrlCount, SUMMARY.retryUrlsTruncated, and SUMMARY.retryUrlsKey show the unique total, omitted count, and key name. Copy the URLs into a new run yourself with your desired fields and settings; no run starts automatically.

Compared with Apify's own AI Web Scraper, which costs $30 per 1,000 pages on the Free plan and $20 per 1,000 on Gold and higher plans (checked October 2026), this Actor is 2–3× cheaper at every plan level.

Switching from Apify's AI Web Scraper

Paste the input JSON you already use. startUrls and prompt are read as they are, so this runs unchanged:

{
    "startUrls": [{ "url": "https://books.toscrape.com/" }],
    "prompt": "Every book with its title, price and whether it is in stock"
}

The same pages cost about a third: $10 instead of $30 per 1,000 pages on the Free plan. Good to know when you switch:

• prompt is the same as our Instructions (used when Instructions is empty). With no Fields, the AI designs the columns from your prompt once and keeps them for the whole run.
• List pages: add "outputMode": "items" to get one row per book, product or job. The page is still charged once.
• Crawl options (extractionMode Scout or Agentic, maxPagesToVisit, maxCrawlDepth) are ignored and the run log says so: only the start URLs are read. To extract the pages a list links to, add "followLinks": true with a linkPattern.
• Typed columns: for numbers and yes/no values, list Fields, e.g. {"title": "string", "price": "number", "in_stock": "boolean"}.
• Failed pages are free, and every row says charged: true or false.

How to use it

When Workers AI returns no items or leaves every required field empty, paid Actor extraction makes one Claude Haiku retry through the backend. A recovered page is charged once under the existing page price. If Claude is unavailable, its daily limit is reached, or the requested JSON Schema has constraints Claude cannot accept, the original empty result remains a free failure row. The backend /v1/extract response identifies the model path with engine; Dataset row fields stay the same.

1. Paste your URLs or domains (one per line; stripe.com works too).
2. Choose a Ready-made fields preset, or describe what you want in one of three ways:
   • Fields to extract: {"company_name": "string", "founded_year": "integer", "headquarters": "string"}
   • Instructions only (leave Fields empty): "Get every job with its title, location, salary and whether it is remote"
   • JSON Schema for nested results
3. For listing pages, set Output to One row per item, or turn on Follow links to detail pages to extract each linked page.
4. Click Start. Export as JSON, CSV or Excel.

A line that is not a URL (a blank line, a company name, a typo) does not stop the run: it becomes one free row with errorType: "invalid_input" and the other lines are processed. If the whole input cannot be used (no valid URL, no fields or instructions, a JSON Schema that is not an object schema, an unknown field type), the run still ends normally with one free row that says what to fix (errorType: "invalid_input", SUMMARY.status: "INPUT_ERROR") instead of failing. The URLs (request list) field from earlier versions still works; for API calls and new integrations use urls, a plain list of strings.

Input example: one row per page

This is the example the input form opens with:

{
    "urls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
    "fields": { "title": "string", "price": "number", "in_stock": "boolean" }
}

Result (values from a real run on this page, $0.01):

{
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "success": true,
    "data": { "title": "A Light in the Attic", "price": 51.77, "in_stock": true },
    "completeness": 1,
    "charged": true
}

The page says "£51.77" and "In stock (22 available)"; the row has a number and true, ready for a spreadsheet.

Ready-made fields: property listing

For a public property page you may access, select property and clear the sample Fields. It defaults to one row per page. For an agency list page, add "outputMode":"items" to return one row per visible listing, still charged once per successful page; a detail page in item mode returns only its main property, not related-listing cards. The fields are name, listingUrl, address, price, currency, bedrooms, bathrooms, area, areaUnit, area_sqm, status, agentName, and agentPhone. Missing details are null. name and address belong to the property, never the agency or office. listingUrl requires a visible link; a general office phone is not an agent phone.

{ "urls": ["https://your-agency.example/listings/123"], "preset": "property" }

Illustrative output row (replace the .example URL with your permitted site):

{ "success": true, "data": { "name": "Oak Cottage", "address": "12 Oak Road", "price": 320000, "bedrooms": 3, "bathrooms": 2, "area_sqm": 95 }, "charged": true }

Ready-made fields: hotel page

For your hotel's own website, select hotel and clear the sample Fields. It uses one row per page with name, address, nightly_rate, check_in_time, check_out_time, and amenities. name is the hotel or room the page is about (not a section heading like "Rooms & Suites"); amenities lists facilities and services only, not room types, menu links or booking perks.

{ "urls": ["https://your-hotel.example/rooms/garden"], "preset": "hotel" }

Illustrative output row:

{ "success": true, "data": { "name": "Garden Room", "address": "8 Harbor Lane", "nightly_rate": 120, "check_in_time": "15:00", "check_out_time": "11:00", "amenities": ["Wi-Fi", "parking"] }, "charged": true }

Ready-made fields: restaurant menu

For your restaurant's own menu page, select menu and clear the sample Fields. It uses one row per menu item with name, description, price, currency, and category. The page is charged once, regardless of the number of items. currency is an ISO code such as EUR when the page's symbol names one currency (€, £, ₹…); a symbol shared by several currencies ($, kr, Rs, ¥) stays as shown unless the page says which.

{ "urls": ["https://your-restaurant.example/menu"], "preset": "menu" }

Illustrative output row:

{ "success": true, "itemIndex": 0, "data": { "name": "Tomato Pasta", "description": "Pasta with tomato sauce", "price": 12, "currency": "USD", "category": "Mains" }, "charged": true }

Ready-made fields: company directory

Use lead_directory on a public company list, such as Y Combinator's Boston directory. Clear the sample Fields. It returns one row per company shown on the page. website, category, location and description are null when absent; a company website is kept only when it appears literally in the page text sent to the AI, including on long lists read in parts. A directory entry is not a verified lead, and AI generated websites or categories must not be treated as verified facts. This preset reads the list page only, not each company profile.

{ "urls": ["https://www.ycombinator.com/companies/location/boston"], "preset": "lead_directory" }

Ready-made fields: product catalog

Use product_catalog on a public product list, such as the Books to Scrape sandbox. Clear the sample Fields. It returns one row per visible product, with title, price, currency, availability and product_url. By default, price is the displayed text, so currency symbols and wording are preserved; it is not a confirmed checkout price. Prices that are not numeric amounts (including +7 color counts and 4 variants) become null; bare numeric amounts remain valid. A price not found in the text sent to the AI also becomes null. currency uses JSON-LD only for the matching product, then a symbol or ISO code beside its price ($ defaults to USD); page-wide OG currency is used only when the price does not show a conflicting currency. Otherwise it is null. A missing product_url is filled only when an exact-title link shares a short product fragment with price or stock details; navigation-only, ambiguous or unmatched links remain null. If you override price to number, the numeric value must occur in that text (for example, 12 matches £12.00); otherwise it becomes null. Other missing values are also null. For a single product detail page, use manual Fields with one row per page instead.

{ "urls": ["https://books.toscrape.com/"], "preset": "product_catalog" }

Taiwan prices. A price shown as NT$1,280 or TWD 1280 stays exactly as displayed and gets currency: "TWD". A plain $1,280 still follows the $ rule above (USD, or the page's own og:price:currency such as TWD); a .tw address alone never makes it TWD. Each product keeps only a price from its own row: if a product's row shows no price (for example 乙 | 售完), the neighbouring product's price is not copied and price is null. Price ranges (NT$1,280 - NT$1,580) and discount wording (NT$900 (8折)) are not one price and become null. Large Taiwan marketplaces such as Shopee, Ruten or momo often block automated readers or load prices with JavaScript, so those pages may come back as free blocked rows or with null prices; try a few pages first. If you can copy the visible list text, texts skips fetching the site and is still one page-extracted event per text with products:

Japanese and Korean prices. A product row such as 抹茶 | 1,280円 returns price: "1,280円", currency: "JPY"; 라면 | 12,000원 and 차 | ₩8,000 return their displayed prices with currency: "KRW". The amount and currency must be supported by the same product's text, and a product link must belong to that product. A .jp or .kr address is not currency evidence. A range, discount, or missing price returns null for price; successful pages with multiple products still incur only one page-extracted event, and empty pages are free. These example strings are illustrative, not verified live listings.

{ "texts": ["甲 | NT$1,280 | 有貨\n乙 | NT$900 | 售完"], "preset": "product_catalog" }

You can add Fields to any preset or override a preset field's type. A full JSON Schema still takes priority over both. The run's private SUMMARY.preset records only property, hotel, menu, lead_directory or product_catalog for measuring use; it does not record the URL or page content there. Without a preset, output and charging follow the existing settings. Both new list presets keep the same page-extracted price: at most one event per page with results, regardless of item count; empty and failed pages are free.

Input example: list page, one row per item

{
    "urls": ["https://books.toscrape.com/"],
    "fields": { "title": "string", "price": "number", "rating": "integer", "in_stock": "boolean" },
    "outputMode": "items",
    "instructions": "rating is the number of stars (1-5)."
}

Real result: 20 rows for the 20 books on the page, charged as one page:

[
    { "url": "https://books.toscrape.com/", "itemIndex": 0, "itemCount": 20, "data": { "title": "A Light in the Attic", "price": 51.77, "rating": null, "in_stock": true } },
    { "url": "https://books.toscrape.com/", "itemIndex": 1, "itemCount": 20, "data": { "title": "Tipping the Velvet", "price": 53.74, "rating": null, "in_stock": true } }
]

rating is null because this site shows stars only as an image style, not as text: the AI does not guess values it cannot read.

Long lists: read in parts

A list page longer than about 12,000 characters is split into parts of about 5,000 characters (one paragraph repeated at each boundary, so an item cut in two is seen whole), up to 16 parts (about 80,000 characters), 4 at a time. The items are merged in page order, exact repeats are removed and an item split at a boundary is joined back together. The model is told to use only values written in each part; if an item cut off at the end of a part still comes back with different numbers than its full copy at the start of the next part, the full copy is kept. Placeholder words such as "None" or "N/A" are returned as null. Real result for https://www.scrapethissite.com/pages/simple/ (250 countries; fields country, capital, population, area; October 2026): 250 rows from 5 parts in 89 seconds, no repeats, charged as one page.

If some parts fail or run out of time, the items from the other parts are still returned and the page is charged once, because you got data: the rows then carry itemsIncomplete: true and chunksFailed, and SUMMARY.status is PARTIAL_RESULTS. A page is charged only when at least one item came back.

When no item is returned, the free failure row includes failureStage to help you retry: start_links means a followLinks start page queued no links; the row tells you whether none matched Link pattern (check the pattern and page links) or matching links were excluded by robots.txt (check the site's rules for those pages). list_items means the list extraction replied but returned no usable items (check whether the page shows the requested items and whether Fields match them); list_chunks means every list extraction call failed (check the row's chunk error). The first two keep errorType: "no_items"; an all-chunk timeout keeps errorType: "timeout". An empty model reply does not prove the website has no items. These rows have charged: false.

Input example: just a prompt

{
    "urls": ["https://news.ycombinator.com/", "https://news.ycombinator.com/news?p=2"],
    "instructions": "Get every story with its title, points, number of comments and the site domain",
    "outputMode": "items"
}

Real result: the AI chose the fields title, points, num_comments and site_domain from the first page, used them for both pages (60 rows) and saved them in the key-value store as SCHEMA:

{ "url": "https://news.ycombinator.com/news?p=2", "itemIndex": 3, "data": { "title": "MongoDB CEO resigns to join Meta", "points": 329, "num_comments": 270, "site_domain": "reuters.com" }, "schemaInferred": true }

To keep exactly the same columns next time, copy SCHEMA.fields into Fields to extract.

Input example: follow links from a list page to detail pages

{
    "urls": ["https://books.toscrape.com/"],
    "followLinks": true,
    "linkPattern": "https://books.toscrape.com/catalogue/*/index.html",
    "maxPagesPerStartUrl": 5,
    "fields": { "title": "string", "price": "number", "in_stock": "boolean", "stock_count": "integer", "star_rating": "integer", "upc": "string", "isbn": "string" }
}

The start page is read for free and only its links are used; the 5 linked book pages are extracted and charged ($0.05). Real result (one of 5 rows):

{
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "foundOn": "https://books.toscrape.com/",
    "success": true,
    "data": { "title": "A Light in the Attic", "price": 51.77, "in_stock": true, "stock_count": 22, "star_rating": null, "upc": "a897fe39b1053632", "isbn": null },
    "completeness": 0.71,
    "missingFields": ["star_rating", "isbn"]
}

missingFields shows at a glance that the star rating (drawn with CSS icons) and the ISBN (not on the page) were not found; the other 5 of 7 fields were. linkPattern takes globs (* within one path segment, ** across segments) or plain text the link must contain, e.g. item?id= for Hacker News stories: with the instructions "Story title, points, author, number of comments, and the top comment text", 3 story pages from the front page came back with all 5 fields (completeness: 1). Leave it empty to follow every link to the same site (menus included, so a pattern is better).

Advanced: JSON Schema

For nested results, use JSON Schema instead of Fields (in item mode it describes one item):

{
    "type": "object",
    "properties": {
        "jobs": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": { "title": { "type": "string" }, "location": { "type": "string" }, "remote": { "type": "boolean" } }
            }
        }
    },
    "required": ["jobs"]
}

Before the run, the schema is made safe for the AI: every property you list is asked for (so none is silently left out) and may be null, so a value that is not on the page comes back as null instead of an invented one, also where your schema says required. The top level must be an object with properties (up to 50, schema up to 40,000 characters); anything else ends the run with a free invalid_input row before any page is read. Fields can also be given as a JSON text.

Flat columns for Google Sheets, n8n and CSV

Turn on Flat columns ("flatOutput": true) to get every field in its own column instead of inside data. The number of rows and the price stay the same. The books example above with "flatOutput": true gives rows like this (the first book, and a failed page):

[
    { "input": "https://books.toscrape.com/", "inputIndex": 0, "url": "https://books.toscrape.com/", "finalUrl": "https://books.toscrape.com/", "pageTitle": "All products | Books to Scrape - Sandbox", "success": true, "recordIndex": 0, "recordCount": 20, "title": "A Light in the Attic", "price": 51.77, "rating": null, "in_stock": true, "completeness": 0.75, "missingFields": "rating", "typeErrors": null, "charged": true, "error": null, "errorType": null, "mode": "fast", "via": "backend", "contentTruncated": false, "chunksRead": 1, "chunksFailed": 0, "itemsIncomplete": false, "extractedAt": "2026-10-01T00:00:00.000Z" },
    { "input": "https://example.com/blocked", "inputIndex": 1, "url": "https://example.com/blocked", "finalUrl": null, "pageTitle": null, "success": false, "recordIndex": null, "recordCount": 0, "title": null, "price": null, "rating": null, "in_stock": null, "completeness": null, "missingFields": null, "typeErrors": null, "charged": false, "error": "blocked (not charged)", "errorType": "blocked", "mode": null, "via": null, "contentTruncated": null, "chunksRead": null, "chunksFailed": null, "itemsIncomplete": null, "extractedAt": "2026-10-01T00:00:00.000Z" }
]

• Same columns, same order, in every row, failed pages and invalid input lines included (empty values, recordCount: 0), so a Google Sheets "append row" step or an n8n node maps the columns once. The columns come from your fields, not from the values found. failureStage is a fixed page column beside errorType; it is null on successful rows and failures without a stage.
• Nested objects (JSON Schema) become columns joined with _: {"price": {"amount": 49, "currency": "EUR"}} → price_amount, price_currency.
• Lists of values become one text cell joined with , (["S", "M", "L"] → "S, M, L"); an empty list is null. Lists of objects (e.g. a jobs list in one-row-per-page mode) become one JSON text cell; for one row per job, use One row per item instead.
• recordIndex / recordCount replace itemIndex / itemCount (one row per page: 0 / 1). The page title is in pageTitle, so a field called title keeps its name. A field whose name is already a page column (e.g. url) gets data_ in front (data_url). An extracted field named failureStage becomes data_failureStage, leaving the page diagnostic in failureStage. missingFields is a comma-separated text, typeErrors a JSON text.

Writing good prompts and field names

Instead of	Write
version	newest_stable_version
price (when a page shows several)	sale_price and regular_price, plus the instruction "numbers only, without currency"
"Get the data"	"Get every job with its title, city, salary range and whether it is remote"
"Find the contact"	"The company's general contact email and phone number from the page; null if not shown"
"Summarize the product"	key_features (array) and a separate one_sentence_summary (string)

• Name each field for exactly one fact, and say the unit or format in Instructions ("dates as YYYY-MM-DD", "prices in USD as numbers").
• Say which item to pick when a page shows several ("only the first job", "the main product, not recommendations"), or use item mode to get all of them.
• Ask only for what is on the page. The AI returns null for missing values instead of guessing, so null means "not on the page".
• Long pages: set a Content selector (e.g. main) so the AI reads the relevant part. With one row per page about 32,000 characters are read; in item mode up to about 80,000 characters, in parts (contentTruncated: true when more was cut).

Chain it after a crawler

Set Dataset ID to the dataset of another run (for example our Website to Markdown Crawler or a search results scraper) and Dataset field to the field with the URLs (empty = the first URL field found). The Actor then extracts data from every URL in that dataset. To use an AI Page Classifier dataset, set Dataset labels to 1–100 labels, for example {"datasetId":"YOUR_DATASET_ID","datasetLabels":["product"],"fields":{"title":"string"}}. Labels are trimmed and matched exactly with case sensitivity. Only rows with success: true, a matching label and a usable URL are extracted. Omit datasetLabels to keep the original behavior. An empty or invalid label list, or labels without a Dataset ID, gives a free input error. When nothing matches, SUMMARY.status is NO_RESULTS and SUMMARY.dataset.matched is 0; no page-extracted event occurs. With matches, SUMMARY.dataset.matched counts eligible source URLs before deduplication. Output datasetIndex is the original zero-based source row index; a pasted URL adopted before the same Dataset URL has no Dataset index. Each successful selected page has one existing page-extracted event; skipped and failed pages are free.

Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/ai-web-data-extractor) to Claude, Cursor or any MCP client, then ask e.g. "Get the name, price and rating of every product on these three category pages".

{ "urls": ["https://books.toscrape.com/"], "instructions": "Every book with title and price", "outputMode": "items" }

Failed pages are not charged and carry an errorType (robots_disallowed when the site's robots.txt excludes the page, blocked for 403s and bot checks such as a "Client Challenge" page, login_required for login walls, not_found, timeout, no_items, no_data when none of the fields were found…). The key-value store record SUMMARY has an overall status (SUCCESS, PARTIAL_RESULTS, FAILED, NO_RESULTS, LIMIT_REACHED, INPUT_ERROR when the input itself could not be used).

SUMMARY.timing shows where the time went: how many pages were read (read) and extracted by the AI (ai) with their average and longest time in seconds, how many pages ran out of time while reading and while extracting (timeoutByStage), and the maxConcurrency and itemTimeoutSecs the run used. If many pages time out, a higher itemTimeoutSecs (up to 900) gives each page more time.

Via API:

curl -X POST "https://api.apify.com/v2/acts/tidytools~ai-web-data-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com"],"fields":{"title":"string","summary":"string"}}'

Schedules and integrations: run it daily or weekly with Apify Schedules, get a webhook when a run finishes, or send the results to Zapier, Make, n8n, Google Sheets, Slack and other apps with Apify integrations. Results can be exported as JSON, CSV, Excel or XML. For Google Sheets, n8n and CSV, turn on Flat columns (flatOutput) to get one plain column per field.

Output fields

Field	Meaning
input, inputIndex	The line you entered and its position in your list (after invalid lines and duplicates were removed), so every row maps back to your input
data	The extracted values (one item in item mode)
itemIndex, itemCount	Item mode: position of the item on the page and the number of items found
recordIndex, recordCount, pageTitle	With Flat columns only: the extracted fields are top-level columns instead of data, recordIndex/recordCount replace itemIndex/itemCount (recordCount: 0 on failed rows) and the page title is pageTitle (see Flat columns)
title, finalUrl	Page title and URL after redirects
mode, via	fast or browser; fetched by backend (our servers), direct (Apify's network) or browser
contentTruncated	true when the page was longer than the part the AI reads
chunksRead, chunksFailed, itemsIncomplete	Item mode: parts of the page read, parts that failed or ran out of time, and true when some items may be missing (failed parts, or a page longer than 16 parts)
schemaInferred	true when the fields were designed by the AI from your instructions
reviewWarnings	On a complete invoice document only: an optional non-blocking invoice_amount_mismatch review cue when supported subtotal + tax differs from supported total by more than 0.01. No amount is recalculated.
documentUrl, lineItemIndex	With lineItemRows: true for invoice/receipt: final document URL and zero-based item position (null when no items).
item_description, item_quantity, item_unit_price, item_amount	With lineItemRows: true: one item's fields, inside data normally or top-level with Flat columns. Receipt unit price is null.
completeness, missingFields	Share of fields (0–1) that have a value, and the fields that are empty (null, "", [], or a placeholder word such as "None" or "N/A", which is returned as null). Computed by the Actor, free
typeErrors	Values that could not be converted to the field's type (kept as returned), e.g. {"field": "rating", "expected": "integer", "value": "three"}. Declared nested values are checked too (invoice line_items, receipt items, statement transactions, lists of objects in your own schema, up to 8 levels and 1,000 entries per list): a number written as text such as "2" becomes 2; a value that is not one number, or would lose its sign or debit/credit direction ("oops", "(10.00)", "10.00 DR", "10.00 debit", "Withdrawal 25"), is kept as returned and listed with its path, e.g. {"field": "line_items", "path": "line_items[0].amount", "expected": "number", "value": "oops"} (up to 200 per row). A bad nested value does not change completeness, does not fail the page and does not change the charge
startUrl (also foundOn)	With Follow links: the start page the link was found on; input is that start URL as you entered it
datasetIndex	With Dataset labels: original source row index; otherwise the old value index
charged	true when this page was charged (in item mode every row of a charged page says true; the page is charged once)
error, errorType	For failed pages and invalid input lines (not charged): invalid_input, blocked, login_required, not_found, timeout, no_items, no_data…
failureStage	On free no-item list rows: start_links, list_items, or list_chunks (see Long lists).

Limitations

• AI extraction is very good but not perfect; verify critical values.
• Only public http/https pages; no logins. For sites that block data centers, add a proxy under Advanced settings (Apify proxy usage is billed to your Apify account).
• One row per page reads about 32,000 characters of each page; item mode reads up to about 80,000 characters in 16 parts. Longer lists come back with itemsIncomplete: true: list the paginated URLs instead.
• Time limit per page (Advanced settings, default 180 seconds) stops a page that takes too long; it comes back as a free timeout row. In item mode the items from the parts that finished in time are kept. Raise it for very long lists with a slower model.
• AI model (Advanced settings, same price): Auto uses Llama 3.3 70B for one row per page and Qwen3 30B for item mode. On the 250-country page both found all 250 countries, and Qwen3 30B wrote the list about 3 times faster. With Auto, a call that fails or stalls is repeated once on Llama 4 Scout (same price), and a part of a list page that is only a menu (category links, breadcrumbs) is dropped instead of being returned as items.
• Values that exist only in images or styling (e.g. star icons) cannot be read.
• Follow links goes one level deep (start page → linked pages); it does not click "next page" buttons. List the paginated URLs as start URLs instead.
• Type conversion only touches top-level fields; values of nested JSON Schema objects are returned as the AI gave them.
• Flat columns joins lists of values with , , so a value that itself contains a comma cannot be told apart in that cell; use the normal output when that matters. The Extracted data table view in the Output tab is made for the normal output; open All fields or export the dataset to see every flat column (integrations and the API always get every column).

Is it legal?

You are responsible for how you use extracted data. Respect the target site's terms of service, copyright and privacy laws, and don't collect personal data without a legal basis.

FAQ

Why does my run say "partial" / why did some pages come back as timeout? In a bulk run of hundreds or thousands of URLs, some pages can be slower than the Time limit per page (Advanced settings, default 180 seconds): a very long page, a slow site or a busy AI model. Such a page ends as a free errorType: "timeout" row (charged: false); the other pages are unaffected and the run still finishes normally (it is only marked partial because some rows failed). To get those pages: copy the url of the charged: false rows in the dataset, or the URLs from the run's RETRY_URLS key-value store record, into a new run, raise Time limit per page (up to 900), set AI model to another model (same price), or set a Content selector (e.g. main) so less text is read. The error text of each failed row says which step ran out of time.

Are failed pages charged? No. Blocked pages, login walls, unreachable sites, errors, time-outs and pages where none of the fields were found are free and say charged: false.

Can it extract every item from a list page? Item mode returns one row per extracted product, job or result and reads long pages in parts (up to about 80,000 characters), still for one page price when items are returned. Items may be missing if parts fail or the page is too long; check itemsIncomplete: true and chunksFailed. Use actual paginated URLs or a narrower contentSelector for long lists.

What if my fields or JSON Schema are wrong? The run does not fail: it ends normally with one free row (errorType: "invalid_input") that says what to change, and nothing is charged. The same goes for a list with no valid URL.

What happens when a value is not on the page? It comes back as null; the AI is told never to invent data. Every row has completeness and missingFields, and pages where none of the fields were found are free. Yes/no fields are null (not false) when the page does not say.

Do I need an OpenAI or other LLM API key? No. The AI model runs on our side and is included in the price per page. Under AI model (Advanced settings) you can pick another model at the same price.

How is it different from a regular web scraper? You describe the fields instead of writing CSS selectors or XPath, so the same input works on many sites and keeps working when a site's layout changes.

Does the Actor collect usage data? The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

Support

Open an issue in the Issues tab with the URL and your input.

Found this useful? A short review on the Store helps other users find it.

Product catalog Q1 inputs (live run after integration)

Each readable page with product rows should create one page-extracted event; a blocked or empty page creates zero. These are validation inputs, not claims that this build has been run live.

{"actor":"ai-extractor","cases":[{"name":"decathlon_hiking_boots","input":{"urls":["https://www.decathlon.com/collections/hiking-boots"],"preset":"product_catalog"},"expectedEvents":{"page-extracted":1},"expect":"Visible product prices with $ infer USD unless matching page metadata gives another currency; one event for a successful page."},{"name":"allbirds_mens","input":{"urls":["https://www.allbirds.com/collections/mens"],"preset":"product_catalog"},"expectedEvents":{"page-extracted":1},"expect":"Color counts such as +7 are null prices; product URLs resolve only from matching links inside product fragments; one event for a successful page."},{"name":"books_sandbox","input":{"urls":["https://books.toscrape.com/"],"preset":"product_catalog"},"expectedEvents":{"page-extracted":1},"expect":"Visible pound prices infer GBP and matching links inside product fragments give product URLs; one event for a successful page."}]}