# Page Scraping Analyzer (`apify/page-analyzer`) Actor

Performs analysis of a webpage to figure out the best way how to scrape its data. Provide a URL and data points to find and get back a detailed dashboard showing how the data can be scraped. Works with initial and rendered HTML, JavaScript variables and dynamically loaded data.

- **URL**: https://apify.com/apify/page-analyzer.md
- **Developed by:** [Apify](https://apify.com/apify) (Apify)
- **Categories:** Automation, Developer tools
- **Features:** Full permissions
- **Stats:** 1,334 total users, 1 monthly users, 91.4% runs succeeded, 21 bookmarks
- **User rating**: 4.67 out of 5 stars

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Page Scraping Analyzer

Page Scraping Analyzer is an actor that helps its users find data sources on a website. Its main purpose is to help a user quickly analyze their options for extracting data from a website and provide CSS selectors, JavaScript code and HTTP requests that can be used to extract the data.

### When to use Page Scraping Analyzer

Page Scraping Analyzer can be used as a first step in a web scraper developement. Its goal is to automate the process of analyzing a website manually using tools like browsers developer tools or Postman to:

1. Analyze the structure of the website
2. Find the CSS selectors of HTML elements containing a keyword
3. Find keywords in additional sources that might not be visible on the screen like JSON+LD, metadata, schema.org data
4. Observe and replicate XHR requests that might contain the data a user wants to scrape

### Where is data stored on a website?

There are many sources of data on a website, some are not even visible on the screen. The same data point can be present in more than one source.

Here are some examples of where data can be stored on a website:

- Initial HTML response (can be scraped by HTTP-only scrapers like Cheerio)
  - HTML elements rendered on the server
  - Rich JSON data inside `<script>` tags (JSON+LD, schema.org, Next.js data)
- Rendered HTML (can be only scraped with a browser)
  - HTML elements rendered on the client
  - JavaScript variables available on the `window` object - data for can come from either:
    - Initial HTML response - Can be parsed from the `script` tags with HTTP only
    - XHR responses - Loaded later after the initial HTML response
- XHR responses (can be scraped with HTTP-only scrapers like Cheerio)
  - Usually comes as JSON data loaded from an internal API. Common formats are:
    - REST API
    - GraphQL API
    - WebSocket connections
  - Can be in any other format like HTML snippets

### How Page Scraping Analyzer works

The Page Scraping Analyzer works in multiple steps looking for data sources. For every step, it stores the sources and provides a CSS selector, JavaScript code or an HTTP request that can be used to extract the data.

It uses both browser and HTTP to provide all options to scrape the available data.

**With browser:**

1. Open the page and records the initial HTML response. Finds all HTML elements and `<script>` tags containing the keywords.
2. Waits for the page to render. Finds all HTML elements and JavaScript variables containing the keywords. Stores a diff between the initial HTML response and the rendered HTML.
3. Waits for the page to load all XHR requests. Finds all XHR responses containing the keywords.

**With HTTP:**

1. The same as step 1 with the browser - It is useful to know that you can scrape the same data with HTTP-only scrapers like Cheerio because it is much faster and cheaper.
2. Tries to replicate the XHR requests recorded by the browser to see if they can be scraped only with HTTP:
   - First tries to use only generic HTTP headers
   - If it fails, it tries to use the headers recorded by the browser without cookies
   - If it fails, it tries to use the headers recorded by the browser with cookies (this can still be automated but requires to get cookies from the browser and then use them for X HTTP requests)

### What scraping methods to choose after analysis

Some websites will require to combine multiple sources of data. Some are faster & cheaper to use, some are in nicer formats. Generally, it is best to try them in this order:

1. XHR requests with HTTP - extremely fast and cheap and usually in a nice format like JSON. Might require combining multiple requests to get all the data. Might require complex headers and body to be replicated.
2. `<script>` tags from the initial HTML response - often contains all the data in a nice JSON format. Requires parsing the JSON out of the script text
3. HTML elements from the initial HTML response - requires using multiple CSS selectors to get all the data.
4. JavaScript variables from the rendered HTML - usually contains all data in nice JavaScript objects.
5. HTML elements from the rendered HTML - requires using multiple CSS selectors to get all the data.
6. Intercepting XHR requests with a browser - requires waiting and sometimes interaction with the page. Might require combining multiple requests to get all the data.
7. HTML elements rendered after all XHR responses were processed - requires long waiting and sometimes interaction with the page. Requires using multiple CSS selectors to get all the data.

# Actor input Schema

## `url` (type: `string`):

URL of a website to by analyzed. Use a page URL that you would want to scrape. You can run this actor again for different page type (e.g. one run for search results page, another run for product detail page).

## `keywords` (type: `array`):

Keywords to be searched for during analysis

## `proxyConfig` (type: `object`):

You should use proxies for scraping. Use datacenter proxies but fallback to residential ones if blocked too much.

## Actor input object example

```json
{
  "url": "http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "keywords": [
    "A Light in the Attic",
    "51.77",
    "In stock",
    "22 available",
    "a897fe39b1053632",
    "It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love th It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love that Silverstein. Need proof of his genius? RockabyeRockabye baby, in the treetopDon't you know a treetopIs no safe place to rock?And who put you up there,And your cradle, too?Baby, I think someone down here'sGot it in for you. Shel, you never sounded so good. ...more"
  ],
  "proxyConfig": {
    "useApifyProxy": true
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "keywords": [
        "A Light in the Attic",
        "51.77",
        "In stock",
        "22 available",
        "a897fe39b1053632",
        "It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love th It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love that Silverstein. Need proof of his genius? RockabyeRockabye baby, in the treetopDon't you know a treetopIs no safe place to rock?And who put you up there,And your cradle, too?Baby, I think someone down here'sGot it in for you. Shel, you never sounded so good. ...more"
    ],
    "proxyConfig": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("apify/page-analyzer").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "url": "http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "keywords": [
        "A Light in the Attic",
        "51.77",
        "In stock",
        "22 available",
        "a897fe39b1053632",
        "It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love th It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love that Silverstein. Need proof of his genius? RockabyeRockabye baby, in the treetopDon't you know a treetopIs no safe place to rock?And who put you up there,And your cradle, too?Baby, I think someone down here'sGot it in for you. Shel, you never sounded so good. ...more",
    ],
    "proxyConfig": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("apify/page-analyzer").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "keywords": [
    "A Light in the Attic",
    "51.77",
    "In stock",
    "22 available",
    "a897fe39b1053632",
    "It'\''s hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein'\''s humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love th It'\''s hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein'\''s humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love that Silverstein. Need proof of his genius? RockabyeRockabye baby, in the treetopDon'\''t you know a treetopIs no safe place to rock?And who put you up there,And your cradle, too?Baby, I think someone down here'\''sGot it in for you. Shel, you never sounded so good. ...more"
  ],
  "proxyConfig": {
    "useApifyProxy": true
  }
}' |
apify call apify/page-analyzer --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,apify/page-analyzer"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/mJuwidcY8PejY3QKp/builds/KEWgs2hPsnc3CZ3Fe/openapi.json