# Algolia Search Index Scraper – Hits, Facets & Filters (`rl1987/algolia-api-scraper`) Actor

Scrape any site's Algolia search index straight from the Algolia Search REST API — no browser, no HTML parsing. Give it the site's app ID, search-only key and index name to pull hits with every attribute the index stores, or discover every facetable attribute and its value counts first.

- **URL**: https://apify.com/rl1987/algolia-api-scraper.md
- **Developed by:** [R.L.](https://apify.com/rl1987) (community)
- **Categories:** Other
- **Features:** Limited permissions
- **Stats:** 4 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.42 / 1,000 records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Algolia Search Index Scraper — Hits, Facets & Filters

A **generalised [Algolia](https://www.algolia.com/) search-index scraper** for extracting search results and discovering facets from any website that uses Algolia as its search backend — e-commerce catalogs, documentation sites, marketplaces, job boards, directories. Instead of scraping rendered HTML, this Actor talks directly to the site's own Algolia index over the [Algolia Search REST API](https://www.algolia.com/doc/rest-api/search/), so extraction is fast, reliable, and returns the exact structured data the site itself displays. Run it on the [Apify platform](https://apify.com) for scheduling, API access, and monitoring without managing any infrastructure.

### What does this Algolia scraper do?

It runs one of two modes against an index you point it at:

- **Scrape hits** — pages through the index and pushes every record to the dataset, exactly as Algolia returns it, with whatever attributes that index happens to store. You can narrow it with the same query, `filters`, `facetFilters` and `numericFilters` the site's own search UI uses.
- **Discover facets** — a single request that returns every facetable attribute on the index and its values with counts, plus the total `nbHits`. Use it first on an unfamiliar index to find out what you can filter by, then build a targeted scrape.

It needs three things, all of them visible in the site's own browser network traffic: an
**Application ID**, a **search-only API key**, and an **index name**. It ships with Hacker News
Search's public Algolia credentials pre-filled, so you can press **Start** and see real output
before you go looking for your own.

### Why scrape a site's Algolia index instead of its pages?

Thousands of sites (documentation portals, online stores, job boards, real-estate listings) use
Algolia under the hood for instant search. Querying that index directly means you can:

- Read the site's structured records — the same JSON its own front end renders — rather than parsing markup that changes whenever the design does.
- Apply the exact filters, facets and query syntax the site's search UI exposes.
- Find out which attributes are filterable/facetable, and what values exist for each, before committing to a larger scraping or monitoring pipeline.
- Feed clean JSON straight into spreadsheets, databases or downstream automations via Apify's integrations.

### How do I find a site's Algolia credentials?

1. Open the site and open DevTools → **Network**, filtering for `algolia.net` or `algolianet.com`.
2. Type something into the site's search box. You'll see `POST .../1/indexes/<indexName>/query` requests.
3. Read off the three values:
   - **Application ID** — the request host prefix, e.g. `ABC123DEF-dsn.algolia.net`, or the `X-Algolia-Application-Id` header.
   - **API key** — the `X-Algolia-API-Key` header. This is a *search-only* key; sites ship it to the browser on purpose.
   - **Index name** — the path segment after `/1/indexes/`.
4. Paste them into the Actor's input, add a query or filters if you want, and click **Start**. Or enable **Discover facets only** for a fast look at the index first.

### What can I try it on right away?

Two public indexes that are documented as such and work with no setup:

| Index | Application ID | API key | Index name | Facets it exposes |
| --- | --- | --- | --- | --- |
| **Hacker News Search** (the Actor's default input) | `UJ5WYC0L7X` | `28f0e1ec37a5e792e6845e67da5f20dd` | `Item_dev` | `_tags` (`story`, `comment`, `ask_hn`, `show_hn`, …), `created_at_i` |
| **Algolia's own `instant_search` demo catalog** | `latency` | `6be0576ff61c053d5f9a3225e2a90f76` | `instant_search` | `brand`, `categories`, `type`, `price`, `price_range`, `rating`, `free_shipping`, `hierarchicalCategories.lvl0`–`lvl4` |

For your own target, the credentials come out of the site's network traffic as described above.

### What do Algolia's terms mean?

- **Application ID** — identifies the Algolia account. It also names the hosts: `<appId>-dsn.algolia.net` plus the `-1`/`-2`/`-3.algolianet.com` cluster fallbacks this Actor fails over to.
- **Search-only API key** — a key that can query indexes but not write to them. It is the key a site's front end carries, and the only kind this Actor needs.
- **`objectID`** — Algolia's unique identifier for a record within an index. The one field every hit is guaranteed to have.
- **Facet** — an attribute the index is configured to aggregate on, so search results can be counted and filtered by its values (`brand`, `price_range`, `_tags`). Requesting `facets: ["*"]` asks for all of them.
- **`filters` vs `facetFilters` vs `numericFilters`** — `filters` is one SQL-ish expression string (`category:Shoes AND price < 50`). `facetFilters` is an array where a nested array means OR (`["category:Shoes", ["color:Red", "color:Blue"]]` = Shoes AND (Red OR Blue)). `numericFilters` is an array of numeric comparisons (`["price>=10", "price<=100"]`).
- **`nbHits` vs `nbPages`** — `nbHits` is how many records match. `nbPages` is how many pages you are actually allowed to fetch, and the two often disagree badly (see the limit below).
- **`paginationLimitedTo`** — an index-side setting, **default 1,000 records**, that caps how deep anyone can page. Both public indexes above show it: Hacker News reports 46.2 million matching records but only one page at `hitsPerPage: 1000`, and Algolia's demo catalog reports 21,468 matches but 1,000 pages of one hit. This is the single biggest thing to plan around.
- **`_highlightResult` / `_snippetResult`** — extra sub-objects Algolia adds to each hit with `<em>`-marked matches. This Actor strips HTML tags from every string value it pushes, so those come through as plain text.

### Can I really pull a whole catalog?

Usually not in one run, and this is Algolia's limit rather than the Actor's. Most indexes leave
`paginationLimitedTo` at its default of **1,000 records**, so a single query can never hand you more
than that no matter how large `maxItems` or `hitsPerPage` are. The way around it is to slice the
index into chunks that each stay under the cap:

1. Run **Discover facets only** to see the facetable attributes and how many records sit under each value.
2. Pick a facet that partitions the index reasonably — a category, a brand, a type, a date bucket.
3. Run one scrape per value via `facetFilters`, or per range via `numericFilters` (e.g. `["price>=0","price<50"]`, then `["price>=50","price<100"]`, and so on).

### What input does it take?

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `algoliaAppId` | string | Hacker News' `UJ5WYC0L7X` | **Required.** Algolia Application ID. |
| `algoliaApiKey` | string | Hacker News' public key | **Required.** Search-only Algolia API key. |
| `indexName` | string | `Item_dev` | **Required.** Index to query. |
| `searchQuery` | string | `""` | Free-text query. Empty matches all records (browse mode). |
| `filters` | string | `""` | Algolia filter expression, e.g. `category:Shoes AND price < 50`. |
| `facetFilters` | array | `[]` | Array of facet filters; a nested array is an OR group. |
| `numericFilters` | array | `[]` | Array of numeric comparisons, e.g. `["price>=10", "price<=100"]`. |
| `facets` | array | `["*"]` | Facet attributes to request counts for; `["*"]` means all facetable ones. |
| `maxValuesPerFacet` | integer | `100` | Cap on facet values returned per facet (1–1000). |
| `discoverFacetsOnly` | boolean | `false` | Make one `hitsPerPage=0` request and output only the facets and their counts — no items scraped. |
| `attributesToRetrieve` | array | `[]` | Limit which attributes come back per hit. Empty retrieves all. |
| `hitsPerPage` | integer | `1000` | Hits per Algolia request (1–1000). |
| `maxItems` | integer | `0` | Total item cap. `0` = every page Algolia will serve. |
| `extraQueryParams` | object | `{}` | Any other raw Algolia `SearchParams` field — `distinct`, `optionalFilters`, `restrictSearchableAttributes`, `aroundLatLng`, and so on — merged into the request body last, so it overrides everything above. |

Example input for facet discovery:

```json
{
    "algoliaAppId": "latency",
    "algoliaApiKey": "6be0576ff61c053d5f9a3225e2a90f76",
    "indexName": "instant_search",
    "facets": ["*"],
    "discoverFacetsOnly": true
}
```

Example input for a filtered slice of the same index:

```json
{
    "algoliaAppId": "latency",
    "algoliaApiKey": "6be0576ff61c053d5f9a3225e2a90f76",
    "indexName": "instant_search",
    "facetFilters": ["brand:Apple"],
    "numericFilters": ["price>=500"],
    "hitsPerPage": 1000,
    "maxItems": 0
}
```

### What does a row look like?

Each scraped record is pushed to the dataset as Algolia returns it, with HTML stripped from string
values. The attributes are whatever the source index stores, so they differ per site — only
`objectID` is guaranteed. From the `instant_search` demo catalog:

```json
{
    "objectID": "6443034",
    "name": "Apple - MacBook Air\u00ae (Latest Model) - 13.3\" Display - Intel Core i5 - 8GB Memory - 128GB Flash Storage - Silver",
    "brand": "Apple",
    "categories": ["Computers & Tablets", "Laptops", "All Laptops", "MacBooks"],
    "type": "Apple",
    "price": 999.99,
    "price_range": "500 - 2000",
    "rating": 4,
    "free_shipping": true,
    "image": "https://cdn-demo.algolia.com/bestbuy-0118/6443034_sb.jpg",
    "url": "https://api.bestbuy.com/click/-/6443034/pdp"
}
```

When **Discover facets only** is enabled, the dataset instead contains a single item:

```json
{
    "facets": {
        "brand": { "Insignia\u2122": 746, "Samsung": 633 },
        "price_range": { "1 - 50": 9927, "50 - 100": 3470 }
    },
    "facets_stats": {
        "price": { "min": 1, "max": 4999.99, "avg": 242.813, "sum": 5212710 }
    },
    "nbHits": 21468
}
```

The same facet data is also saved to the key-value store under the `FACETS` key. In hit-scraping
mode, `FACETS` gets the facet counts seen on the last page fetched. Download the dataset as JSON,
CSV, Excel, XML or HTML.

### Which fields are in the dataset?

| Field | Description |
| --- | --- |
| `objectID` | Algolia's unique identifier for the record — present on every hit |
| *(all other fields)* | Whatever attributes the source index stores; varies per site and per index |

In facet-discovery mode the single row is `facets`, `facets_stats` and `nbHits` instead.

### How much does it cost?

The current price of every event is on the **Pricing** tab of this Actor's page in Apify Store. That tab is generated from the Actor's own pricing record, so it is always current, and your plan's rung is applied automatically. A figure repeated here would be a second copy that drifts out of date.

You can cap what any single run may spend with **Max total charge**.

### How do I get more out of it?

- Leave `hitsPerPage` at its maximum of 1000 to minimise requests.
- Use `maxItems` to cap a run when you only need a sample.
- Run **Discover facets only** first on a new index, then build a targeted `filters`/`facetFilters` query for the real scrape — and to work around `paginationLimitedTo`, as described above.
- Set `attributesToRetrieve` to just the fields you need, to shrink rows. Note that Algolia still attaches `_highlightResult` and `_snippetResult` sub-objects regardless, so drop those in post-processing if you don't want them.
- Pass any additional Algolia parameter (`distinct`, `optionalFilters`, `restrictSearchableAttributes`, `aroundLatLng`) via `extraQueryParams`.

### FAQ

**Do I need an Algolia account or a paid API key?**

No. You need the search-only key the target site already ships to its own visitors, and an Apify account. The Actor has no Algolia account of its own.

**Do I need a proxy?**

No. The Actor calls Algolia's API hosts directly and has no proxy input. It fails over across `<appId>-dsn.algolia.net` and the `-1`/`-2`/`-3.algolianet.com` cluster hosts, and retries `429`/`5xx` responses up to three times per host with backoff.

**Why did I get far fewer records than `nbHits` said?**

Because of `paginationLimitedTo` — Algolia caps how deep you can page, at 1,000 records by default. `nbHits` counts matches; `nbPages` tells you what you can actually fetch. Split the index with facet or numeric filters to get past it.

**Can I scrape a private index or one I don't have a key for?**

No. The Actor only queries with credentials you supply and does not bypass authentication. A `401`, `403`, `404` or `400` from Algolia fails the run immediately rather than retrying, so a wrong key or index name is obvious straight away.

**What happens if the site rotates its key?**

The run fails with the Algolia error and status code in the log. Re-check the site's current Application ID, key and index name in its network requests and update the input — search-only keys do get rotated.

**Does it work for documentation sites and job boards, not just shops?**

Yes. The Actor makes no assumptions about the record shape — it pushes whatever attributes the index stores. That is why the dataset has no fixed schema beyond `objectID`.

**Can I schedule it?**

Yes — use Apify's scheduler for recurring runs and a webhook to push each finished dataset onward. Join runs on `objectID`, which is stable within an index.

**Is this allowed?**

You are responsible for how you use the data. Only query indexes and keys you are authorised to query, and respect the target site's Terms of Service. The Actor accesses no private data and bypasses no authentication.

### Global marketplace toolkit

Part of the **Global marketplace toolkit** — product and pricing data across international B2B/B2C marketplaces and classifieds sites:

- [Shopify API Scraper](https://apify.com/rl1987/shopify-api-scraper) — Get product data from almost any Shopify store's public API.
- [DHgate API Scraper](https://apify.com/rl1987/dhgate-api-scraper) — Scrapes DHgate product search results and detail pages via mobile API.
- [Made-in-China.com API Scraper](https://apify.com/rl1987/made-in-china-api-scraper) — Scrapes product search results and details from Made-in-China.com's mobile app API.
- [Back Market API Scraper](https://apify.com/rl1987/backmarket-api-scraper) — Scrape refurbished-device listings and full product detail from Back Market's API.
- [Senukai.lt API Scraper](https://apify.com/rl1987/senukai-api-scraper) — Scrape Senukai.lt product listings and full detail — prices, discounts, EAN/GTIN, images.
- [Kleinanzeigen API Scraper](https://apify.com/rl1987/kleinanzeigen-api-scraper) — Scrape Kleinanzeigen (Germany) classified-ad listings and full details via mobile API.

### Did you find this useful?

Rate this actor on Apify! Your feedback helps other users find it and helps us keep improving it.

# Actor input Schema

## `algoliaAppId` (type: `string`):

The Algolia Application ID (find it in the site's network requests as `X-Algolia-Application-Id` or in the request URL host `<appId>-dsn.algolia.net`). Defaults to Hacker News Search's own public Algolia integration (hn.algolia.com).

## `algoliaApiKey` (type: `string`):

A search-only Algolia API key (header `X-Algolia-API-Key`). Most public sites expose this in their frontend JS bundle or network requests. Defaults to Hacker News Search's public search-only key.

## `indexName` (type: `string`):

Name of the Algolia index to query. Defaults to Hacker News Search's "Item_dev" index (stories and comments).

## `searchQuery` (type: `string`):

Free-text query string. Leave empty to match all records (browse mode).

## `filters` (type: `string`):

Algolia filter expression, e.g. `category:Shoes AND price < 50`. See https://www.algolia.com/doc/api-reference/api-parameters/filters/

## `facetFilters` (type: `array`):

Algolia facetFilters array, e.g. \["category:Shoes", \["color:Red", "color:Blue"]]. Each entry is a string or a nested array (OR group).

## `numericFilters` (type: `array`):

Algolia numericFilters array, e.g. \["price>=10", "price<=100"].

## `facets` (type: `array`):

Facet attribute names to request counts for. Use \["\*"] to discover every facetable attribute and its values.

## `maxValuesPerFacet` (type: `integer`):

Maximum number of facet values returned per facet.

## `discoverFacetsOnly` (type: `boolean`):

When enabled, the Actor makes a single hitsPerPage=0 request and only outputs the discovered facets and their values/counts, without scraping any search result items.

## `attributesToRetrieve` (type: `array`):

Limit which attributes are returned per hit. Leave empty to retrieve all attributes (`*`).

## `hitsPerPage` (type: `integer`):

Number of hits fetched per Algolia request page.

## `maxItems` (type: `integer`):

Maximum number of items to scrape in total. 0 = unlimited (scrape all pages Algolia returns).

## `extraQueryParams` (type: `object`):

Advanced: any additional Algolia SearchParams fields (e.g. distinct, optionalFilters, restrictSearchableAttributes) merged directly into the request body.

## Actor input object example

```json
{
  "algoliaAppId": "UJ5WYC0L7X",
  "algoliaApiKey": "28f0e1ec37a5e792e6845e67da5f20dd",
  "indexName": "Item_dev",
  "searchQuery": "arcteryx",
  "filters": "",
  "facetFilters": [],
  "numericFilters": [],
  "facets": [
    "*"
  ],
  "maxValuesPerFacet": 100,
  "discoverFacetsOnly": false,
  "attributesToRetrieve": [],
  "hitsPerPage": 1000,
  "maxItems": 0,
  "extraQueryParams": {}
}
```

# Actor output Schema

## `results` (type: `string`):

Rows produced by the run.

## `facets` (type: `string`):

Aggregated facet values, facet statistics and total hit count for the query.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "algoliaAppId": "UJ5WYC0L7X",
    "algoliaApiKey": "28f0e1ec37a5e792e6845e67da5f20dd",
    "indexName": "Item_dev",
    "searchQuery": "arcteryx"
};

// Run the Actor and wait for it to finish
const run = await client.actor("rl1987/algolia-api-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "algoliaAppId": "UJ5WYC0L7X",
    "algoliaApiKey": "28f0e1ec37a5e792e6845e67da5f20dd",
    "indexName": "Item_dev",
    "searchQuery": "arcteryx",
}

# Run the Actor and wait for it to finish
run = client.actor("rl1987/algolia-api-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "algoliaAppId": "UJ5WYC0L7X",
  "algoliaApiKey": "28f0e1ec37a5e792e6845e67da5f20dd",
  "indexName": "Item_dev",
  "searchQuery": "arcteryx"
}' |
apify call rl1987/algolia-api-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rl1987/algolia-api-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/r1TznFP4fd6AGdJNZ/builds/VALwaiL1QErWCq80q/openapi.json