scrapers

we build scrapers and keep them working.

We build data extractors, run them on Apify, and rebuild them when the sites change. We do not sell lists of people, and we do not fight sites that do not want to be read.

what we build

the kinds of job

  • Online stores and marketplaces: products, prices, stock, ratings, category and search listings.
  • Property portals: listings with price, size, rooms and location.
  • News and articles: headline, date, body text, canonical link.
  • Public job boards on the common hiring platforms: title, team, location, posting date, description.
  • Public registries and open government data: filings, recalls, grants, advisories, notices.
  • App and extension stores, package registries: metadata, versions, public ratings.
  • Document and image utilities: HTML to PDF or image, QR codes, image conversion, PDF text.
how a job goes

from reading the site to the daily check

  1. Read the site and its rules. What it shows, what its robots.txt allows, whether the data is public at all.
  2. Build. Plain HTTP when the page is plain, a real browser when the page needs one.
  3. Test on real pages, including the ugly ones: missing prices, sold out items, odd currencies.
  4. Ship one documented output schema.
  5. Check it every day with a small default input.
  6. Rebuild when the site changes, and adapt it to your fields, regions or schedule.

The long version is on the blog: how a scraper job goes.

rules

what the actors follow

  • robots.txt first: it is read and obeyed for every URL, except on the two sites named below. A closed page is skipped, never fetched "just once".
  • A wall is an answer: no captcha solving, no stealth plugins, no rotating user agents. A captcha or block page sends the run to another proxy and up the tiers, at most twice over, and then the run stops and says so in plain words.
  • No personal data: no contact extractors, no lead lists, no people profiles. Review and job outputs carry no person fields, and emails and phone numbers in free text are removed.
  • No logins: nothing behind a login and nothing behind a paywall.
  • Polite pacing: delays between requests and a cap per run. One run never hammers one host.
  • Cheapest proxy that works: direct first, then datacenter, then residential. Browser runs block images, media and fonts.
  • One documented schema: a field the page does not show stays empty. Nothing is guessed or filled in.
  • Checked every day: every actor has a small default input that must work. A site change means a rebuild, not a shrug.
  • Pay per result: a buyer pays for rows delivered, not for time spent.
two exceptions

where we read past robots.txt

Two of our actors read pages that a robots.txt does not open to them. We decided on each one by itself, kept it narrow and wrote down where it ends. Every other actor follows robots.txt for every URL.

  • Facebook Ad Library Scraper, decided on 19 October 2026. The robots.txt of facebook.com closes the whole site to any crawler it does not name. The Ad Library is where Meta shows anyone, without a login, the ads running on its apps. The actor opens Ad Library pages and nothing else, in an ordinary browser that runs the page's own script, including the script check Meta puts before the page. No login, no stealth plugins, no fingerprint tricks, no captcha or checkpoint solving. A login wall, a checkpoint, a captcha or a block ends the run, and if Meta asks us to stop, we stop.
  • TikTok Shop Product Scraper, decided on 20 October 2026. shop.tiktok.com answers a request for its robots.txt with a captcha page, so the file cannot be read. The actor reads product pages and nothing else: no shop or creator pages, search, categories, reviews or videos. The robots.txt of www.tiktok.com is read and obeyed, and the first verification page ends the run.
code

a few pieces of how they are built

These are written for this page. They show the shape of the real thing without exposing it.

the input is the first contract

A listing scraper asks for a few start pages, a cap, a country and one switch. The proxy block defaults to off, and the prefilled values are what runs when anybody clicks start without typing.

Input schema of a store listing scraperjson
{
  "title": "Store listing input",
  "type": "object",
  "schemaVersion": 1,
  "properties": {
    "startUrls": {"title": "Category or search pages", "type": "array", "editor": "requestListSources",
                  "prefill": [{"url": "https://shop.example.com/c/headphones"}]},
    "maxItems": {"title": "Max results", "type": "integer", "minimum": 1, "default": 100, "prefill": 20},
    "country": {"title": "Country", "type": "string", "enum": ["US", "GB", "DE", "FR"], "default": "US"},
    "includeOutOfStock": {"title": "Include out of stock", "type": "boolean", "default": false},
    "proxy": {"title": "Proxy", "type": "object", "editor": "proxy", "default": {"useApifyProxy": false}}
  },
  "required": ["startUrls"]
}

robots.txt is checked before every request

robots.txt check for every URLpython
from functools import lru_cache
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser

UA = "ExampleBot/1.0 (+https://example.com/bot)"

@lru_cache(maxsize=256)
def _rules(origin: str) -> RobotFileParser:
    rp = RobotFileParser(origin + "/robots.txt")
    rp.read()
    return rp

def allowed(url: str) -> bool:
    parts = urlsplit(url)
    return _rules(f"{parts.scheme}://{parts.netloc}").can_fetch(UA, url)

a wall stops the run

A wall is an answer, not an error to retrypython
WALL_MARKERS = ("captcha", "verify you are human", "access denied", "unusual traffic")

class Blocked(Exception):
    """The site asked us to stop. We stop and report it."""

def looks_blocked(html: str) -> bool:
    head = html[:20_000].lower()
    return any(marker in head for marker in WALL_MARKERS)

the cheapest proxy tier that answers

Cheapest tier that returns a real pagepython
import httpx

LADDER = ("direct", "datacenter", "residential")      # proxies["direct"] is None

async def fetch_with_ladder(url, proxies):
    for tier in LADDER:
        async with httpx.AsyncClient(proxy=proxies.get(tier), timeout=30, headers={"User-Agent": UA}) as client:
            resp = await client.get(url)
        if resp.status_code == 200 and not looks_blocked(resp.text):
            return tier, resp
        if resp.status_code == 404:
            return tier, None                     # the page is gone, a better proxy will not bring it back
    raise Blocked(f"{url}: no tier got a real page")

one record, whatever the shop prints

Normalising a pricepython
import re
from dataclasses import dataclass
from decimal import Decimal
from typing import Optional

@dataclass
class Product:
    url: str
    title: str
    price: Optional[Decimal]
    currency: Optional[str]
    in_stock: Optional[bool]
    rating: Optional[float]
    reviews_count: Optional[int]

SYMBOLS = {"$": "USD", "£": "GBP", "€": "EUR"}

def parse_price(text: str) -> tuple[Optional[Decimal], Optional[str]]:
    m = re.search(r"([$£€])\s*([\d.,]+)", text or "")
    if not m:
        return None, None                         # not shown on the page: stays empty, never guessed
    raw = m.group(2).rstrip(".,")
    if re.fullmatch(r"\d{1,3}(,\d{3})+(\.\d+)?", raw):   # 1,299 or 1,299.00
        raw = raw.replace(",", "")
    elif "," in raw:                                     # 1.299,00 or 12,50
        raw = raw.replace(".", "").replace(",", ".")
    return Decimal(raw), SYMBOLS[m.group(1)]

pay per result, stop cleanly at the limit

Pay per result, stop cleanlypython
import time
from dataclasses import asdict
from apify import Actor

async def main():
    async with Actor:
        inp = await Actor.get_input() or {}
        max_items, pushed = int(inp.get("maxItems", 100)), 0
        deadline = time.time() + 55 * 60               # finish with what we have instead of being cut off
        async for product in crawl(inp["startUrls"]):    # yields Product records, robots-checked
            if pushed >= max_items or time.time() > deadline:
                break
            res = await Actor.push_data(asdict(product))   # one row = one charged result
            pushed += 1
            if getattr(res, "event_charge_limit_reached", False):
                break                                  # the buyer's spending limit: stop, do not fail
        await Actor.set_status_message(f"Done: {pushed} results")

how a job goes   robots.txt first   the proxy ladder   what apify taught us

where to get them

the apify store, and the utilities next door

Our scrapers and file tools are on the Apify Store, billed per result. These are the ones we built and keep working. The list is read from the Store every day, and new ones join it once they pass their tests.

shops and marketplaces

property

jobs

markets and money

news, sites and ads

files, images and media

more

Some of them have a page of their own with every input field, an example and a Python call: data exports.

Separately from the scrapers, we run small file and data utilities: convert, validate, look things up. They work in the browser at utils.voiddo.com, over HTTP and as an MCP endpoint for agents at data.voiddo.com, and from code through the npm client @v0idd0/utils. These are not the scrapers.

limits

what we say no to

We do not build scrapers for networks of people profiles, contact or email finders, classifieds of private sellers, sites whose robots.txt closes the data (the two exceptions above aside), or sites behind behavioural bot walls. The reasons are mostly about people, not technique: what we will not scrape, and why.

If you are not sure whether your job fits, read what a scraper can and cannot do, or write to support@voiddo.com with the site and the fields you need.

posts

about scrapers

⌂ 换网址