we build scrapers and keep them working.
We build data extractors, run them on Apify, and rebuild them when the sites change. We do not sell lists of people, and we do not fight sites that do not want to be read.
the kinds of job
- Online stores and marketplaces: products, prices, stock, ratings, category and search listings.
- Property portals: listings with price, size, rooms and location.
- News and articles: headline, date, body text, canonical link.
- Public job boards on the common hiring platforms: title, team, location, posting date, description.
- Public registries and open government data: filings, recalls, grants, advisories, notices.
- App and extension stores, package registries: metadata, versions, public ratings.
- Document and image utilities: HTML to PDF or image, QR codes, image conversion, PDF text.
from reading the site to the daily check
- Read the site and its rules. What it shows, what its robots.txt allows, whether the data is public at all.
- Build. Plain HTTP when the page is plain, a real browser when the page needs one.
- Test on real pages, including the ugly ones: missing prices, sold out items, odd currencies.
- Ship one documented output schema.
- Check it every day with a small default input.
- Rebuild when the site changes, and adapt it to your fields, regions or schedule.
The long version is on the blog: how a scraper job goes.
what the actors follow
- robots.txt first: it is read and obeyed for every URL, except on the two sites named below. A closed page is skipped, never fetched "just once".
- A wall is an answer: no captcha solving, no stealth plugins, no rotating user agents. A captcha or block page sends the run to another proxy and up the tiers, at most twice over, and then the run stops and says so in plain words.
- No personal data: no contact extractors, no lead lists, no people profiles. Review and job outputs carry no person fields, and emails and phone numbers in free text are removed.
- No logins: nothing behind a login and nothing behind a paywall.
- Polite pacing: delays between requests and a cap per run. One run never hammers one host.
- Cheapest proxy that works: direct first, then datacenter, then residential. Browser runs block images, media and fonts.
- One documented schema: a field the page does not show stays empty. Nothing is guessed or filled in.
- Checked every day: every actor has a small default input that must work. A site change means a rebuild, not a shrug.
- Pay per result: a buyer pays for rows delivered, not for time spent.
where we read past robots.txt
Two of our actors read pages that a robots.txt does not open to them. We decided on each one by itself, kept it narrow and wrote down where it ends. Every other actor follows robots.txt for every URL.
- Facebook Ad Library Scraper, decided on 19 October 2026. The robots.txt of facebook.com closes the whole site to any crawler it does not name. The Ad Library is where Meta shows anyone, without a login, the ads running on its apps. The actor opens Ad Library pages and nothing else, in an ordinary browser that runs the page's own script, including the script check Meta puts before the page. No login, no stealth plugins, no fingerprint tricks, no captcha or checkpoint solving. A login wall, a checkpoint, a captcha or a block ends the run, and if Meta asks us to stop, we stop.
- TikTok Shop Product Scraper, decided on 20 October 2026. shop.tiktok.com answers a request for its robots.txt with a captcha page, so the file cannot be read. The actor reads product pages and nothing else: no shop or creator pages, search, categories, reviews or videos. The robots.txt of www.tiktok.com is read and obeyed, and the first verification page ends the run.
a few pieces of how they are built
These are written for this page. They show the shape of the real thing without exposing it.
the input is the first contract
A listing scraper asks for a few start pages, a cap, a country and one switch. The proxy block defaults to off, and the prefilled values are what runs when anybody clicks start without typing.
{
"title": "Store listing input",
"type": "object",
"schemaVersion": 1,
"properties": {
"startUrls": {"title": "Category or search pages", "type": "array", "editor": "requestListSources",
"prefill": [{"url": "https://shop.example.com/c/headphones"}]},
"maxItems": {"title": "Max results", "type": "integer", "minimum": 1, "default": 100, "prefill": 20},
"country": {"title": "Country", "type": "string", "enum": ["US", "GB", "DE", "FR"], "default": "US"},
"includeOutOfStock": {"title": "Include out of stock", "type": "boolean", "default": false},
"proxy": {"title": "Proxy", "type": "object", "editor": "proxy", "default": {"useApifyProxy": false}}
},
"required": ["startUrls"]
}robots.txt is checked before every request
from functools import lru_cache
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser
UA = "ExampleBot/1.0 (+https://example.com/bot)"
@lru_cache(maxsize=256)
def _rules(origin: str) -> RobotFileParser:
rp = RobotFileParser(origin + "/robots.txt")
rp.read()
return rp
def allowed(url: str) -> bool:
parts = urlsplit(url)
return _rules(f"{parts.scheme}://{parts.netloc}").can_fetch(UA, url)a wall stops the run
WALL_MARKERS = ("captcha", "verify you are human", "access denied", "unusual traffic")
class Blocked(Exception):
"""The site asked us to stop. We stop and report it."""
def looks_blocked(html: str) -> bool:
head = html[:20_000].lower()
return any(marker in head for marker in WALL_MARKERS)the cheapest proxy tier that answers
import httpx
LADDER = ("direct", "datacenter", "residential") # proxies["direct"] is None
async def fetch_with_ladder(url, proxies):
for tier in LADDER:
async with httpx.AsyncClient(proxy=proxies.get(tier), timeout=30, headers={"User-Agent": UA}) as client:
resp = await client.get(url)
if resp.status_code == 200 and not looks_blocked(resp.text):
return tier, resp
if resp.status_code == 404:
return tier, None # the page is gone, a better proxy will not bring it back
raise Blocked(f"{url}: no tier got a real page")one record, whatever the shop prints
import re
from dataclasses import dataclass
from decimal import Decimal
from typing import Optional
@dataclass
class Product:
url: str
title: str
price: Optional[Decimal]
currency: Optional[str]
in_stock: Optional[bool]
rating: Optional[float]
reviews_count: Optional[int]
SYMBOLS = {"$": "USD", "£": "GBP", "€": "EUR"}
def parse_price(text: str) -> tuple[Optional[Decimal], Optional[str]]:
m = re.search(r"([$£€])\s*([\d.,]+)", text or "")
if not m:
return None, None # not shown on the page: stays empty, never guessed
raw = m.group(2).rstrip(".,")
if re.fullmatch(r"\d{1,3}(,\d{3})+(\.\d+)?", raw): # 1,299 or 1,299.00
raw = raw.replace(",", "")
elif "," in raw: # 1.299,00 or 12,50
raw = raw.replace(".", "").replace(",", ".")
return Decimal(raw), SYMBOLS[m.group(1)]pay per result, stop cleanly at the limit
import time
from dataclasses import asdict
from apify import Actor
async def main():
async with Actor:
inp = await Actor.get_input() or {}
max_items, pushed = int(inp.get("maxItems", 100)), 0
deadline = time.time() + 55 * 60 # finish with what we have instead of being cut off
async for product in crawl(inp["startUrls"]): # yields Product records, robots-checked
if pushed >= max_items or time.time() > deadline:
break
res = await Actor.push_data(asdict(product)) # one row = one charged result
pushed += 1
if getattr(res, "event_charge_limit_reached", False):
break # the buyer's spending limit: stop, do not fail
await Actor.set_status_message(f"Done: {pushed} results")how a job goes robots.txt first the proxy ladder what apify taught us
the apify store, and the utilities next door
Our scrapers and file tools are on the Apify Store, billed per result. These are the ones we built and keep working. The list is read from the Store every day, and new ones join it once they pass their tests.
shops and marketplaces
- Alibaba Scraper: Prices, MOQ, Verified Suppliers · how to run it
- Alibaba Supplier Shortlist by Price, MOQ and Trust · how to run it
- AliExpress Scraper: Prices, Sold Counts
- Amazon Bestsellers Scraper: New Releases
- Amazon Product Details Scraper: ASIN, Price, Sales Rank · how to run it
- Amazon Search Scraper: Monthly Sales, Badges, Prices
- Coles Price Scraper: Specials, Unit Prices
- TikTok Shop Scraper: Products, Prices, Units Sold, Shops · how to run it
- Vinted Product Scraper: Search Listings, Prices, Brands, Sizes
property
- Fotocasa Scraper: Spain Homes for Sale and Rent
- Rightmove Scraper: UK Property for Sale and to Rent
- Zillow Search Scraper: Homes for Sale, Rent and Sold
jobs
- ATS Jobs Scraper: Greenhouse, Lever, Workday, Ashby · guide · how to run it
- Freelancer.com Jobs and Projects Scraper · how to run it
- Handshake Jobs Scraper
- Hirist Jobs Scraper: India Tech Jobs, Salary · how to run it
- IIMJobs Scraper: India Finance and MBA Jobs
- Instahyre Jobs Scraper: India Jobs, Skills · how to run it
- IrishJobs.ie and Jobs.ie Scraper: Ireland Jobs and Salaries
- Jobs.ch & Jobup.ch Scraper: Swiss Jobs
- Reed Jobs Scraper: UK Jobs and Salaries
- Remote Jobs Scraper: RemoteOK, WWR, Jobicy · guide
- Rozee.pk Jobs Scraper
- Wellfound Jobs Scraper: Salary, Equity, Stage
- Workday Jobs Scraper: Salary, Dates, Remote · guide · how to run it
markets and money
- Currency Exchange Rates (ECB) · guide
- Yahoo Finance Scraper: Options Chain, History, Stock Quotes · how to run it
news, sites and ads
- Eventbrite Event Data Scraper: City, Dates, Ticket Prices
- Facebook Ads Library Scraper: Ads by Keyword or Page · guide · how to run it
- News Article Extractor: Full Text from Links and RSS
- SEO Audit Tool (On-Page Checker, Canonical, Alt Text)
- Similarweb Scraper: Top Websites by Category
- Substack Posts Scraper: Archive, Reactions, Comments · guide
- Website Content Crawler: Pages to Clean Text · guide
files, images and media
- Audio Transcription: Audio to Text Converter, Podcast, Video
- Bulk File Downloader and Image Downloader: Web Pages, PDFs · how to run it
- Convert HTML to PDF (A4, Letter, Legal, A3 and more) · guide
- Extract Text from PDF: Markdown & Metadata · guide · how to run it
- HTML to Image Converter: Convert HTML to PNG, JPG and WEBP · how to run it
- Image Converter: WEBP to PNG, PNG to JPG, JPG to WEBP, GIF
- QR Code Generator (PNG and SVG)
- Video Frame Extractor: Extract Frames from Video, Thumbnails
- Website Screenshot Generator: URL to PNG, JPEG or WEBP · guide · how to run it
more
- Bluesky Scraper
- CarGurus Zipcode Search Scraper · how to run it
- Computrabajo Salary Benchmark by Job Title and Country · how to run it
- Computrabajo Scraper: LATAM Jobs and Salaries · guide · how to run it
- Domain Availability Checker: Bulk RDAP · how to run it
- DuckDuckGo Search Scraper
- elempleo Scraper: Colombia Jobs and Salaries
- Eluta.ca Job Scraper
- ForexFactory Economic Calendar Scraper
- France Travail Scraper: Pole Emploi Jobs and Salaries
- Google Maps Scraper: Places, Phones, Hours · how to run it
- Gupy Jobs Scraper: Brazil Job Vacancies · how to run it
- Hiring.Cafe Jobs Scraper
- Internshala Scraper
- JobTeaser Job Scraper
- JSON Schema Validator: Bulk, Draft 7, 2019-09, 2020-12 · how to run it
- JustJoin.it Scraper: Polish IT Jobs and Salaries
- Kariyer Scraper: Kariyer.net Turkey Job Listings
- Luma Event Scraper
- Made-in-China.com Product Scraper
- MLB Daily Scoreboard Scraper · how to run it
- Moz Top 500 Scraper: Domain Rankings
- Myntra Product Scraper
- North Data Scraper
- PeoplePerHour Job Scraper · how to run it
- Pracuj.pl Salary Benchmark by Job Title and City · how to run it
- Pracuj.pl Scraper: Poland Jobs, Salaries, Contract Type · how to run it
- Rakuten Ichiba Listings Scraper · how to run it
- Royal Caribbean Ships Scraper · how to run it
- Screener.in Stock Fundamentals Scraper
- Snapchat Scraper: Spotlight, Highlights · how to run it
- Substack Newsletter Scraper
- Tennis Scores and Results Scraper: ATP, WTA, Odds, Rankings · how to run it
- Time Zone Converter: IANA Zones, DST, Offsets, Batch · how to run it
- Truth Social Scraper
- WHOIS Domain Lookup: Registrar, Age, Expiry, Name Servers
- Y Combinator Company Directory Scraper
Some of them have a page of their own with every input field, an example and a Python call: data exports.
Separately from the scrapers, we run small file and data utilities: convert, validate, look things up. They work in the browser at utils.voiddo.com, over HTTP and as an MCP endpoint for agents at data.voiddo.com, and from code through the npm client @v0idd0/utils. These are not the scrapers.
what we say no to
We do not build scrapers for networks of people profiles, contact or email finders, classifieds of private sellers, sites whose robots.txt closes the data (the two exceptions above aside), or sites behind behavioural bot walls. The reasons are mostly about people, not technique: what we will not scrape, and why.
If you are not sure whether your job fits, read what a scraper can and cannot do, or write to support@voiddo.com with the site and the fields you need.
about scrapers
5 min readSubstack API in Python: every post of a newsletter with the archive cursor
Substack's RSS feed stops at the newest 20 posts. Its archive endpoint has them all, 25 a page, and it now pages by a cursor: an offset, even offset=0, has answered HTTP 400 since early November. The code that works, measured on three newsletters with 907 to 1,510 posts.
10 min readComputrabajo salaries in Python: half the postings show pay, and A convenir comes out as 0
We read 36 Computrabajo search pages in six countries and 24 job pages. 49 % of 706 postings showed a salary, always as one monthly figure, from 21 % in Argentina to 68 % in Colombia. Where the pay is A convenir, the structured data says 0. How to read the pages, and what to clean before you take a median.
12 min readWorkable, Recruitee, Personio and BambooHR job board APIs: no key, and a date that costs a request
Workable, Recruitee, Personio and BambooHR give a board's open jobs with no key, whole in one request; only BambooHR's list has no date. The four addresses, the catch in each, and about sixty lines of Python that put all four into one shape.
11 min readWorkday jobs API in Python: a list that stops at 2,000 jobs, and dates that stop at 30+ days
Read a Workday careers site as JSON in Python: 20 jobs a request, and a list that stops at 2,000 of NVIDIA's 2,807 jobs. The two endpoints, facets that cut a big site into slices, and about forty lines of Python, measured on four career sites.
7 min readRemote job APIs: five free feeds, and the catch in each
Remote OK, We Work Remotely, Jobicy, Working Nomads and Arbeitnow publish their jobs without a key. What each feed holds, how far back it goes, the terms, and sixty lines of Python that put them in one list.
5 min readFacebook Ad Library API: which ads you get, which you don't
Meta's API returns political ads from everywhere and every kind of ad shown in the UK or the EU. An ad shown only in the US for running shoes is not in it. What each way into the Ad Library gives you, and how to keep the ads the API leaves out.
6 min readFree exchange rate API with no key: ECB rates in Python
The European Central Bank publishes its daily euro rates as plain files: no key, no sign-up. What is in them, the three things that trip people up (weekends, holidays, retired currencies) and about forty lines of Python that handle all three.
6 min readWebsite to LLM text: a site into clean pages with Python
For search over your docs or a chat with a site, a model needs the words of each page, not its menus. About seventy lines of standard-library Python do it for a small site. We ran them on this one: 46 pages, 33,009 words, 132 chunks.
8 min readGreenhouse, Lever, Ashby and Workday job board APIs
The open jobs of Greenhouse, Lever, Ashby and Workday boards as JSON with no key: the four endpoints, what each answer holds and the catch in each. Then about forty lines of Python that put all four into one shape.
- 3 min read
robots.txt readers disagree, and the built-in one is the odd one out
Python's built-in reader takes the first matching rule and ignores wildcards. The standard takes the longest match. We ran both on 12,000 random files and listed where they split.
- 2 min read
The price is a string: reading money off a web page
Shops print the same number a dozen ways. A careful parser reads what is clear, leaves the rest empty and never fills a gap with a plausible guess.
- 3 min read
A timestamp without a zone is a guess
Scraped dates arrive as "2026-03-08 02:30" with nothing else. Some of those times never happened, some happened twice, and the gap between two cities moves during the year.
- 2 min read
Retry-After is a sentence, not a suggestion
When a site says "slow down", it often says how long to wait. Reading that line, backing off with jitter and knowing when to give up is most of polite scraping.
- 2 min read
What Apify taught us about publishing scrapers
Seven things we learned about publishing on the Apify Store: the automated tester, identity checks, rate-limited visibility changes, pay per result and the first "users".
- 3 min read
How a scraper job goes, from reading the site to the daily check
Read the site, build, test on real pages, ship one clean schema, check it every day, rebuild when the site changes. Most of the work is the last two.
- 2 min read
robots.txt first, and what happens when a site says stop
Every URL is checked against the site's robots.txt before it is fetched, with two exceptions we name. When a site puts up a wall, the run stops and says so in plain words.
- 2 min read
What a scraper can and cannot do
A plain list for anyone thinking of buying one: what is realistic, what is not, and what we will say no to.
- 2 min read
Proxies: use the cheapest tier that answers
Direct first, then datacenter, then residential. Residential traffic is billed by the gigabyte, so the ladder is a cost control as much as a technique.
- 2 min read
What we will not scrape, and why
A list of sites we decided not to build for. The reasons are general, which is the point: most of them are not about technique.