<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <id>https://scrapingant.com/blog/</id>
    <title>ScrapingAnt Blog</title>
    <updated>2026-11-17T00:00:00.000Z</updated>
    <generator>https://github.com/jpmonette/feed</generator>
    <link rel="alternate" href="https://scrapingant.com/blog/"/>
    <subtitle>ScrapingAnt Blog</subtitle>
    <icon>https://scrapingant.com/blog/img/favicon.ico</icon>
    <entry>
        <title type="html"><![CDATA[Where Do Residential Proxy IPs Come From? Ethical Sourcing Explained]]></title>
        <id>https://scrapingant.com/blog/where-do-residential-proxies-come-from</id>
        <link href="https://scrapingant.com/blog/where-do-residential-proxies-come-from"/>
        <updated>2026-11-17T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Residential proxy IPs come from real home devices. Learn how providers get them, how botnet-sourced pools were exposed, and 6 questions to ask before you buy.]]></summary>
        <content type="html"><![CDATA[<p>Every residential proxy IP belongs to a real device in a real home: a phone, a laptop, a smart TV. When you send a request through a residential proxy, it leaves the internet through somebody's connection. So the first question to ask about any residential proxy network is simple: <strong>does that somebody know?</strong></p>
<p><img decoding="async" loading="lazy" alt="How a residential proxy request travels from your scraper through a home device to a website" src="https://scrapingant.com/blog/assets/images/residential-proxy-request-path-1d4f71f768bd4c5b76097e9c4a62fbaf.png" width="2400" height="880" class="img_ev3q"></p>
<p>For some networks the honest answer was no. In the last three years, law enforcement and Google have taken down proxy networks built on millions of hijacked devices. This guide explains how residential IPs are sourced, what "ethically sourced" means in practice, which questions separate a clean pool from a risky one, and how ScrapingAnt answers them.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="where-do-residential-proxies-come-from">Where do residential proxies come from?<a href="https://scrapingant.com/blog/where-do-residential-proxies-come-from#where-do-residential-proxies-come-from" class="hash-link" aria-label="Direct link to Where do residential proxies come from?" title="Direct link to Where do residential proxies come from?" translate="no">​</a></h2>
<p>Residential proxy IPs come from consumer devices whose owners share their internet connection with a proxy network, either knowingly through an app they agreed to, or unknowingly through malware. There are three sourcing models:</p>
<p><img decoding="async" loading="lazy" alt="Three ways residential proxy IPs are sourced: bandwidth-sharing apps, SDKs inside other apps, and malware" src="https://scrapingant.com/blog/assets/images/residential-ip-sourcing-models-6e23637038ae9e88dcfea713528b3fd3.png" width="2400" height="960" class="img_ev3q"></p>
<ul>
<li class=""><strong>Bandwidth-sharing apps.</strong> The owner installs an app whose whole purpose is to share unused bandwidth. They know, because sharing is the reason the app exists.</li>
<li class=""><strong>SDKs inside other apps.</strong> A game, utility or TV app shows an opt-in screen offering perks in exchange for sharing bandwidth. Whether the owner really knows depends on how clear that screen is.</li>
<li class=""><strong>Malware and trojanized apps.</strong> A "free VPN" or pirated program silently turns the device into an exit node. The owner never agreed.</li>
</ul>
<p>Many providers don't run their own apps. They buy capacity from supply partners that do, which means the provider's ethics are only as good as its vetting of those partners.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-does-ethically-sourced-mean-for-residential-proxies">What does "ethically sourced" mean for residential proxies?<a href="https://scrapingant.com/blog/where-do-residential-proxies-come-from#what-does-ethically-sourced-mean-for-residential-proxies" class="hash-link" aria-label="Direct link to What does &quot;ethically sourced&quot; mean for residential proxies?" title="Direct link to What does &quot;ethically sourced&quot; mean for residential proxies?" translate="no">​</a></h2>
<p>An ethically sourced residential IP comes from a person who was clearly told their connection would carry other people's traffic and agreed to it. Across the industry, the term covers four things:</p>
<ol>
<li class=""><strong>Informed consent.</strong> The device owner sees a plain explanation and opts in. Consent buried in terms of service doesn't count.</li>
<li class=""><strong>Control.</strong> The owner can stop sharing whenever they want.</li>
<li class=""><strong>Fair exchange.</strong> The owner gets something back, such as payment per gigabyte or app perks.</li>
<li class=""><strong>Vetted supply.</strong> The provider checks how each supplier obtains consent and drops suppliers that can't show it.</li>
</ol>
<p>No law or regulator defines the phrase, so anyone can print it on a pricing page. That's why the questions further down matter more than the label.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="are-residential-proxies-legal">Are residential proxies legal?<a href="https://scrapingant.com/blog/where-do-residential-proxies-come-from#are-residential-proxies-legal" class="hash-link" aria-label="Direct link to Are residential proxies legal?" title="Direct link to Are residential proxies legal?" translate="no">​</a></h2>
<p>Yes, residential proxies are legal when device owners consented to share their connection and the proxies are used for lawful purposes such as collecting public web data. They become illegal when the devices were enrolled without permission or the traffic is used for fraud.</p>
<p>Two things to know:</p>
<ul>
<li class=""><strong>Privacy law treats an IP address as personal data.</strong> Under GDPR, using someone's connection requires consent that is informed and can be withdrawn. Ethical sourcing is a legal requirement in Europe, not a nice-to-have.</li>
<li class=""><strong>Hijacking devices is a crime.</strong> The operators of botnet-based proxy networks have been charged with computer fraud and wire fraud.</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-happens-when-residential-ips-are-not-ethically-sourced">What happens when residential IPs are not ethically sourced?<a href="https://scrapingant.com/blog/where-do-residential-proxies-come-from#what-happens-when-residential-ips-are-not-ethically-sourced" class="hash-link" aria-label="Direct link to What happens when residential IPs are not ethically sourced?" title="Direct link to What happens when residential IPs are not ethically sourced?" translate="no">​</a></h2>
<p>Pools built on hijacked devices get seized, and their customers lose their proxies overnight. Two cases show the scale:</p>
<ul>
<li class=""><strong>911 S5 (2024).</strong> The US Department of Justice dismantled what the FBI called likely the world's largest botnet: <a href="https://scmagazine.com/news/fbi-takes-down-911-s5-botnet-likely-the-worlds-largest-at-19m-ips" target="_blank" rel="noopener noreferrer" class="">more than 19 million IP addresses</a> in nearly 200 countries. Devices were infected through free VPN programs, and their owners had no idea their IPs were being rented out. Investigators tied the network to an estimated $5.9 billion in fraudulent unemployment claims.</li>
<li class=""><strong>IPIDEA (2026).</strong> In January 2026 Google <a href="https://www.bleepingcomputer.com/news/security/google-disrupts-ipidea-residential-proxy-networks-fueled-by-malware/" target="_blank" rel="noopener noreferrer" class="">took down the infrastructure</a> behind a group of proxy brands including 922 Proxy, ABC Proxy, IP2World, Luna Proxy and PY Proxy. Google found proxy code hidden in at least 600 Android apps and more than 3,000 Windows files. In a single week, over 550 threat groups had used the network's exit nodes.</li>
</ul>
<p><img decoding="async" loading="lazy" alt="911 S5 and IPIDEA proxy network takedowns by the numbers" src="https://scrapingant.com/blog/assets/images/proxy-botnet-takedowns-6881261de92b77b85412f526490e0499.png" width="2400" height="1040" class="img_ev3q"></p>
<p>For a buyer, a dirty pool is a practical problem as well as an ethical one:</p>
<ul>
<li class=""><strong>Your supply can vanish.</strong> When domains are seized, every scraper pointed at that network stops working the same day.</li>
<li class=""><strong>The IPs are burned.</strong> Addresses abused for fraud end up on blocklists, so your success rate drops even for harmless requests.</li>
<li class=""><strong>You share exits with criminals.</strong> Websites that see attacks from a pool treat all of its traffic as hostile.</li>
<li class=""><strong>It lands on your compliance desk.</strong> If your company has to explain where its data came from, "through hijacked home routers" is not an answer you want to give.</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="how-can-you-tell-if-a-proxy-provider-sources-ips-ethically">How can you tell if a proxy provider sources IPs ethically?<a href="https://scrapingant.com/blog/where-do-residential-proxies-come-from#how-can-you-tell-if-a-proxy-provider-sources-ips-ethically" class="hash-link" aria-label="Direct link to How can you tell if a proxy provider sources IPs ethically?" title="Direct link to How can you tell if a proxy provider sources IPs ethically?" translate="no">​</a></h2>
<p>Ask the provider six direct questions and judge how specific the answers are. A provider with a clean pool can answer all of them without hesitation:</p>
<ol>
<li class=""><strong>Where do your residential IPs come from?</strong> You want a sourcing model, not a slogan.</li>
<li class=""><strong>Did every device owner opt in?</strong> Look for explicit consent, shown before any sharing starts.</li>
<li class=""><strong>Were they told what sharing means?</strong> "Your connection will carry web requests from our customers" is clear. "Join our peer network" is not.</li>
<li class=""><strong>Do you vet your supply partners?</strong> If the provider buys capacity, ask how it checks each partner's consent process.</li>
<li class=""><strong>Is any part of the pool from free VPNs or bundled software?</strong> These are the classic channels for unwanted enrollment.</li>
<li class=""><strong>Is the price too good?</strong> Hijacked bandwidth costs nothing to acquire. A price far below the rest of the market deserves a second look.</li>
</ol>
<p>Red flags: no sourcing page at all, a pool size that keeps jumping, and answers that go vague as soon as you ask about consent.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="how-does-scrapingant-source-its-residential-proxies">How does ScrapingAnt source its residential proxies?<a href="https://scrapingant.com/blog/where-do-residential-proxies-come-from#how-does-scrapingant-source-its-residential-proxies" class="hash-link" aria-label="Direct link to How does ScrapingAnt source its residential proxies?" title="Direct link to How does ScrapingAnt source its residential proxies?" translate="no">​</a></h2>
<p>Every IP in the ScrapingAnt residential pool comes from a real user who knowingly agreed to share their connection. Here are our answers to the same questions:</p>
<ul>
<li class=""><strong>Where do the IPs come from?</strong> From real home users around the world who share their connection through a vetted partner network.</li>
<li class=""><strong>Did every device owner opt in?</strong> Yes. Consent comes first. A device joins the pool only after its owner agrees.</li>
<li class=""><strong>Were they told what sharing means?</strong> Yes. Users are informed that their connection will be used as part of a proxy network before they agree.</li>
</ul>
<p>That clean sourcing is what you buy with <a href="https://scrapingant.com/residential-proxies" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt residential proxies</a>: real home IPs with <a href="https://scrapingant.com/proxy-locations" target="_blank" rel="noopener noreferrer" class="">country, state and city targeting</a> and sticky sessions of up to 10 minutes, from a pool you can put in front of your legal team without worry. Plans start at $30.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="key-takeaways">Key takeaways<a href="https://scrapingant.com/blog/where-do-residential-proxies-come-from#key-takeaways" class="hash-link" aria-label="Direct link to Key takeaways" title="Direct link to Key takeaways" translate="no">​</a></h2>
<ul>
<li class="">Residential proxy IPs come from real people's devices, so consent is the whole question.</li>
<li class="">"Ethically sourced" means informed opt-in, control for the device owner, a fair exchange and vetted suppliers.</li>
<li class="">Botnet-sourced networks with tens of millions of IPs have been dismantled, taking their customers' scrapers down with them.</li>
<li class="">Ask any provider the six questions above. Specific answers are the best proof you can get.</li>
<li class="">ScrapingAnt residential proxies come only from users who consented and were informed. <a href="https://scrapingant.com/residential-proxies" target="_blank" rel="noopener noreferrer" class="">See plans and locations</a>.</li>
</ul>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="residential proxies" term="residential proxies"/>
        <category label="proxy" term="proxy"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Railway, GitLab, LiteLLM Breaches: How to Rotate API Keys]]></title>
        <id>https://scrapingant.com/blog/rotate-api-keys-after-breach</id>
        <link href="https://scrapingant.com/blog/rotate-api-keys-after-breach"/>
        <updated>2026-10-24T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Railway leaked env vars, GitLab patched CVSS 10 flaws and LiteLLM shipped a credential-stealing release. How to rotate your API keys, ScrapingAnt's included.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Railway, GitLab, LiteLLM Breaches: How to Rotate API Keys" src="https://scrapingant.com/blog/assets/images/rotate-api-keys-after-breach-df5c654c235423c6e60f0a9d90bc05c9.jpg" width="1200" height="630" class="img_ev3q"></p>
<p>October 2026 has been a rough month for anyone who keeps API keys in the cloud. In a single week, a hosting platform reported that environment variables were decrypted by an intruder, GitLab shipped emergency fixes for a CVSS 10.0 remote code execution bug, and a popular Python package published a release that stole credentials from every machine that installed it.</p>
<p>If your scrapers, data pipelines or CI jobs touched any of these, assume the keys they held are exposed and rotate them. Below is a short summary of each incident from the vendors' own reports, a practical rotation checklist, and exact steps to reset your ScrapingAnt API key and proxy credentials.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-happened">What happened<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#what-happened" class="hash-link" aria-label="Direct link to What happened" title="Direct link to What happened" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="railway-encrypted-environment-variables-accessed-and-partly-decrypted">Railway: encrypted environment variables accessed and partly decrypted<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#railway-encrypted-environment-variables-accessed-and-partly-decrypted" class="hash-link" aria-label="Direct link to Railway: encrypted environment variables accessed and partly decrypted" title="Direct link to Railway: encrypted environment variables accessed and partly decrypted" translate="no">​</a></h3>
<p>Railway published its <a href="https://blog.railway.com/p/security-disclosure-wednesday-october-21st" target="_blank" rel="noopener noreferrer" class="">security disclosure</a> on October 21 and a <a href="https://blog.railway.com/p/preliminary-report-october-security-incident" target="_blank" rel="noopener noreferrer" class="">preliminary report</a> on October 22. According to Railway:</p>
<ul>
<li class="">An attacker ran a "sustained and targeted campaign" against its services. Early evidence suggests the goal was collecting AI provider API keys.</li>
<li class="">On October 21 the attacker reached CI credentials, ran an unauthorized query against encrypted environment variables, and decrypted a subset of them.</li>
<li class="">Environment variables may have been exposed for at most 1.85% of workspaces, about 100,000 of them.</li>
<li class="">Railway asked vendors including GitHub, GitLab, Anthropic, OpenAI and Stripe to revoke affected keys, invalidated user sessions and API tokens, and rolled its own credentials.</li>
</ul>
<p>Railway recommends that <strong>all customers who signed up before October 21 rotate the keys stored on Railway</strong>, remove public network access to databases, and review GitHub and container registry audit logs.</p>
<p>If you deploy a scraper on Railway and keep <code>SCRAPINGANT_API_KEY</code> (or any other key) in its variables, that key is in scope.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="gitlab-two-cvss-100-flaws-in-self-managed-instances">GitLab: two CVSS 10.0 flaws in self-managed instances<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#gitlab-two-cvss-100-flaws-in-self-managed-instances" class="hash-link" aria-label="Direct link to GitLab: two CVSS 10.0 flaws in self-managed instances" title="Direct link to GitLab: two CVSS 10.0 flaws in self-managed instances" translate="no">​</a></h3>
<ul>
<li class=""><strong>CVE-2026-110264</strong> (fixed <a href="https://docs.gitlab.com/releases/patches/patch-release-gitlab-19-5-1-released/" target="_blank" rel="noopener noreferrer" class="">October 21</a>): an integer overflow in regular expression compilation that "could have allowed an unauthenticated user to execute arbitrary code on the GitLab server." It affects self-managed GitLab CE/EE from 7.0 up to the fixed versions 18.11.13, 19.0.10, 19.1.9, 19.2.9, 19.3.6, 19.4.3 and 19.5.1. GitLab.com is already patched. The same release also fixes CVE-2026-105777, which could let an unauthenticated user retrieve sensitive internal server credentials.</li>
<li class=""><strong>CVE-2026-85706</strong> (fixed <a href="https://docs.gitlab.com/releases/patches/patch-release-gitlab-19-3-2-released/" target="_blank" rel="noopener noreferrer" class="">September 10</a>): a path traversal in the repository commits API that let an unauthenticated user read arbitrary files from the server. Security researchers <a href="https://www.infoq.com/news/2026/10/gitlab-critical-vulnerabilities/" target="_blank" rel="noopener noreferrer" class="">reported active exploitation</a> shortly after disclosure.</li>
</ul>
<p>Patching closes the hole but does not un-leak anything. If your self-managed instance ran a vulnerable version, treat CI/CD variables, runner tokens, deploy tokens and SSH keys on it as exposed.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="litellm-a-credential-stealing-release-on-pypi-and-docker">LiteLLM: a credential-stealing release on PyPI and Docker<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#litellm-a-credential-stealing-release-on-pypi-and-docker" class="hash-link" aria-label="Direct link to LiteLLM: a credential-stealing release on PyPI and Docker" title="Direct link to LiteLLM: a credential-stealing release on PyPI and Docker" translate="no">​</a></h3>
<p>On October 21, LiteLLM <a href="https://docs.litellm.ai/blog/security-incident-oct-21-2026" target="_blank" rel="noopener noreferrer" class="">reported</a> that <code>litellm==1.107.0</code> on PyPI, and the <code>v1.107.0</code>, <code>latest</code>, <code>main-latest</code> and <code>main-stable</code> Docker tags, were built from an attacker-controlled repository. The malicious code "steals credentials and loads remote-code execution tools" and runs on Python startup, even if LiteLLM is never imported. LiteLLM's advice: if 1.107.0 was installed on a host with internet access, "assume the exfil succeeded," rebuild the environment and rotate every credential that host could reach.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="why-scraping-and-data-teams-should-care">Why scraping and data teams should care<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#why-scraping-and-data-teams-should-care" class="hash-link" aria-label="Direct link to Why scraping and data teams should care" title="Direct link to Why scraping and data teams should care" translate="no">​</a></h2>
<p>Scraping workloads tend to sit exactly where these incidents hit: a container on a PaaS, a scheduled CI job, a Python environment with a dozen AI and data SDKs installed. A typical scraper holds several keys at once: a web scraping API key, proxy credentials, an LLM key for extraction, database and storage credentials.</p>
<p>A leaked scraping key costs you in two ways. Anyone who has it can spend your API credits, and your monthly quota can run out in the middle of a job you actually need. Rotating takes a minute. Waiting to see whether a key gets abused does not save anything.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="how-to-rotate-api-keys-after-a-breach-a-checklist">How to rotate API keys after a breach: a checklist<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#how-to-rotate-api-keys-after-a-breach-a-checklist" class="hash-link" aria-label="Direct link to How to rotate API keys after a breach: a checklist" title="Direct link to How to rotate API keys after a breach: a checklist" translate="no">​</a></h2>
<ol>
<li class=""><strong>Inventory first.</strong> List every key stored on the affected platform or reachable from the affected machine: environment variables, CI secrets, <code>.env</code> files, config maps, notebook secrets.</li>
<li class=""><strong>Prioritise by blast radius.</strong> Rotate keys that can spend money or reach data first (payment, cloud, AI providers, scraping APIs, databases), then everything else.</li>
<li class=""><strong>Rotate at the source, then redeploy.</strong> Generate the new key in the provider's dashboard, update every place that uses it, and redeploy. Many services, ScrapingAnt included, invalidate the old key the moment you generate a new one, so have the new value ready to paste.</li>
<li class=""><strong>Check usage and audit logs.</strong> Look for usage you don't recognise around the incident dates (October 20 to 22 for Railway, October 21 for LiteLLM).</li>
<li class=""><strong>Don't stop at patching.</strong> Upgrading GitLab or pinning LiteLLM to a safe version stops new damage. Only rotation makes stolen keys useless.</li>
<li class=""><strong>Make the next rotation cheaper.</strong> Keep keys in one secrets manager or one set of environment variables, so a future rotation means changing one value instead of hunting through repositories.</li>
</ol>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="how-to-reset-your-scrapingant-api-key">How to reset your ScrapingAnt API key<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#how-to-reset-your-scrapingant-api-key" class="hash-link" aria-label="Direct link to How to reset your ScrapingAnt API key" title="Direct link to How to reset your ScrapingAnt API key" translate="no">​</a></h2>
<p>Your ScrapingAnt API key is used for the Web Scraping API, the Markdown and AI extraction endpoints, proxy mode, the MCP server and our SDKs.</p>
<ol>
<li class="">Sign in and open your <a href="https://app.scrapingant.com/profile" target="_blank" rel="noopener noreferrer" class="">Profile page</a>.</li>
<li class="">In <strong>Account security</strong>, click <strong>Regenerate token</strong> under the <strong>API token</strong> field.</li>
<li class="">Copy the new token and update every integration that uses it.</li>
</ol>
<p>The new token is generated as soon as you click, and the old one stops working, so requests that still use it will get a <code>403</code>. We also email your account address when the API token changes. If you get that email and didn't reset the key yourself, reset it again and contact support.</p>
<p>Full details, including every place the key is typically used, are in the docs: <a href="https://docs.scrapingant.com/api-key-security" target="_blank" rel="noopener noreferrer" class="">Reset your API key</a>.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="how-to-reset-your-scrapingant-proxy-credentials">How to reset your ScrapingAnt proxy credentials<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#how-to-reset-your-scrapingant-proxy-credentials" class="hash-link" aria-label="Direct link to How to reset your ScrapingAnt proxy credentials" title="Direct link to How to reset your ScrapingAnt proxy credentials" translate="no">​</a></h2>
<p>Residential and datacenter proxy plans use their own username and password, not the API key.</p>
<ol>
<li class="">Open your <a href="https://app.scrapingant.com/profile" target="_blank" rel="noopener noreferrer" class="">Profile page</a> and find <strong>Proxy credentials</strong>.</li>
<li class="">Click <strong>Reset</strong> next to <strong>Residential proxies</strong> or <strong>Datacenter proxies</strong>, and confirm.</li>
<li class="">Copy the new password. Your username stays the same.</li>
</ol>
<p>Resetting ends all active proxy sessions. For datacenter proxies, the new password can take up to 15 minutes to take effect. See <a href="https://proxydocs.scrapingant.com/reset-credentials" target="_blank" rel="noopener noreferrer" class="">Reset your proxy credentials</a> for the full guide.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="keep-the-key-out-of-your-code">Keep the key out of your code<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#keep-the-key-out-of-your-code" class="hash-link" aria-label="Direct link to Keep the key out of your code" title="Direct link to Keep the key out of your code" translate="no">​</a></h2>
<p>The easiest key to rotate is the one that lives in exactly one place. Read it from the environment instead of hard-coding it:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token keyword" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> os</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> requests</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">response </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> requests</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">get</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token string" style="color:rgb(206, 145, 120)">"https://api.scrapingant.com/v2/general"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    params</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token string" style="color:rgb(206, 145, 120)">"url"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"https://example.com"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token string" style="color:rgb(206, 145, 120)">"x-api-key"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> os</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">environ</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">"SCRAPINGANT_API_KEY"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">print</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">response</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">status_code</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><br></div></code></pre></div></div>
<p>When you rotate, you update <code>SCRAPINGANT_API_KEY</code> in your platform's variables or secrets manager, redeploy, and you're done. A few more habits that pay off:</p>
<ul>
<li class="">Never commit keys to Git. Deleting the commit later does not remove the key from forks, clones or caches.</li>
<li class="">Don't call the API from browser or mobile code, where anyone can read the key. Send requests from your backend.</li>
<li class="">Give each environment or project its own secrets entry so you know exactly what to update.</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="faq">FAQ<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#faq" class="hash-link" aria-label="Direct link to FAQ" title="Direct link to FAQ" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="do-i-need-to-rotate-my-scrapingant-api-key-if-i-used-railway">Do I need to rotate my ScrapingAnt API key if I used Railway?<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#do-i-need-to-rotate-my-scrapingant-api-key-if-i-used-railway" class="hash-link" aria-label="Direct link to Do I need to rotate my ScrapingAnt API key if I used Railway?" title="Direct link to Do I need to rotate my ScrapingAnt API key if I used Railway?" translate="no">​</a></h3>
<p>If you stored it as a Railway environment variable before October 21, 2026, yes. Railway recommends rotating all keys stored on the platform for customers who signed up before that date, and it takes less than a minute.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="does-resetting-the-api-key-cancel-my-subscription-or-credits">Does resetting the API key cancel my subscription or credits?<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#does-resetting-the-api-key-cancel-my-subscription-or-credits" class="hash-link" aria-label="Direct link to Does resetting the API key cancel my subscription or credits?" title="Direct link to Does resetting the API key cancel my subscription or credits?" translate="no">​</a></h3>
<p>No. Regenerating the token only replaces the key. Your plan and remaining credits stay the same.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="i-use-gitlabcom-not-self-managed-gitlab-am-i-affected-by-cve-2026-110264">I use GitLab.com, not self-managed GitLab. Am I affected by CVE-2026-110264?<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#i-use-gitlabcom-not-self-managed-gitlab-am-i-affected-by-cve-2026-110264" class="hash-link" aria-label="Direct link to I use GitLab.com, not self-managed GitLab. Am I affected by CVE-2026-110264?" title="Direct link to I use GitLab.com, not self-managed GitLab. Am I affected by CVE-2026-110264?" translate="no">​</a></h3>
<p>GitLab says GitLab.com already runs the patched version. The flaw affects self-managed instances that haven't upgraded to a fixed version.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="is-upgrading-litellm-enough">Is upgrading LiteLLM enough?<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#is-upgrading-litellm-enough" class="hash-link" aria-label="Direct link to Is upgrading LiteLLM enough?" title="Direct link to Is upgrading LiteLLM enough?" translate="no">​</a></h3>
<p>No. LiteLLM says to treat any host that installed 1.107.0 as compromised, rebuild it, and rotate every credential it could reach.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="how-often-should-i-rotate-api-keys">How often should I rotate API keys?<a href="https://scrapingant.com/blog/rotate-api-keys-after-breach#how-often-should-i-rotate-api-keys" class="hash-link" aria-label="Direct link to How often should I rotate API keys?" title="Direct link to How often should I rotate API keys?" translate="no">​</a></h3>
<p>Rotate immediately after any incident that might have exposed a key, and when someone who had access leaves your team. Beyond that, a regular schedule (for example every few months) keeps the process practiced, so it's quick when you really need it.</p>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="security" term="security"/>
        <category label="api" term="api"/>
        <category label="web scraping" term="web scraping"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HTML vs Markdown vs Plain Text for LLM Extraction]]></title>
        <id>https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction</id>
        <link href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction"/>
        <updated>2026-10-11T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Compare HTML, Markdown and plain text with fixed-model extraction tests, token counts and field-loss examples. Choose and validate an MCP output format.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="HTML vs Markdown vs Plain Text for LLM Extraction" src="https://scrapingant.com/blog/assets/images/html-markdown-plain-text-llm-extraction-aafbd07c4b1a9b00b063d96174b8c7d1.png" width="1200" height="630" class="img_ev3q"></p>
<p><strong>Choose the smallest representation that preserves the information your extraction task needs.</strong> Plain text can be sufficient for visible values. Markdown is a useful candidate when headings, tables and link destinations matter. Keep HTML when the answer depends on DOM attributes or script data that conversion removes.</p>
<p>We tested the same six-field extraction contract on three saved inputs, then on 15 hosted MCP returns. HTML produced correct records in each tested case. Markdown worked for the purchase-link case but returned empty arrays for the table. Local text worked for the table; hosted raw-mode text did not. These small, fixed-model results make the practical rule sharper: <strong>preserve the fields, then validate complete records before comparing cost.</strong></p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"></path></svg></span>Evidence scope</div><div class="admonitionContent_BuS1"><p>This study includes reference transformations, field preservation, tokenization, <strong>15 authenticated static-page MCP retrievals and 58 fixed-model extraction responses</strong>. Two additional requests failed before inference and remain in the evidence. The three cases contain seven unique products; repeated responses are not independent websites. Hosted acquisition and same-capture comparisons are reported separately. The named author reviewed the methodology, conclusions and byline before publication.</p></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="which-format-should-you-send-to-an-llm">Which format should you send to an LLM?<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#which-format-should-you-send-to-an-llm" class="hash-link" aria-label="Direct link to Which format should you send to an LLM?" title="Direct link to Which format should you send to an LLM?" translate="no">​</a></h2>
<table><thead><tr><th>Your required information</th><th>Candidate to test first</th><th>Check before accepting it</th></tr></thead><tbody><tr><td>Visible prose or isolated displayed values</td><td>Plain text</td><td>Labels, boundaries and every required value still survive</td></tr><tr><td>Headings, ordinary tables, named links and their destinations</td><td>Markdown</td><td>Row/column associations and actual URLs survive conversion</td></tr><tr><td><code>data-*</code> attributes, element identity, DOM relationships or script-carried data</td><td>HTML, or a purpose-built parser before the model</td><td>The field exists in the acquired state and remains in the input</td></tr><tr><td>A complex table, code example or ambiguous price</td><td>Compare the actual outputs</td><td>The structure your task depends on survives; validate the model's records</td></tr></tbody></table>
<p>These are candidates to test, not universal accuracy rankings. Our table experiment shows why: Markdown preserved every required literal but the fixed model returned no records. Use an exact-record validator after extraction, whichever format you choose.</p>
<p>ScrapingAnt's <a href="https://scrapingant.com/mcp-server-web-scraping" target="_blank" rel="noopener noreferrer" class="">MCP server</a> exposes HTML, Markdown and text tools with shared retrieval controls. That gives you a practical way to evaluate format choice in one managed workflow. The <a href="https://scrapingant.com/llm-ready-data-extraction" target="_blank" rel="noopener noreferrer" class="">URL to Markdown API</a> page covers conversion and its earlier token study; this article focuses on what your task loses.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="same-contract-same-saved-input">Same contract, same saved input<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#same-contract-same-saved-input" class="hash-link" aria-label="Direct link to Same contract, same saved input" title="Direct link to Same contract, same saved input" translate="no">​</a></h2>
<p>The fixed task selects the table with <code>Product</code>, <code>Price</code>, <code>Units sold</code>, <code>Share</code> headers, or the offers under <code>Desk lamps</code> / <code>Travel bottles</code> on the other pages. Other demonstration tables and documentation links are outside that product region. Every format still receives the complete source conversion.</p>
<p>The contract requests those products in document order with six fields: <code>name</code>, <code>displayed_price</code>, <code>units_sold</code>, <code>share</code>, <code>sku</code>, and <code>buy_url</code>. Preserve displayed strings, resolve relative purchase URLs using the supplied source URL, and return <code>null</code> when that product region lacks a field. Do not guess a SKU from its URL.</p>
<p>We used one existing, authorized catalog capture and two authored local fixtures:</p>
<table><thead><tr><th>Case</th><th style="text-align:right">Products</th><th style="text-align:right">Required non-null slots</th><th>What it isolates</th></tr></thead><tbody><tr><td>Visible table</td><td style="text-align:right">3</td><td style="text-align:right">12</td><td>Names, displayed prices, units and percentage shares</td></tr><tr><td>Purchase links</td><td style="text-align:right">2</td><td style="text-align:right">8</td><td>Visible names/prices/SKUs, with identical “Buy” labels and different URLs</td></tr><tr><td>Attribute-only SKUs</td><td style="text-align:right">2</td><td style="text-align:right">8</td><td>Names/prices/links, with SKUs stored only in <code>data-sku</code></td></tr></tbody></table>
<p>Absent fields still belong in the JSON contract as nulls, but do not inflate preservation scores. The oracle is a separate literal JSON file, not generated from the parser's output. The same AI execution session wrote the local fixtures and oracle; independent review must check their agreement. These deliberately chosen examples explain specific failures, not their prevalence across websites.</p>
<p>Every arm starts from identical UTF-8 source bytes. HTML keeps those bytes. Markdown follows the inspected converter recipe: BeautifulSoup removes <code>script</code> and <code>noscript</code>, then html2text converts the remaining document with links enabled. Text uses a local Chromium selection reference with page requests blocked and document scripts disabled. The browser worker's inspected selection recipe informed that reference; it does not verify the hosted tool's output, especially <code>browser=false</code> text behavior.</p>
<p>The table capture comes unchanged from the <a class="" href="https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark">Requests/Playwright acquisition study</a>. We do not repeat its timing or credit comparison here. The richer fixtures remain local files; no new public fixture endpoint was created.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-survived-before-asking-a-model">What survived, before asking a model<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#what-survived-before-asking-a-model" class="hash-link" aria-label="Direct link to What survived, before asking a model" title="Direct link to What survived, before asking a model" translate="no">​</a></h2>
<figure class="figure_vYhu" aria-label="Content tokens and required field survival on three saved fixtures"><div class="grid_BS5g"><section class="case_Fgje" aria-label="Visible table"><h3>Visible table</h3><table><caption class="srOnly_r239">Visible table<!-- -->: content-only o200k tokens and non-null field slots</caption><thead><tr><th scope="col">Format</th><th scope="col">Tokens</th><th scope="col">Fields</th></tr></thead><tbody><tr><th scope="row">HTML</th><td>1,522</td><td class="present_Nzak">12<!-- -->/<!-- -->12</td></tr><tr><th scope="row">Markdown</th><td>573</td><td class="present_Nzak">12<!-- -->/<!-- -->12</td></tr><tr><th scope="row">Text</th><td>415</td><td class="present_Nzak">12<!-- -->/<!-- -->12</td></tr></tbody></table><p>All required values survive in all three reference outputs.</p></section><section class="case_Fgje" aria-label="Purchase links"><h3>Purchase links</h3><table><caption class="srOnly_r239">Purchase links<!-- -->: content-only o200k tokens and non-null field slots</caption><thead><tr><th scope="col">Format</th><th scope="col">Tokens</th><th scope="col">Fields</th></tr></thead><tbody><tr><th scope="row">HTML</th><td>173</td><td class="present_Nzak">8<!-- -->/<!-- -->8</td></tr><tr><th scope="row">Markdown</th><td>72</td><td class="present_Nzak">8<!-- -->/<!-- -->8</td></tr><tr><th scope="row">Text</th><td>55</td><td class="lost_x6dy">6<!-- -->/<!-- -->8</td></tr></tbody></table><p>Text loses two destinations. Markdown retains the links.</p></section><section class="case_Fgje" aria-label="Attribute-only SKUs"><h3>Attribute-only SKUs</h3><table><caption class="srOnly_r239">Attribute-only SKUs<!-- -->: content-only o200k tokens and non-null field slots</caption><thead><tr><th scope="col">Format</th><th scope="col">Tokens</th><th scope="col">Fields</th></tr></thead><tbody><tr><th scope="row">HTML</th><td>143</td><td class="present_Nzak">8<!-- -->/<!-- -->8</td></tr><tr><th scope="row">Markdown</th><td>45</td><td class="lost_x6dy">6<!-- -->/<!-- -->8</td></tr><tr><th scope="row">Text</th><td>26</td><td class="lost_x6dy">4<!-- -->/<!-- -->8</td></tr></tbody></table><p>Markdown and text lose both data-sku values.</p></section></div><figcaption>Offline reference outputs, October 11, 2026. “Fields” means required non-null literals present; it does not measure LLM extraction accuracy. Tokens exclude prompts and tool envelopes.</figcaption></figure>
<p>“Present” means the expected non-null literal exists in the representation, allowing a relative link path with its separately supplied base URL. All tested purchase URLs lack query or fragment components; the current relative-URL checker tests pathname only and must be extended before evaluating those components. It is a field-slot survival check. Repeated equal values count as separate expected slots; this check does <strong>not</strong> establish product associations or accepted model records.</p>
<p>The purchase-link Markdown includes these two captured excerpts, in order:</p>
<div class="language-markdown codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-markdown codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token url">[</span><span class="token url content">Buy</span><span class="token url">](</span><span class="token url">/buy/lamp-a</span><span class="token url">)</span><br></div></code></pre></div></div>
<div class="language-markdown codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-markdown codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token url">[</span><span class="token url content">Buy</span><span class="token url">](</span><span class="token url">/buy/lamp-b</span><span class="token url">)</span><span class="token plain"> Newsletter | Contact</span><br></div></code></pre></div></div>
<p>The text output contains <code>Buy</code> twice, but neither destination. A downstream process cannot recover which URLs those labels pointed to from that text alone. The footer also illustrates that conversion is not automatic main-content extraction.</p>
<p>For the SKU case, the source contains this complete product element:</p>
<div class="language-html codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-html codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&lt;</span><span class="token tag" style="color:rgb(78, 201, 176)">article</span><span class="token tag" style="color:rgb(78, 201, 176)"> </span><span class="token tag attr-name" style="color:rgb(156, 220, 254)">data-sku</span><span class="token tag attr-value punctuation attr-equals" style="color:rgb(212, 212, 212)">=</span><span class="token tag attr-value punctuation" style="color:rgb(212, 212, 212)">"</span><span class="token tag attr-value" style="color:rgb(206, 145, 120)">BOTTLE-RED</span><span class="token tag attr-value punctuation" style="color:rgb(212, 212, 212)">"</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&gt;</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&lt;</span><span class="token tag" style="color:rgb(78, 201, 176)">h2</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&gt;</span><span class="token plain">Red Bottle</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&lt;/</span><span class="token tag" style="color:rgb(78, 201, 176)">h2</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&gt;</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&lt;</span><span class="token tag" style="color:rgb(78, 201, 176)">p</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&gt;</span><span class="token plain">Price: $12.00</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&lt;/</span><span class="token tag" style="color:rgb(78, 201, 176)">p</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&gt;</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&lt;</span><span class="token tag" style="color:rgb(78, 201, 176)">a</span><span class="token tag" style="color:rgb(78, 201, 176)"> </span><span class="token tag attr-name" style="color:rgb(156, 220, 254)">href</span><span class="token tag attr-value punctuation attr-equals" style="color:rgb(212, 212, 212)">=</span><span class="token tag attr-value punctuation" style="color:rgb(212, 212, 212)">"</span><span class="token tag attr-value" style="color:rgb(206, 145, 120)">/buy/red-bottle</span><span class="token tag attr-value punctuation" style="color:rgb(212, 212, 212)">"</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&gt;</span><span class="token plain">Buy</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&lt;/</span><span class="token tag" style="color:rgb(78, 201, 176)">a</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&gt;</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&lt;/</span><span class="token tag" style="color:rgb(78, 201, 176)">article</span><span class="token tag punctuation" style="color:rgb(212, 212, 212)">&gt;</span><br></div></code></pre></div></div>
<p>The Markdown retains the name, price and link; it contains no <code>BOTTLE-RED</code>. Supplying that SKU in an answer would require some other source or a guess, which this contract forbids.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="a-stronger-loss-test-change-only-the-missing-fact">A stronger loss test: change only the missing fact<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#a-stronger-loss-test-change-only-the-missing-fact" class="hash-link" aria-label="Direct link to A stronger loss test: change only the missing fact" title="Direct link to A stronger loss test: change only the missing fact" translate="no">​</a></h3>
<p>We changed only the two purchase destinations in the link fixture, then only the two <code>data-sku</code> values in the attribute fixture. The corresponding source HTML changed, while these derived outputs stayed byte-identical:</p>
<p><img decoding="async" loading="lazy" alt="Counterfactual field loss: changing purchase URLs leaves text identical; changing attribute-only SKUs leaves Markdown and text identical." src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI2MDAiIGhlaWdodD0iNDAwIiB2aWV3Qm94PSIwIDAgNjAwIDQwMCIgcm9sZT0iaW1nIiBhcmlhLWxhYmVsbGVkYnk9InRpdGxlIGRlc2MiPgo8dGl0bGUgaWQ9InRpdGxlIj5EaWZmZXJlbnQgc291cmNlIGZhY3RzIGJlY29tZSBpZGVudGljYWwgY29udmVydGVkIG91dHB1dHM8L3RpdGxlPgo8ZGVzYyBpZD0iZGVzYyI+Q2hhbmdpbmcgb25seSB0aGUgdHdvIHB1cmNoYXNlIFVSTHMgY2hhbmdlcyBIVE1MIGFuZCBNYXJrZG93biBidXQgbGVhdmVzIHRleHQgaWRlbnRpY2FsLiBDaGFuZ2luZyBvbmx5IHRoZSB0d28gYXR0cmlidXRlIFNLVXMgY2hhbmdlcyBIVE1MIGJ1dCBsZWF2ZXMgTWFya2Rvd24gYW5kIHRleHQgaWRlbnRpY2FsLiBUaGVzZSBhcmUgbG9jYWwgcmVmZXJlbmNlIHRyYW5zZm9ybWF0aW9uIHJlc3VsdHMsIG5vdCBtb2RlbCBhY2N1cmFjeS48L2Rlc2M+CjxyZWN0IHdpZHRoPSI2MDAiIGhlaWdodD0iNDAwIiBmaWxsPSIjZmZmIi8+CjxwYXRoIGQ9Ik0yMCAyMEg1ODAiIHN0cm9rZT0iI2Y1YTYyMyIgc3Ryb2tlLXdpZHRoPSI1Ii8+CjxnIGZvbnQtZmFtaWx5PSJBcmlhbCxzYW5zLXNlcmlmIiBmaWxsPSIjMTcxNzE3Ij4KPHRleHQgeD0iMjQiIHk9IjY1IiBmb250LXNpemU9IjI2IiBmb250LXdlaWdodD0iYm9sZCI+Q2hhbmdlIG9ubHkgcHVyY2hhc2UgVVJMczwvdGV4dD4KPHRleHQgeD0iMjQiIHk9IjEwNiIgZm9udC1zaXplPSIyNCI+SFRNTDogZGlmZmVyZW50PC90ZXh0Pjx0ZXh0IHg9IjI0IiB5PSIxNDMiIGZvbnQtc2l6ZT0iMjQiPk1hcmtkb3duOiBkaWZmZXJlbnQ8L3RleHQ+PHRleHQgeD0iMjQiIHk9IjE4MCIgZm9udC1zaXplPSIyNCIgZm9udC13ZWlnaHQ9ImJvbGQiPlRleHQ6IGlkZW50aWNhbDwvdGV4dD4KPHBhdGggZD0iTTI0IDIwM0g1NzYiIHN0cm9rZT0iI2I1YjViNSIvPgo8dGV4dCB4PSIyNCIgeT0iMjQ0IiBmb250LXNpemU9IjI2IiBmb250LXdlaWdodD0iYm9sZCI+Q2hhbmdlIG9ubHkgYXR0cmlidXRlIFNLVXM8L3RleHQ+Cjx0ZXh0IHg9IjI0IiB5PSIyODUiIGZvbnQtc2l6ZT0iMjQiPkhUTUw6IGRpZmZlcmVudDwvdGV4dD48dGV4dCB4PSIyNCIgeT0iMzIyIiBmb250LXNpemU9IjI0IiBmb250LXdlaWdodD0iYm9sZCI+TWFya2Rvd246IGlkZW50aWNhbDwvdGV4dD48dGV4dCB4PSIyNCIgeT0iMzU5IiBmb250LXNpemU9IjI0IiBmb250LXdlaWdodD0iYm9sZCI+VGV4dDogaWRlbnRpY2FsPC90ZXh0Pgo8L2c+PC9zdmc+" width="600" height="400" class="img_ev3q"></p>
<p>For these transforms, two different required answers become indistinguishable to a consumer limited to the lossy representation. That demonstrates information loss without attributing a failure to a model. It does not prove that a model will correctly extract every field from the richer input.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="absolute-token-counts-with-the-field-denominator-beside-them">Absolute token counts, with the field denominator beside them<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#absolute-token-counts-with-the-field-denominator-beside-them" class="hash-link" aria-label="Direct link to Absolute token counts, with the field denominator beside them" title="Direct link to Absolute token counts, with the field denominator beside them" translate="no">​</a></h2>
<p>These counts cover the exact representation content. They exclude the task, schema, source URL, tool envelope and model protocol overhead. UTF-8 bytes, Unicode code points and model tokens are different units. All three selected inputs happen to be ASCII, so byte and code-point counts coincide here; that does not generalize to Unicode pages.</p>
<table><thead><tr><th>Case / format</th><th style="text-align:right">Bytes / code points</th><th style="text-align:right">o200k tokens</th><th style="text-align:right">cl100k tokens</th><th style="text-align:right">Present slots</th></tr></thead><tbody><tr><td>Table / HTML</td><td style="text-align:right">3,688 / 3,688</td><td style="text-align:right">1,522</td><td style="text-align:right">1,520</td><td style="text-align:right">12/12</td></tr><tr><td>Table / Markdown</td><td style="text-align:right">1,504 / 1,504</td><td style="text-align:right">573</td><td style="text-align:right">576</td><td style="text-align:right">12/12</td></tr><tr><td>Table / text</td><td style="text-align:right">1,051 / 1,051</td><td style="text-align:right">415</td><td style="text-align:right">416</td><td style="text-align:right">12/12</td></tr><tr><td>Links / HTML</td><td style="text-align:right">491 / 491</td><td style="text-align:right">173</td><td style="text-align:right">170</td><td style="text-align:right">8/8</td></tr><tr><td>Links / Markdown</td><td style="text-align:right">218 / 218</td><td style="text-align:right">72</td><td style="text-align:right">72</td><td style="text-align:right">8/8</td></tr><tr><td>Links / text</td><td style="text-align:right">174 / 174</td><td style="text-align:right">55</td><td style="text-align:right">57</td><td style="text-align:right">6/8</td></tr><tr><td>Attributes / HTML</td><td style="text-align:right">399 / 399</td><td style="text-align:right">143</td><td style="text-align:right">139</td><td style="text-align:right">8/8</td></tr><tr><td>Attributes / Markdown</td><td style="text-align:right">128 / 128</td><td style="text-align:right">45</td><td style="text-align:right">44</td><td style="text-align:right">6/8</td></tr><tr><td>Attributes / text</td><td style="text-align:right">75 / 75</td><td style="text-align:right">26</td><td style="text-align:right">26</td><td style="text-align:right">4/8</td></tr></tbody></table>
<p>Using o200k content counts, the table's HTML/Markdown ratio is about 2.66 and HTML/text about 3.67. The attribute fixture has the larger HTML/text ratio, 5.50, while text loses four of eight required slots. These are payload ratios, not model-bill discounts or measured extraction success rates.</p>
<p>Five local repeats produced identical bytes for each case/arm. They check transform determinism on these inputs, not repeated model outcomes or independent sites. The downloadable report includes every output hash, slot check, tokenizer vocabulary digest and runtime version.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-the-fixed-model-experiment-returned">What the fixed model experiment returned<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#what-the-fixed-model-experiment-returned" class="hash-link" aria-label="Direct link to What the fixed model experiment returned" title="Direct link to What the fixed model experiment returned" translate="no">​</a></h2>
<p>We froze <code>gpt-4.1-mini-2025-04-14</code> in Chat Completions with temperature <code>0</code>, top_p <code>1</code>, one response, <code>max_completion_tokens=1000</code>, default service tier and <code>store=false</code>. Every arm received the same system instruction, task, six-field schema and source URL, with the full representation as untrusted page data. There were no tools, external retrieval, conversation history, automatic retries or prompt revisions. The oracle never entered the prompt.</p>
<p>The seeded, interleaved plan contained five rounds of the nine same-capture inputs, followed by the 15 saved hosted outputs. Two pre-inference failures consumed their planned jobs; explicitly authorized continuations preserved both failures and skipped them. The remaining 58 requests completed. The missing responses are offline links/Markdown repeat 0 and offline table/text repeat 0. All available responses, including those outside matched comparisons, remain downloadable.</p>
<p>For the fairest comparison within each case, this table uses only repeat labels with inference responses in <strong>all three arms</strong>. “Accepted” means every ordered record and all six fields exactly match the withheld oracle, including required nulls.</p>
<table><thead><tr><th>Input / matched repeats</th><th style="text-align:right">HTML accepted</th><th style="text-align:right">Markdown accepted</th><th style="text-align:right">Text accepted</th></tr></thead><tbody><tr><td>Same-capture table / repeats 1–4</td><td style="text-align:right">4/4</td><td style="text-align:right">0/4</td><td style="text-align:right">4/4</td></tr><tr><td>Same-capture purchase links / repeats 1–4</td><td style="text-align:right">4/4</td><td style="text-align:right">4/4</td><td style="text-align:right">0/4</td></tr><tr><td>Same-capture attribute SKUs / repeats 0–4</td><td style="text-align:right">5/5</td><td style="text-align:right">0/5</td><td style="text-align:right">0/5</td></tr><tr><td>Saved hosted table outputs / repeats 0–4</td><td style="text-align:right">5/5</td><td style="text-align:right">0/5</td><td style="text-align:right">0/5</td></tr></tbody></table>
<p>These denominators are repeated model responses on the selected inputs. Five successes out of five do not establish 100% reliability. The report includes nominal 95% Wilson repeat intervals: 5/5 corresponds to approximately 56.6–100%, and 0/5 to 0–43.4%, under an independent-Bernoulli approximation. Repeated identical inputs may be correlated, so those intervals do not estimate performance across websites. We do not pool these different tasks into a best-format score.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="read-the-failures-not-just-the-percentages">Read the failures, not just the percentages<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#read-the-failures-not-just-the-percentages" class="hash-link" aria-label="Direct link to Read the failures, not just the percentages" title="Direct link to Read the failures, not just the percentages" translate="no">​</a></h3>
<p>Across all 58 inference responses, 28 pages passed and 30 failed the strict contract. Twenty-five failures returned the valid JSON array <code>[]</code>: every table/Markdown response, every links/text and attributes/text response, and every hosted table/text response. The remaining five failures were attribute-case Markdown: both products and their links were returned, but <code>sku</code> was null instead of the source's <code>data-sku</code> value. Those nulls avoid inventing a SKU, yet fail a task that requires it.</p>
<p>The table's Markdown contains every required value; its empty arrays therefore cannot be explained by absent literals alone. We did not vary prompts or converters to identify the cause. Likewise, local browser-selection text passed the table task while hosted raw-mode text returned empty arrays. Their payloads differ in title, hidden row and cell layout, but this run does not isolate which difference caused the result.</p>
<p>The useful decisions are narrower than a format ranking. Test text for a displayed-value task, Markdown for link-sensitive output, and HTML or a deterministic parser for attributes. If complete records fail, investigate the actual payload and prompt before scaling. This run gives no basis for assuming Markdown or text succeeds merely because it uses fewer tokens.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="validate-the-records-then-compare-cost-per-correct-output">Validate the records, then compare cost per correct output<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#validate-the-records-then-compare-cost-per-correct-output" class="hash-link" aria-label="Direct link to Validate the records, then compare cost per correct output" title="Direct link to Validate the records, then compare cost per correct output" translate="no">​</a></h2>
<p>The runnable <code>contract.py</code> checks exact ordered records and field sets. A missing or extra product, reordered or duplicated row, invented destination, or numeric coercion of a displayed price fails the page. Null fields remain part of this strict record check.</p>
<p>This executable example exercises that scorer against the literal oracle, then introduces a wrong destination. It does not call a model:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token keyword" style="color:rgb(86, 156, 214)">from</span><span class="token plain"> copy </span><span class="token keyword" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> deepcopy</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">from</span><span class="token plain"> contract </span><span class="token keyword" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> TRUTH</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> score</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">candidate </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> deepcopy</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">TRUTH</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">"links"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">print</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">score</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"links"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> candidate</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">"accepted_page"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">candidate</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">0</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">"buy_url"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"https://catalog.example/buy/wrong"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">print</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">score</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"links"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> candidate</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">"accepted_page"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><br></div></code></pre></div></div>
<p>Its captured output is:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">True</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">False</span><br></div></code></pre></div></div>
<p>When adapting this experiment, keep the source, prompt, schema, model snapshot, settings and retries fixed. Interleave arms, retain every response and failure, and score against a withheld oracle. Report exact accepted pages as the primary denominator and correct field slots separately.</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">Model cost per accepted page = all model charges / accepted pages</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Retrieval credits per accepted page = all charged credits / accepted pages</span><br></div></code></pre></div></div>
<p>Incorrect, truncated and failed charged attempts stay in the numerator. If no page is accepted, the ratio is undefined. Keep model cash and API credits separate unless you have an explicit subscription allocation. Token counts alone cannot supply the denominator or observed charges.</p>
<p>Here are all available model observations. Costs are calculated from reported usage at the published standard rates, <strong>not billing receipts</strong>. “Known cost” excludes the two failures with unknown charges; the last column includes every attempt in that case/arm only when all costs are known. USD amounts are rounded to eight decimal places.</p>
<table><thead><tr><th>Case / format</th><th style="text-align:right">Inference / attempts</th><th style="text-align:right">Accepted pages</th><th style="text-align:right">Known token-rate cost (USD)</th><th style="text-align:right">All-attempt cost / accepted page (USD)</th></tr></thead><tbody><tr><td>Table / HTML</td><td style="text-align:right">5/5</td><td style="text-align:right">5</td><td style="text-align:right">0.00318920</td><td style="text-align:right">0.00063784</td></tr><tr><td>Table / Markdown</td><td style="text-align:right">5/5</td><td style="text-align:right">0</td><td style="text-align:right">0.00190400</td><td style="text-align:right">Undefined: zero accepted</td></tr><tr><td>Table / text</td><td style="text-align:right">4/5</td><td style="text-align:right">4</td><td style="text-align:right">0.00237920</td><td style="text-align:right">Unknown: one failure's charge</td></tr><tr><td>Links / HTML</td><td style="text-align:right">5/5</td><td style="text-align:right">5</td><td style="text-align:right">0.00208000</td><td style="text-align:right">0.00041600</td></tr><tr><td>Links / Markdown</td><td style="text-align:right">4/5</td><td style="text-align:right">4</td><td style="text-align:right">0.00150240</td><td style="text-align:right">Unknown: one failure's charge</td></tr><tr><td>Links / text</td><td style="text-align:right">5/5</td><td style="text-align:right">0</td><td style="text-align:right">0.00084600</td><td style="text-align:right">Undefined: zero accepted</td></tr><tr><td>Attributes / HTML</td><td style="text-align:right">5/5</td><td style="text-align:right">5</td><td style="text-align:right">0.00206800</td><td style="text-align:right">0.00041360</td></tr><tr><td>Attributes / Markdown</td><td style="text-align:right">5/5</td><td style="text-align:right">0</td><td style="text-align:right">0.00178400</td><td style="text-align:right">Undefined: zero accepted</td></tr><tr><td>Attributes / text</td><td style="text-align:right">5/5</td><td style="text-align:right">0</td><td style="text-align:right">0.00078800</td><td style="text-align:right">Undefined: zero accepted</td></tr><tr><td>Hosted table / HTML</td><td style="text-align:right">5/5</td><td style="text-align:right">5</td><td style="text-align:right">0.00269000</td><td style="text-align:right">0.00053800</td></tr><tr><td>Hosted table / Markdown</td><td style="text-align:right">5/5</td><td style="text-align:right">0</td><td style="text-align:right">0.00190400</td><td style="text-align:right">Undefined: zero accepted</td></tr><tr><td>Hosted table / text</td><td style="text-align:right">5/5</td><td style="text-align:right">0</td><td style="text-align:right">0.00163200</td><td style="text-align:right">Undefined: zero accepted</td></tr></tbody></table>
<p>The 58 responses reported 48,709 input tokens, including 14,976 cached tokens, and 4,860 output tokens. Using $0.40/M uncached input, $0.10/M cached input and $1.60/M output gives $0.0227668 of known token-rate cost. The two other attempts have no reported usage, so total actual cost remains unknown. Their combined $0.0072 allowance reservation is <strong>not observed billing</strong>. Published rates are from the <a href="https://developers.openai.com/api/docs/models/gpt-4.1-mini" target="_blank" rel="noopener noreferrer" class="">official model page</a>, checked October 11, 2026.</p>
<p>Repeated inputs received caching discounts, and a successful full record costs more output tokens than <code>[]</code>. Consequently, these run-specific costs are not isolated effects of format size or forecasts for fresh pages. The <a href="https://scrapingant.com/blog/assets/files/PROTOCOL-75bca3f1068fcfae68a09bf5c5303994.txt" target="_blank" class="">protocol download</a> records the complete plan, usage, retained failures and accounting. The <a href="https://scrapingant.com/blog/assets/files/execution-report-f7415661b24b7bbe0b814d5b91112488.json" target="_blank" class="">execution report</a> separates all-available observations from matched repeats and keeps unknown-charge ratios null.</p>
<p>For a captured model-to-validator workflow, see the <a class="" href="https://scrapingant.com/blog/building-ai-driven-scrapers-in-2025-agents-mcp-and">AI scraper guide</a>.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-the-hosted-mcp-tools-actually-returned">What the hosted MCP tools actually returned<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#what-the-hosted-mcp-tools-actually-returned" class="hash-link" aria-label="Direct link to What the hosted MCP tools actually returned" title="Direct link to What the hosted MCP tools actually returned" translate="no">​</a></h2>
<p>On October 11, 2026, we made five interleaved calls to each tool for the existing static table URL, with <code>browser=false</code>, <code>proxy_type=datacenter</code> and <code>proxy_country=US</code> fixed. There were no retries or browser/proxy fallbacks. Server <code>1.30.0</code> negotiated MCP protocol <code>2025-11-25</code>.</p>
<table><thead><tr><th>Hosted format</th><th style="text-align:right">Attempts</th><th style="text-align:right">Outputs containing all 12 required literals</th><th style="text-align:right">UTF-8 bytes</th><th style="text-align:right">o200k tokens</th></tr></thead><tbody><tr><td>HTML</td><td style="text-align:right">5</td><td style="text-align:right">5</td><td style="text-align:right">3,688</td><td style="text-align:right">1,522</td></tr><tr><td>Markdown</td><td style="text-align:right">5</td><td style="text-align:right">5</td><td style="text-align:right">1,504</td><td style="text-align:right">573</td></tr><tr><td>Text</td><td style="text-align:right">5</td><td style="text-align:right">5</td><td style="text-align:right">1,083</td><td style="text-align:right">436</td></tr></tbody></table>
<p>Each arm returned identical content in its five observations. All five HTML responses matched the frozen catalog bytes. The Markdown matched its local reference, but hosted text <strong>did not match browser selection</strong>: it retained the document title and the CSS-hidden stock row, and placed table cells on separate lines. All required priced-product values remained present. These token counts still cover content only.</p>
<p>That difference matters when choosing a tool: raw-mode text is not evidence of what a browser would visibly select. Inspect the actual payload rather than assuming that “text” removes hidden elements or preserves browser table boundaries. We did not test hosted browser-mode text or the two richer local-only fixtures.</p>
<p>A scoped read-only account-counter snapshot before the run and after every call increased by one each time: 15 credits in total, consistent with the documented schedule. The content-only MCP response supplied no per-request credit receipt, and concurrent account usage cannot be ruled out. We report these contemporaneous deltas separately from documented pricing and model-provider costs.</p>
<p>Each of the 15 saved hosted returns subsequently entered the same fixed-model task, without another acquisition. HTML produced five accepted pages; Markdown and text each produced five empty arrays. For this arm's five accepted HTML pages, the contemporaneous retrieval-counter delta was five credits; that is an account observation, not an exclusive per-request receipt. No accepted page exists for the other hosted arms, so a retrieval-cost-per-accepted-page ratio for them is undefined. The model usage costs above remain separate.</p>
<p>This is a captured MCP-tool retrieval followed by a fixed extraction request. It does not test an interactive agent choosing tools, following links, retrying errors or operating a production pipeline.</p>
<p>Live calls reacquire the URL. Matching HTML and Markdown output hashes do not expose the underlying source bytes of separately acquired Markdown/text calls. Keep this hosted acquisition check separate from the offline comparison in which every arm starts with exactly the same saved source.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="choosing-the-corresponding-scrapingant-mcp-tool">Choosing the corresponding ScrapingAnt MCP tool<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#choosing-the-corresponding-scrapingant-mcp-tool" class="hash-link" aria-label="Direct link to Choosing the corresponding ScrapingAnt MCP tool" title="Direct link to Choosing the corresponding ScrapingAnt MCP tool" translate="no">​</a></h2>
<p>Public, unauthenticated discovery on October 11, 2026 advertised server <code>1.30.0</code>, protocol <code>2025-11-25</code>, and these tools:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">get_web_page_html: url, browser, proxy_type, proxy_country</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">get_web_page_markdown: url, browser, proxy_type, proxy_country</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">get_web_page_text: url, browser, proxy_type, proxy_country</span><br></div></code></pre></div></div>
<p>The runnable discovery script only initialized, sent the initialized notification and listed tools. It read no credentials and called no retrieval tool. Full schemas and exchanges are in the <a href="https://scrapingant.com/blog/assets/files/public-discovery-741b1e5b6a9bfec7ddc7e38514294f43.json" target="_blank" class="">discovery JSON</a>.</p>
<p>After configuring a host using the <a href="https://docs.scrapingant.com/mcp-server" target="_blank" rel="noopener noreferrer" class="">MCP documentation</a>, select a tool by the required fields. The following is an <strong>illustrative tool-call request, not an executed retrieval capture</strong>. An initialized MCP client can send it with its configured API key:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"jsonrpc"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"2.0"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"id"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">3</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"method"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"tools/call"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"params"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"name"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"get_web_page_markdown"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"arguments"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"url"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"https://scrapingant.github.io/scrapingant-examples/fixtures/html-tables.html"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"browser"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token boolean">false</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"proxy_type"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"datacenter"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"proxy_country"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"US"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><br></div></code></pre></div></div>
<p>Use <code>get_web_page_html</code> when you need attributes, or test <code>get_web_page_text</code> when its actual text payload meets your contract. A real MCP call reacquires the URL, so check source drift and retrieval failures separately from same-capture format comparison. Transport success is insufficient evidence of usable content.</p>
<p>The observed MCP schema has no <code>js_snippet</code>, <code>wait_for_selector</code> or extraction-schema parameter. Those direct-API controls should not be pasted into this tool request. The separate structured JSON extractor is not one of the three observed MCP tools. Rendering and proxy choices also change acquisition behavior; changing them while comparing formats confounds the result.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">What Is an MCP Server? Architecture, Transports and a Working Example</h3><p class="description_eu34">What an MCP server is, how host, client and server talk over JSON-RPC, the two transports, a real handshake transcript, and a 23-line server of your own.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/what-is-mcp-server">→ Read Article</a><span class="readTime_W_dz">9<!-- --> min</span></div></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="try-the-format-choice-on-a-workload-worth-running">Try the format choice on a workload worth running<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#try-the-format-choice-on-a-workload-worth-running" class="hash-link" aria-label="Direct link to Try the format choice on a workload worth running" title="Direct link to Try the format choice on a workload worth running" translate="no">​</a></h2>
<p>Saved HTML and adequate target APIs or feeds may already satisfy your task. ScrapingAnt is relevant when you want managed URL retrieval, browser rendering or proxy selection alongside the three output choices. This fixture study does not measure production reliability or the operating effort those capabilities might save.</p>
<p><a href="https://app.scrapingant.com/signup" target="_blank" rel="noopener noreferrer" class="">Try these formats on one of your pages</a>, then use the <a href="https://docs.scrapingant.com/mcp-server" target="_blank" rel="noopener noreferrer" class="">MCP setup documentation</a>. ScrapingAnt's verified signup offer includes 10,000 free API credits monthly with no credit card required. Start with one permitted URL, specify the fields, inspect actual returned content, and validate complete records before expanding the job.</p>
<p>MCP requests use the normal retrieval credit schedule. For a non-Google URL with <code>browser=false</code> and a datacenter proxy, the documented cost is one API credit per request; our account-counter observations were consistent with that amount. These are separate forms of evidence; account deltas do not establish exclusive per-request charges. Model-provider usage is additional. Evaluate your successful workload, failure/retry costs and the <a href="https://scrapingant.com/#pricing" target="_blank" rel="noopener noreferrer" class="">current plans</a> before choosing a paid subscription; no paid price or universal savings guarantee is inferred here.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="reproduce-and-adapt-the-evidence">Reproduce and adapt the evidence<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#reproduce-and-adapt-the-evidence" class="hash-link" aria-label="Direct link to Reproduce and adapt the evidence" title="Direct link to Reproduce and adapt the evidence" translate="no">​</a></h2>
<p>Download the <a href="https://scrapingant.com/blog/assets/files/reproduction-85c39c31e89da732f77121dd528721c1.zip" target="_blank" class="">complete reproduction ZIP</a>, <a href="https://scrapingant.com/blog/assets/files/summary-5568a6e66bd26597a25e0f3f6a5ab7ba.csv" target="_blank" class="">CSV</a>, <a href="https://scrapingant.com/blog/assets/files/report-786df53928bf9f15499c52028d0d74cc.json" target="_blank" class="">full report</a>, or <a href="https://scrapingant.com/blog/assets/files/ground_truth-4ad7ed7a03373a75cf066c18a376d5a5.json" target="_blank" class="">literal oracle</a>.</p>
<p>With Python 3.12 installed, unpack the ZIP and run from its example directory:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3.12 -m venv .venv</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">. .venv/bin/activate</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python -m pip install -r requirements.txt</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python -m playwright install chromium</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">./run.sh</span><br></div></code></pre></div></div>
<p>Installation and the first tokenizer-cache load need internet access. After those prerequisites, the replay uses local files and blocks page requests. It makes no paid retrieval/model calls even if credentials are configured, and compares fresh outputs to the saved evidence without overwriting it.</p>
<p>Change your own oracle when changing your task, then record a new observation separately. Do not update expected results merely to make a failing validation pass. Preserve failed output and distinguish absent information from a model mistake.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="limits-and-common-questions">Limits and common questions<a href="https://scrapingant.com/blog/html-markdown-plain-text-llm-extraction#limits-and-common-questions" class="hash-link" aria-label="Direct link to Limits and common questions" title="Direct link to Limits and common questions" translate="no">​</a></h2>
<p><strong>Is Markdown better than HTML for LLM extraction?</strong> It retains the purchase links we tested and produced four accepted link-case responses. It loses attribute-only SKUs and returned empty arrays for our table task. Choose using your required fields and measured records, not a universal Markdown preference.</p>
<p><strong>Does Markdown remove page noise?</strong> ScrapingAnt's documented conversion removes script/noscript elements; navigation, footers, cookie banners and advertising can remain. The link fixture's captured Markdown includes its footer. See the <a href="https://docs.scrapingant.com/llm-markdown" target="_blank" rel="noopener noreferrer" class="">conversion reference</a>.</p>
<p><strong>What about tables, code and images?</strong> Inspect the conversion output. The documented converter emits ordinary tables as pipe-separated rows, preformatted code as indented text, and images with alt text/source. That does not prove preservation of merged-cell relationships, code language metadata or image meaning. This study measured a simple table and catalog fields, not those richer tasks.</p>
<p><strong>Can text or Markdown make an unsafe source safe?</strong> Treat every representation as untrusted page data. Format conversion is not an instruction-safety boundary. The fixed prompt tells the model to treat page content as data, but this study does not measure resistance to hostile instructions.</p>
<p><strong>Does this prove a production winner?</strong> No. There are seven unique fixture records, one original catalog capture and two selected local cases. The hosted check makes 15 requests to one static URL. The same execution session created the local cases/oracle. Different converters, browser state, target content, prompts, model versions or tokenizers can change results. We measured repeated fixed-model outcomes, but unknown charges prevent a total all-attempt model cost. Production reliability, prompt robustness and paying-customer impact remain unmeasured.</p>
<p>Study evidence recorded on <strong>2026-10-11</strong> with model <strong>gpt-4.1-mini-2025-04-14</strong>, Python <strong>3.12.11</strong>, BeautifulSoup <strong>4.12.2</strong>, html2text <strong>2020.1.16</strong>, tiktoken <strong>0.11.0</strong>, Playwright <strong>1.62.0</strong> and Chromium <strong>151.0.7922.34</strong>. Code and source provenance are included in the reproduction ZIP.</p>
<p><em>AI agents produced the local fixtures, transformations, authenticated MCP captures, fixed-model measurements and draft. Oleg Kulyk is the publication owner and reviewed the methodology, conclusions and byline before publication.</em></p>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="web scraping" term="web scraping"/>
        <category label="data extraction" term="data extraction"/>
        <category label="mcp tools" term="mcp tools"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Requests vs Playwright vs ScrapingAnt: Cost per Valid Record]]></title>
        <id>https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark</id>
        <link href="https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark"/>
        <updated>2026-10-09T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Compare Requests, Playwright and ScrapingAnt on controlled fixtures: validated output, measured latency, actual API credits and reproducible Python code.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Requests vs Playwright vs ScrapingAnt: Cost per Valid Record" src="https://scrapingant.com/blog/assets/images/requests-playwright-scrapingant-benchmark-32b8508fd111071b218c6656d85de00f.png" width="1200" height="630" class="img_ev3q"></p>
<p>Choose a scraping method by the records it delivers. A fast response containing a loading placeholder cannot satisfy a price-monitoring job. A browser may add unnecessary work when the required table is already in the HTML.</p>
<p>The recorded benchmark compared Python Requests, direct Playwright and both ScrapingAnt modes against two controlled, already published fixtures. All four extracted the static catalog correctly. Only the browser arms passed the delayed JavaScript diagnostic. For the catalog, ScrapingAnt raw delivered the same validated records for <strong>90% fewer API credits than ScrapingAnt rendered</strong>: <strong>0.3333 credits per accepted record observation</strong>, versus <strong>3.3333 credits</strong> with rendering. Direct Requests and Playwright had lower observed latency; their total operating cost was not measured.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-measured-results-output-before-speed">The measured results: output before speed<a href="https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark#the-measured-results-output-before-speed" class="hash-link" aria-label="Direct link to The measured results: output before speed" title="Direct link to The measured results: output before speed" translate="no">​</a></h2>
<p>These are public-origin results from October 9, 2026: ten randomized, interleaved attempts per target and arm, no retries, 80 acquisitions in total. The 40 ScrapingAnt calls consumed exactly 220 credits according to their response receipts. This is a small fixture study, not evidence of production reliability or a universal cost winner.</p>
<p>The static catalog has three product records with four required fields. Each arm accepted all ten attempts: thirty record observations, repeating the same three unique products.</p>
<section class="panel_FO7_" aria-labelledby="benchmark-static-title"><h3 id="benchmark-static-title">Static catalog</h3><p class="scope_Dlak">Observed October 9, 2026 · Existing public-origin fixtures · 10 attempts per arm · No retries</p><div class="charts_dC55"><figure><figcaption>Exact contract accepted</figcaption><div class="barRow_e0FC"><span>Requests</span><span class="track_hO9M" aria-hidden="true"><span class="fill_H80q" style="width:100%"></span></span><strong>10<!-- -->/<!-- -->10</strong></div><div class="barRow_e0FC"><span>Playwright</span><span class="track_hO9M" aria-hidden="true"><span class="fill_H80q" style="width:100%"></span></span><strong>10<!-- -->/<!-- -->10</strong></div><div class="barRow_e0FC"><span>ScrapingAnt raw</span><span class="track_hO9M" aria-hidden="true"><span class="fill_H80q" style="width:100%"></span></span><strong>10<!-- -->/<!-- -->10</strong></div><div class="barRow_e0FC"><span>ScrapingAnt rendered</span><span class="track_hO9M" aria-hidden="true"><span class="fill_H80q" style="width:100%"></span></span><strong>10<!-- -->/<!-- -->10</strong></div></figure><figure><figcaption>Median of valid attempts</figcaption><div class="barRow_e0FC"><span>Requests</span><span class="track_hO9M" aria-hidden="true"><span class="latencyFill_HBo5" style="width:4.189649699140656%"></span></span><strong>55.77 ms</strong></div><div class="barRow_e0FC"><span>Playwright</span><span class="track_hO9M" aria-hidden="true"><span class="latencyFill_HBo5" style="width:17.82344850856792%"></span></span><strong>237.26 ms</strong></div><div class="barRow_e0FC"><span>ScrapingAnt raw</span><span class="track_hO9M" aria-hidden="true"><span class="latencyFill_HBo5" style="width:52.00609325200749%"></span></span><strong>692.28 ms</strong></div><div class="barRow_e0FC"><span>ScrapingAnt rendered</span><span class="track_hO9M" aria-hidden="true"><span class="latencyFill_HBo5" style="width:100%"></span></span><strong>1331.14 ms</strong></div><p class="chartNote_Q7Zp">Linear scale from 0 to <!-- -->1331.14 ms<!-- -->. All-attempt times and observed ranges are below.</p></figure></div><p id="benchmark-static-scroll" class="tableHint_FSDY">Every attempt remains in the accounting. On small screens, scroll the table horizontally.</p><div class="tableScroll_sgfb" tabindex="0" role="region" aria-label="Catalog result table" aria-describedby="benchmark-static-scroll"><table><caption>Commercial record contract<!-- -->; targets are not pooled</caption><thead><tr><th scope="col">Arm</th><th scope="col">Valid / attempts</th><th scope="col">Accepted observations</th><th scope="col">All-attempt latency</th><th scope="col">Actual API credits</th></tr></thead><tbody><tr><th scope="row">Requests</th><td>10<!-- --> / <!-- -->10</td><td>30<!-- --> <!-- -->catalog records</td><td>Median <!-- -->55.77 ms<small>Range <!-- -->52.95–61.32<!-- --> ms</small></td><td>No API charge<small>Total USD unmeasured</small></td></tr><tr><th scope="row">Playwright</th><td>10<!-- --> / <!-- -->10</td><td>30<!-- --> <!-- -->catalog records</td><td>Median <!-- -->237.26 ms<small>Range <!-- -->219.49–277.16<!-- --> ms</small></td><td>No API charge<small>Total USD unmeasured</small></td></tr><tr><th scope="row">ScrapingAnt raw</th><td>10<!-- --> / <!-- -->10</td><td>30<!-- --> <!-- -->catalog records</td><td>Median <!-- -->692.28 ms<small>Range <!-- -->648.56–785.27<!-- --> ms</small></td><td>10<!-- --> total<small>0.3333 per accepted observation</small></td></tr><tr><th scope="row">ScrapingAnt rendered</th><td>10<!-- --> / <!-- -->10</td><td>30<!-- --> <!-- -->catalog records</td><td>Median <!-- -->1331.14 ms<small>Range <!-- -->1193.03–1414.95<!-- --> ms</small></td><td>100<!-- --> total<small>3.3333 per accepted observation</small></td></tr></tbody></table></div><p class="limit_K8ah">30 accepted observations per arm repeat the same 3 unique records.<!-- --> Repeated fixture attempts do not estimate production reliability.</p></section>
<p>Requests had the lowest observed median time for this complete HTML table. Within ScrapingAnt, raw retrieval satisfied the same contract at one tenth of the rendered arm's API credit cost. Rendering bought no additional accepted catalog data in this sample.</p>
<p>The second fixture changes a text element after a configured 1500 ms delay and creates a readiness marker. It is a <strong>DOM readiness diagnostic</strong>, not a product catalog. Ten accepted sentinel observations are not ten extracted commercial records.</p>
<section class="panel_FO7_" aria-labelledby="benchmark-delayed-title"><h3 id="benchmark-delayed-title">Delayed DOM diagnostic</h3><p class="scope_Dlak">Observed October 9, 2026 · Existing public-origin fixtures · 10 attempts per arm · No retries</p><div class="charts_dC55"><figure><figcaption>Exact contract accepted</figcaption><div class="barRow_e0FC"><span>Requests</span><span class="track_hO9M" aria-hidden="true"><span class="fill_H80q" style="width:0%"></span></span><strong>0<!-- -->/<!-- -->10</strong></div><div class="barRow_e0FC"><span>Playwright</span><span class="track_hO9M" aria-hidden="true"><span class="fill_H80q" style="width:100%"></span></span><strong>10<!-- -->/<!-- -->10</strong></div><div class="barRow_e0FC"><span>ScrapingAnt raw</span><span class="track_hO9M" aria-hidden="true"><span class="fill_H80q" style="width:0%"></span></span><strong>0<!-- -->/<!-- -->10</strong></div><div class="barRow_e0FC"><span>ScrapingAnt rendered</span><span class="track_hO9M" aria-hidden="true"><span class="fill_H80q" style="width:100%"></span></span><strong>10<!-- -->/<!-- -->10</strong></div></figure><figure><figcaption>Median of valid attempts</figcaption><div class="barRow_e0FC"><span>Requests</span><span class="track_hO9M" aria-hidden="true"></span><strong>No valid output</strong></div><div class="barRow_e0FC"><span>Playwright</span><span class="track_hO9M" aria-hidden="true"><span class="latencyFill_HBo5" style="width:65.30029786788243%"></span></span><strong>2028.79 ms</strong></div><div class="barRow_e0FC"><span>ScrapingAnt raw</span><span class="track_hO9M" aria-hidden="true"></span><strong>No valid output</strong></div><div class="barRow_e0FC"><span>ScrapingAnt rendered</span><span class="track_hO9M" aria-hidden="true"><span class="latencyFill_HBo5" style="width:100%"></span></span><strong>3106.86 ms</strong></div><p class="chartNote_Q7Zp">Linear scale from 0 to <!-- -->3106.86 ms<!-- -->. All-attempt times and observed ranges are below.</p></figure></div><p id="benchmark-delayed-scroll" class="tableHint_FSDY">Every attempt remains in the accounting. On small screens, scroll the table horizontally.</p><div class="tableScroll_sgfb" tabindex="0" role="region" aria-label="Diagnostic result table" aria-describedby="benchmark-delayed-scroll"><table><caption>Readiness diagnostic contract<!-- -->; targets are not pooled</caption><thead><tr><th scope="col">Arm</th><th scope="col">Valid / attempts</th><th scope="col">Accepted observations</th><th scope="col">All-attempt latency</th><th scope="col">Actual API credits</th></tr></thead><tbody><tr><th scope="row">Requests</th><td>0<!-- --> / <!-- -->10</td><td>0<!-- --> <!-- -->diagnostic sentinels</td><td>Median <!-- -->51.90 ms<small>Range <!-- -->44.31–148.62<!-- --> ms</small><small>All outputs invalid</small></td><td>No API charge<small>Total USD unmeasured</small></td></tr><tr><th scope="row">Playwright</th><td>10<!-- --> / <!-- -->10</td><td>10<!-- --> <!-- -->diagnostic sentinels</td><td>Median <!-- -->2028.79 ms<small>Range <!-- -->2021.13–2173.07<!-- --> ms</small></td><td>No API charge<small>Total USD unmeasured</small></td></tr><tr><th scope="row">ScrapingAnt raw</th><td>0<!-- --> / <!-- -->10</td><td>0<!-- --> <!-- -->diagnostic sentinels</td><td>Median <!-- -->695.07 ms<small>Range <!-- -->643.28–1566.77<!-- --> ms</small><small>All outputs invalid</small></td><td>10<!-- --> total<small>Undefined per accepted observation</small></td></tr><tr><th scope="row">ScrapingAnt rendered</th><td>10<!-- --> / <!-- -->10</td><td>10<!-- --> <!-- -->diagnostic sentinels</td><td>Median <!-- -->3106.86 ms<small>Range <!-- -->3013.67–6754.54<!-- --> ms</small></td><td>100<!-- --> total<small>10 per accepted observation</small></td></tr></tbody></table></div><p class="limit_K8ah">Each successful browser arm repeats one text sentinel 10 times. This is not a commercial product-record sample.<!-- --> Repeated fixture attempts do not estimate production reliability.</p></section>
<p>Both raw arms returned HTTP 200 but kept the initial <code>Web Scraping is hard</code> text, so all their diagnostic outputs failed the contract. ScrapingAnt raw still incurred ten credits. Its cost per accepted sentinel is <strong>undefined</strong>, because it accepted none. The fast raw responses do not replace valid output.</p>
<p>Direct Playwright and ScrapingAnt with <code>browser=true</code> and <code>wait_for_selector='#loaded'</code> passed all ten diagnostic attempts. Their observed medians were 2028.79 ms and 3106.86 ms respectively, including the fixture's intentional delay. The rendered API range was 3013.67–6754.54 ms; the slowest attempt remains in the dataset. No attempt timed out and no retry was made.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="which-approach-should-you-start-with">Which approach should you start with?<a href="https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark#which-approach-should-you-start-with" class="hash-link" aria-label="Direct link to Which approach should you start with?" title="Direct link to Which approach should you start with?" translate="no">​</a></h2>
<table><thead><tr><th>Your extraction contract</th><th>Starting approach</th><th>What to verify</th></tr></thead><tbody><tr><td>All required fields are already in response HTML</td><td>Requests, or ScrapingAnt raw when managed retrieval fits your deployment</td><td>Exact values, complete records, target status and total cost</td></tr><tr><td>Required data is created by JavaScript</td><td>Direct Playwright, or ScrapingAnt rendered with a readiness selector</td><td>The page-created signal, then full field and completeness validation</td></tr><tr><td>You need scripted interactions or browser-context control</td><td>Direct Playwright</td><td>The state and steps required by your application</td></tr><tr><td>You want raw and rendered retrieval through one managed API</td><td>Evaluate both ScrapingAnt modes on your own permitted target</td><td>Accepted output, actual credit receipts and your operational cost model</td></tr></tbody></table>
<p>Start with the simplest acquisition that passes the validator. Switch to rendering when raw HTML lacks required data, and keep the same validator after the switch. This benchmark shows that rule on controlled pages; it does not settle the cost of running your application.</p>
<p>For HTTP client choices, see <a class="" href="https://scrapingant.com/blog/requests-vs-httpx">Requests vs HTTPX</a>. For a worked missing-JavaScript diagnosis, use the <a class="" href="https://scrapingant.com/blog/scrape-dynamic-website-with-python">dynamic Python scraping tutorial</a>.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="define-a-valid-record-before-making-requests">Define a valid record before making requests<a href="https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark#define-a-valid-record-before-making-requests" class="hash-link" aria-label="Direct link to Define a valid record before making requests" title="Direct link to Define a valid record before making requests" translate="no">​</a></h2>
<p>For the catalog, success means exactly these ordered records. All four displayed cells must match. A missing row, duplicate table, extra row or wrong value fails the entire page contract.</p>
<table><thead><tr><th>Product</th><th>Displayed price</th><th>Units sold</th><th>Share</th></tr></thead><tbody><tr><td>Ant Farm Deluxe</td><td>$1,234.50</td><td>12,345</td><td>45.5%</td></tr><tr><td>Ant Farm Mini</td><td>$99.00</td><td>1,020</td><td>3.8%</td></tr><tr><td>Magnifier</td><td>$12.25</td><td>987,654</td><td>50.7%</td></tr></tbody></table>
<p>The diagnostic requires exactly one <code>#test</code> element with <code>I ❤️ ScrapingAnt (after 1500 ms)</code> and exactly one page-created <code>#loaded</code> marker. Original text and marker absence fail. The literal oracle in <code>contract.py</code> is written independently of the fixture code.</p>
<p>HTTP 200 alone is insufficient. Every arm must also supply target status 200 and pass the full output contract. An API result with missing target-status evidence would be unaccepted under this protocol. All captured API responses supplied the required status header. A selector tells us when to inspect the DOM; its appearance does not establish completeness.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">Find Elements with Playwright in Python: Extract Validated Records</h3><p class="description_eu34">Find one or many elements with Python Playwright locators, wait for complete records, and extract text, attributes and JSON with tested examples.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/playwright-find-elements">→ Read Article</a><span class="readTime_W_dz">11<!-- --> min</span></div></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="four-methods-one-parser-and-validator">Four methods, one parser and validator<a href="https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark#four-methods-one-parser-and-validator" class="hash-link" aria-label="Direct link to Four methods, one parser and validator" title="Direct link to Four methods, one parser and validator" translate="no">​</a></h2>
<table><thead><tr><th>Arm</th><th>Retrieval</th><th>Readiness</th></tr></thead><tbody><tr><td>Python Requests</td><td>One GET, no configured proxy or browser</td><td>Parse response HTML</td></tr><tr><td>Direct Playwright</td><td>Bundled Chromium, reused browser, fresh context and page per attempt</td><td><code>DOMContentLoaded</code>, then attached contract selector</td></tr><tr><td>ScrapingAnt raw</td><td><code>/v2/general</code>, <code>browser=false</code>, datacenter proxy, <code>timeout=10</code></td><td>Parse returned HTML; no selector parameter</td></tr><tr><td>ScrapingAnt rendered</td><td>Same endpoint, proxy type and timeout; <code>browser=true</code>, <code>return_page_source=false</code></td><td><code>wait_for_selector</code> uses the same contract selector as Playwright</td></tr></tbody></table>
<p>With <code>browser=true</code>, ScrapingAnt renders the target page with JavaScript and returns its HTML. With <code>browser=true</code>, <code>wait_for_selector</code> tells ScrapingAnt to wait for the specified DOM element to appear. See the <a href="https://docs.scrapingant.com/headless-browser" target="_blank" rel="noopener noreferrer" class="">rendering documentation</a> and <a href="https://docs.scrapingant.com/wait-for-selector" target="_blank" rel="noopener noreferrer" class="">selector parameter</a>.</p>
<p>The rendered arm explicitly sets <code>return_page_source=false</code>. The documented browser-enabled page-source mode costs two credits through a datacenter proxy and does not render JavaScript. It is a separate mode outside this four-arm comparison. See the <a href="https://docs.scrapingant.com/request-response-format" target="_blank" rel="noopener noreferrer" class="">request parameter reference</a> and <a href="https://docs.scrapingant.com/credits-cost" target="_blank" rel="noopener noreferrer" class="">credit schedule</a>.</p>
<p>The Playwright acquisition uses the remaining readiness budget for navigation and its selector wait:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">response </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">goto</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">url</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> wait_until</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token string" style="color:rgb(206, 145, 120)">'domcontentloaded'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> timeout</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain">remaining</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">row</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">'status'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> row</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">'target_status'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> response</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">status </span><span class="token keyword" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> response </span><span class="token keyword" style="color:rgb(86, 156, 214)">else</span><span class="token plain"> </span><span class="token boolean">None</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> row</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">'status'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">==</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">200</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">wait_for_selector</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">TARGETS</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token plain">target</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">'selector'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> state</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token string" style="color:rgb(206, 145, 120)">'attached'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> timeout</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain">remaining</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">body </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">content</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">encode</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'utf-8'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><br></div></code></pre></div></div>
<p>The managed parameters are built by <code>api_parameters()</code>:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">params </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token builtin" style="color:rgb(86, 156, 214)">dict</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">url</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain">url</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> browser</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token string" style="color:rgb(206, 145, 120)">'true'</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> arm </span><span class="token operator" style="color:rgb(212, 212, 212)">==</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">'api_rendered'</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">else</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">'false'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">              proxy_type</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token string" style="color:rgb(206, 145, 120)">'datacenter'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> timeout</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token number" style="color:rgb(181, 206, 168)">10</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> arm </span><span class="token operator" style="color:rgb(212, 212, 212)">==</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">'api_rendered'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    params</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">update</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">return_page_source</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token string" style="color:rgb(206, 145, 120)">'false'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> wait_for_selector</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain">TARGETS</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token plain">target</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">'selector'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><br></div></code></pre></div></div>
<p>These are excerpts from the runnable harness, not standalone scripts. Every acquisition feeds the same parser and literal assertions. No hidden JSON endpoint, resource blocking, stealth configuration or application retry was added. Requests and the API client disable redirects; Playwright retains normal browser navigation. These fixed fixture URLs returned 200 without redirects in the source checks.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="api-credits-per-valid-result">API credits per valid result<a href="https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark#api-credits-per-valid-result" class="hash-link" aria-label="Direct link to API credits per valid result" title="Direct link to API credits per valid result" translate="no">​</a></h2>
<p>For one target and one record contract:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">API credits per accepted record observation</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  = all attempted requests' actual credit receipts</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    / accepted record observations</span><br></div></code></pre></div></div>
<p>Count charged invalid outputs in the numerator. A missing receipt makes actual billed cost unknown. Zero accepted records makes the ratio undefined. Neither case becomes “free.” Repeated observations count extraction work; unique records count distinct data. Report both, and keep catalog records separate from diagnostic sentinels.</p>
<section class="panel_FO7_" aria-labelledby="credit-comparison-title"><h3 id="credit-comparison-title">Receipt-based API cost</h3><p class="scope_Dlak">Actual Ant-credits-cost receipts · 10 attempts per arm and target · Units kept separate</p><div class="charts_dC55"><figure><figcaption>Catalog: credits per accepted record observation</figcaption><div class="barRow_e0FC"><span>ScrapingAnt raw</span><span class="track_hO9M" aria-hidden="true"><span class="fill_H80q" style="width:9.999999999999998%"></span></span><strong>0.3333</strong></div><div class="barRow_e0FC"><span>ScrapingAnt rendered</span><span class="track_hO9M" aria-hidden="true"><span class="fill_H80q" style="width:100%"></span></span><strong>3.3333</strong></div><p class="chartNote_Q7Zp">Linear scale 0–<!-- -->3.3333<!-- -->. <!-- -->Raw: 10 credits / 30 observations. Rendered: 100 / 30.</p></figure><figure><figcaption>Diagnostic: credits per accepted sentinel</figcaption><div class="barRow_e0FC"><span>ScrapingAnt raw</span><span class="track_hO9M" aria-hidden="true"></span><strong>Undefined</strong></div><div class="barRow_e0FC"><span>ScrapingAnt rendered</span><span class="track_hO9M" aria-hidden="true"><span class="fill_H80q" style="width:100%"></span></span><strong>10</strong></div><p class="chartNote_Q7Zp">Linear scale 0–<!-- -->10<!-- -->. <!-- -->Raw spent 10 credits with no accepted result. Rendered: 100 credits / 10 sentinels.</p></figure></div><p class="limit_K8ah">Undefined means zero accepted outputs, not zero cost. These are API credits, not dollars or measured total operating costs.</p></section>
<p>For the catalog, the raw calculation is <code>10 / 30 = 0.3333</code> credits per record observation; rendered is <code>100 / 30 = 3.3333</code>. On the diagnostic, raw is <code>10 / 0</code>, which is undefined; rendered is <code>100 / 10 = 10</code> credits per sentinel. That last number measures this readiness diagnostic, not the cost of extracting a commercial product.</p>
<p>For these ordinary non-Google targets, the documented rates are one credit for raw datacenter retrieval and ten for JavaScript rendering through a datacenter proxy. Every captured <code>Ant-credits-cost</code> receipt matched the arm's expected rate. List pricing planned the budget; actual receipts supply the measured numerator. <a href="https://docs.scrapingant.com/credits-cost" target="_blank" rel="noopener noreferrer" class="">Credit reference</a>.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="credits-are-not-total-cash-cost">Credits are not total cash cost<a href="https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark#credits-are-not-total-cash-cost" class="hash-link" aria-label="Direct link to Credits are not total cash cost" title="Direct link to Credits are not total cash cost" translate="no">​</a></h3>
<p>Requests and direct Playwright incur no ScrapingAnt API charge. Their compute, proxy, setup and maintenance costs were not measured. A subscription's allocated dollar cost also depends on your plan and utilization. This evidence therefore cannot declare a total-cost winner across all four methods.</p>
<p>Use a separate model with your own assumptions:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">total cost per accepted observation</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  = (allocated subscription + compute + additional proxies</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">     + amortized setup and maintenance labor)</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    / accepted observations</span><br></div></code></pre></div></div>
<p>Supply plan utilization, host/proxy rates, labor hours and an amortization period. Avoid charging for managed browser/proxy work again when it is already included in the API allocation. No labor-saving figure or break-even point is measured here.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="reproduce-and-inspect-the-evidence">Reproduce and inspect the evidence<a href="https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark#reproduce-and-inspect-the-evidence" class="hash-link" aria-label="Direct link to Reproduce and inspect the evidence" title="Direct link to Reproduce and inspect the evidence" translate="no">​</a></h2>
<p>Use the <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/4d2ecb4076e2135a8240d3497d32765f20202810/examples/http-vs-browser-cost" target="_blank" rel="noopener noreferrer" class="">immutable source and evidence packet</a>, or download the <a href="https://scrapingant.com/blog/assets/files/evidence-packet-70c45679489c4263eaf69bd660d264ad.zip" target="_blank" class="">complete packet (ZIP)</a>. It includes pinned dependencies, fixture copies, the harness, regression tests, raw HTML captures and offline reports. The original local acquisition revision was <code>05cbdf3080507b8f3f2d41ad65bb3ddd42b5d587</code>; the published runtime files are byte-identical, and their source hashes are embedded in the public-origin report.</p>
<p>From its <code>examples/http-vs-browser-cost/</code> directory, on a POSIX system with Python 3.12:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3.12 -m venv .venv</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">source .venv/bin/activate</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python -m pip install -r requirements.txt</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python -m playwright install chromium</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">./run.sh</span><br></div></code></pre></div></div>
<p>Browser and package downloads require network access. On a fresh Linux host, follow Playwright's <a href="https://playwright.dev/python/docs/browsers#install-system-dependencies" target="_blank" rel="noopener noreferrer" class="">system dependency installation instructions</a>. The captured environment here is macOS.</p>
<p>The default command runs regression tests and forty <strong>loopback</strong> acquisitions. It never reads an API key or calls ScrapingAnt. It refuses to overwrite an output directory. Use a fresh <code>--out</code> path for another run:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python runner.py --out run_output/another-local-run</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python verify_capture.py expected_output/local/report.json</span><br></div></code></pre></div></div>
<p>To verify the captured four-arm evidence without network requests or credits:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python verify_capture.py expected_output/live/report.json</span><br></div></code></pre></div></div>
<p>The strict verifier requires complete live provenance maps, expected completed receipts and a total within the approved ceiling. It then checks the randomized schedule, all initiated attempts, raw-body and runtime-source hashes, contract decisions, credit totals and recomputed summary. File hashes protect artifact integrity; they do not authenticate provider billing. Legacy local captures without runtime hashes are explicitly labeled. Interrupted work remains as unfinished or aborted with uncertain cost, rather than disappearing from the denominator.</p>
<p>Inspect the <a href="https://scrapingant.com/blog/assets/files/live-results-91a4878b31f78d83e8d1cd861c49be56.json" target="_blank" class="">complete public-origin report (JSON)</a>, <a href="https://scrapingant.com/blog/assets/files/live-attempts-44acd894fccaac6b96da2461dbcf97d6.csv" target="_blank" class="">all eighty attempts (CSV)</a> and <a href="https://scrapingant.com/blog/assets/files/live-fixture-check-30e9267f4bc51bae61d87dc5ff375ccf.json" target="_blank" class="">source/public fixture checks (JSON)</a>. Empty credit fields on direct arms mean no API receipt, not zero-dollar operating cost.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="a-new-live-run-requires-a-bounded-budget">A new live run requires a bounded budget<a href="https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark#a-new-live-run-requires-a-bounded-budget" class="hash-link" aria-label="Direct link to A new live run requires a bounded budget" title="Direct link to A new live run requires a bounded budget" translate="no">​</a></h3>
<p>The existing owned targets are the <a href="https://scrapingant.github.io/scrapingant-examples/fixtures/html-tables.html" target="_blank" rel="noopener noreferrer" class="">static catalog</a> and <a href="https://scrapingant.github.io/scrapingant-examples/fixtures/dynamic-delayed.html" target="_blank" rel="noopener noreferrer" class="">delayed DOM fixture</a>. The runner checks that both return HTTP 200 and match tracked source bytes before any API work. The October 9 checks passed. Source drift stops the run for review.</p>
<p>The recorded run used prior owner approval for at most <strong>40 API calls and 220 credits</strong>, with no pilot, paid warmup or retry. Every new live run needs its own explicit budget approval. Configure <code>SCRAPINGANT_API_KEY</code> securely in the process environment; the harness sends it only in the fixed service endpoint's authentication header. Do not place a key in command arguments or target URLs.</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python runner.py --live --repeats 10 --approved-credits 220 --out run_output/live</span><br></div></code></pre></div></div>
<p>This command reproduces the design; the captured execution used the distinct output directory <code>run_output/live-approved-20261009</code>. The flag guards spending but does not grant permission. The runner reserves documented cost before each API call and stops on a missing or unexpected receipt. It permits the expected arm charge, or zero with an API error status. Interrupted calls require billing reconciliation before further paid work. No subscription purchase, residential proxy, model request or new endpoint is required.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="when-scrapingant-fits-the-task">When ScrapingAnt fits the task<a href="https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark#when-scrapingant-fits-the-task" class="hash-link" aria-label="Direct link to When ScrapingAnt fits the task" title="Direct link to When ScrapingAnt fits the task" translate="no">​</a></h2>
<p>ScrapingAnt provides raw and JavaScript-rendered HTML through one API, with proxy type and readiness in the request configuration. Use <code>browser=false</code> when the required data is already in the response, or <code>browser=true</code> with <code>wait_for_selector</code> when JavaScript creates it. The service runs the rendered browser and proxy retrieval; your application requests and validates the returned HTML.</p>
<p>The measured run makes that choice concrete. Raw mode delivered every catalog record at one tenth of rendered mode's API credit cost. Rendered mode with a readiness selector delivered the JavaScript-created output that raw retrieval missed. One managed API lets you choose the acquisition mode for each extraction contract.</p>
<p>These are service capabilities and observed fixture results. Their operational value depends on your deployment and maintenance needs; setup time, engineering savings and production reliability were not measured. Requests is sufficient for this static catalog. Direct Playwright also handles the delayed diagnostic and remains an option when you need browser-context or interaction control.</p>
<p>Start with raw retrieval on your own permitted target, validate every required field, and enable rendering when the contract needs JavaScript-created data. Use the <a href="https://docs.scrapingant.com/request-response-format" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt request reference</a> to configure that first request. The <a href="https://scrapingant.com/playwright-alternative-web-scraping-api" target="_blank" rel="noopener noreferrer" class="">Playwright alternative page</a> covers the broader managed-service choice.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="environment-timing-and-limits">Environment, timing and limits<a href="https://scrapingant.com/blog/requests-playwright-scrapingant-benchmark#environment-timing-and-limits" class="hash-link" aria-label="Direct link to Environment, timing and limits" title="Direct link to Environment, timing and limits" translate="no">​</a></h2>
<p>The recorded runs used Python 3.12.10, Requests 2.34.2, BeautifulSoup 4.15.0, Playwright 1.62.0 and bundled Chromium 151.0.7922.34 on macOS 26.6.2 arm64. Client geography was not supplied. API worker browser version and reuse are unknown.</p>
<p>Ten rounds shuffle targets and arms with seed 607; concurrency is one. At least one second separates public-origin acquisitions. One Chromium browser is reused, with a fresh context and page per attempt. Requests and API Sessions are reused. Timings include acquisition, context/page creation where applicable, extraction and validation. They exclude browser launch and context cleanup. The public-origin browser launch was 228.74 ms, reported separately.</p>
<p>Direct acquisition and browser readiness share a ten-second budget; total acquisition plus validation has a fifteen-second caller deadline. The API worker timeout is ten seconds, with transit inside the caller deadline. Worker execution is not observable as direct client timing. Requests connect/read timeouts alone are not a wall-clock deadline, so this POSIX runner enforces one. <a href="https://requests.readthedocs.io/en/latest/user/advanced/#timeouts" target="_blank" rel="noopener noreferrer" class="">Requests timeout documentation</a>, <a href="https://playwright.dev/python/docs/api/class-page" target="_blank" rel="noopener noreferrer" class="">Playwright navigation and waits</a>.</p>
<p>The direct and managed arms use different network/proxy paths. These observations compare their acquisition stacks from one client environment, not intrinsic library speed. No concurrent blog build or rendering audit ran during the public-origin acquisition, but unrelated host activity was not controlled.</p>
<p>Ten attempts are a small sample of deterministic pages. The per-fixture Wilson 95% interval is approximately 72.2–100% for ten accepted attempts and 0–27.8% for zero. These are descriptive binomial intervals; shared fixtures and one environment limit their interpretation. They do not estimate production reliability or establish p95, p99 or statistical superiority.</p>
<p>Separate <a href="https://scrapingant.com/blog/assets/files/local-results-761167503a52042af7bdc72bc7a2618d.json" target="_blank" class="">local loopback report (JSON)</a> and <a href="https://scrapingant.com/blog/assets/files/local-attempts-fc8c27b3c7f6cdacf1efd941ad714cf6.csv" target="_blank" class="">forty local attempts (CSV)</a> remain available. Their final capture overlapped a local blog build; local timing excludes Internet transit and is illustrative. Earlier captures are retained in the packet outside the final denominators. None is substituted for an inconvenient public-origin result.</p>
<p>These fixtures do not test production load, browser memory, proxy quality, geography, anti-bot behavior, pagination, authenticated sessions, long-run maintenance or service uptime. Compare methods after they meet your record contract, using every attempt's measured cost and explicitly labeled operational assumptions.</p>
<p><em>Examples tested on 2026-10-09 with Python 3.12.10, Requests 2.34.2 and Playwright 1.62.0. Code and captures: <a href="https://scrapingant.com/blog/assets/files/evidence-packet-70c45679489c4263eaf69bd660d264ad.zip" target="_blank" class="">evidence packet</a>.</em></p>
<p><em>ScrapingAnt publishes this comparison. AI agents drafted this article and executed the recorded benchmark and checks. Oleg Kulyk approved publication and is responsible for the code, measurements and corrections.</em></p>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="web scraping" term="web scraping"/>
        <category label="python" term="python"/>
        <category label="playwright" term="playwright"/>
        <category label="data extraction" term="data extraction"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Apify Pricing: Reconcile Actor Charges and Platform Usage]]></title>
        <id>https://scrapingant.com/blog/apify-pricing</id>
        <link href="https://scrapingant.com/blog/apify-pricing"/>
        <updated>2026-10-02T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Current Apify monthly prices and a tested offline worksheet for Actor events, included usage, prepaid allowance, overage, and cost per accepted record.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Apify Pricing: Reconcile Actor Charges and Platform Usage" src="https://scrapingant.com/blog/assets/images/apify-pricing-09bdd338d9addf15903601fafa443cec.png" width="1200" height="630" class="img_ev3q"></p>
<div class="theme-admonition theme-admonition-info admonition_xJq3 alert alert--info"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg></span>Updated October 7, 2026</div><div class="admonitionContent_BuS1"><p>Added observed Website Content Crawler runs on 30 owned synthetic pages to demonstrate cost per accepted output. The <a href="https://scrapingant.com/blog/assets/files/observed-output-costs-2026-10-07-92baa5d3c24e186a141c8db6e708f763.csv" target="_blank" class="">dated observation CSV</a> preserves full-precision inputs. The seven offline ledgers remain synthetic, and their pricing snapshot remains dated October 2.</p></div></div>
<p>Apify's monthly Starter plan is $19 with $19 of prepaid usage. That does not mean every Actor costs $19, or that its event price always has compute and proxy charges added on top. The chosen Actor's charging model determines which line items consume that allowance.</p>
<p>This guide starts with the current prices, then provides an <strong>offline charge-reconciliation worksheet</strong>. It helps you add known charges once and expose missing inputs. All seven example ledgers are <strong>synthetic</strong>: their Actor IDs, versions, consumption and accepted-record counts are invented test inputs. A separate worked example uses observed Actor runs on owned synthetic pages. Neither set is a customer invoice. ScrapingAnt publishes this article and sells a competing retrieval API.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="current-apify-monthly-prices">Current Apify monthly prices<a href="https://scrapingant.com/blog/apify-pricing#current-apify-monthly-prices" class="hash-link" aria-label="Direct link to Current Apify monthly prices" title="Direct link to Current Apify monthly prices" translate="no">​</a></h2>
<p>Checked <strong>October 2, 2026</strong>, in USD against <a href="https://apify.com/pricing" target="_blank" rel="noopener noreferrer" class="">Apify's official pricing</a>. These are monthly plans; annual billing, custom contracts and partial cycles are outside the worksheet.</p>
<table><thead><tr><th>Plan</th><th style="text-align:right">Monthly fee</th><th style="text-align:right">Prepaid usage</th><th style="text-align:right">Compute price per CU</th></tr></thead><tbody><tr><td>Free</td><td style="text-align:right">$0</td><td style="text-align:right">$5</td><td style="text-align:right">$0.20</td></tr><tr><td>Starter</td><td style="text-align:right">$19</td><td style="text-align:right">$19</td><td style="text-align:right">$0.20</td></tr><tr><td>Scale</td><td style="text-align:right">$199</td><td style="text-align:right">$199</td><td style="text-align:right">$0.16</td></tr><tr><td>Business</td><td style="text-align:right">$999</td><td style="text-align:right">$999</td><td style="text-align:right">$0.13</td></tr></tbody></table>
<p>Prepaid usage pays eligible charges within the account. Paid usage above the allowance goes on the <strong>next invoice</strong>; ordinary unused usage expires. Free access stops when its allowance is exhausted until the next cycle. A paid plan's fee remains due when usage is below its allowance. <a href="https://apify.com/pricing" target="_blank" rel="noopener noreferrer" class="">Source: prepaid usage and exhaustion FAQs</a>.</p>
<p>A larger allowance alone does not establish the best plan. Check required resource limits and the chosen Actor's effective prices before buying. Actor owners decide whether to participate in Store discounts; the worksheet requires the actual event rate rather than applying a universal discount. <a href="https://docs.apify.com/actors/running/actors-in-store#apify-store-discounts" target="_blank" rel="noopener noreferrer" class="">Source: Store discounts</a>.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="identify-what-the-actor-price-includes">Identify what the Actor price includes<a href="https://scrapingant.com/blog/apify-pricing#identify-what-the-actor-price-includes" class="hash-link" aria-label="Direct link to Identify what the Actor price includes" title="Direct link to Identify what the Actor price includes" translate="no">​</a></h2>
<p>Apify documents three relevant cases. <a href="https://docs.apify.com/actors/running/actors-in-store#pricing-models" target="_blank" rel="noopener noreferrer" class="">Sources: Store pricing models</a> and <a href="https://docs.apify.com/actors/publishing/monetize/pay-per-event#pass-platform-usage-costs-on-to-users" target="_blank" rel="noopener noreferrer" class="">passing platform usage to users</a>.</p>
<table><thead><tr><th>Charging model</th><th>Developer event charges</th><th>Actor-run platform usage</th></tr></thead><tbody><tr><td>Pay per event, usage included</td><td>Charged events × effective event price</td><td>Included; do not add it again</td></tr><tr><td>Pay per event, usage extra</td><td>Charged events × effective event price</td><td>Add billable resource usage</td></tr><tr><td>Usage-only</td><td>No additional developer event fee</td><td>Add billable resource usage</td></tr></tbody></table>
<p><strong>Post-run dataset interactions remain billable in every model.</strong> Reading or writing data after the run and retaining storage can create separate usage. An included run price does not cover every future use of its output. <a href="https://docs.apify.com/actors/running/actors-in-store#pricing-models" target="_blank" rel="noopener noreferrer" class="">Source: post-run storage costs</a>.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-changed-for-rental-actors-on-october-1">What changed for rental Actors on October 1?<a href="https://scrapingant.com/blog/apify-pricing#what-changed-for-rental-actors-on-october-1" class="hash-link" aria-label="Direct link to What changed for rental Actors on October 1?" title="Direct link to What changed for rental Actors on October 1?" translate="no">​</a></h3>
<p>Apify published a rental sunset: new rental Actors and rental-price changes stopped on April 1, 2026; remaining rentals were scheduled to retire or migrate to usage-only on October 1, 2026. This is a <strong>published policy schedule</strong>, not our observation that every Actor migrated. The documentation and pricing page still contained rental references when checked on October 2. Confirm the selected Actor's actual model; the worksheet rejects rental input rather than guessing its replacement. <a href="https://docs.apify.com/actors/running/actors-in-store#rental-actors" target="_blank" rel="noopener noreferrer" class="">Source: rental model sunset</a>.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="build-the-bill-of-materials">Build the bill of materials<a href="https://scrapingant.com/blog/apify-pricing#build-the-bill-of-materials" class="hash-link" aria-label="Direct link to Build the bill of materials" title="Direct link to Build the bill of materials" translate="no">​</a></h2>
<p>Record the Actor ID, version or build, billing cycle, plan, dated official price source and charging model. Then collect charged event counts and resource totals for that cycle. Keep the Actor-run scope separate from post-run storage and access, and from unrelated account usage.</p>
<p>The <a href="https://apify.com/pricing" target="_blank" rel="noopener noreferrer" class="">official resource table</a> supplies rates; it cannot tell you how much an arbitrary target will consume. Selected Free/Starter rates used by the examples are:</p>
<table><thead><tr><th>Meter</th><th style="text-align:right">Price</th><th>Required quantity</th></tr></thead><tbody><tr><td>Compute</td><td style="text-align:right">$0.20 per CU</td><td>Billed CU</td></tr><tr><td>Residential proxy</td><td style="text-align:right">$8 per GB</td><td>Billed traffic GB</td></tr><tr><td>Dataset storage</td><td style="text-align:right">$1 per 1,000 GB-hours</td><td>Integrated storage over time</td></tr><tr><td>Dataset reads</td><td style="text-align:right">$0.0004 per 1,000 operations</td><td>Read operations</td></tr><tr><td>Dataset writes</td><td style="text-align:right">$0.005 per 1,000 operations</td><td>Write operations</td></tr><tr><td>Request queue writes</td><td style="text-align:right">$0.01 per 1,000 operations</td><td>Write operations</td></tr><tr><td>Internal transfer</td><td style="text-align:right">$0.05 per GB</td><td>Billed internal GB</td></tr><tr><td>External transfer</td><td style="text-align:right">$0.20 per GB</td><td>Billed external GB</td></tr></tbody></table>
<p>The worksheet also retains verified rates for key-value storage, queue storage/reads, SERPs and Website Unblocker. It preserves the published divisors: a price per <strong>1,000 operations</strong> is not a price per operation. Datacenter IP offers are documented separately but not automatically priced; their applicable cadence, add-on amounts and allowance treatment must be known or listed as unresolved.</p>
<p>One CU corresponds to an hour with 1,024 MB allocated in Apify's <a href="https://docs.apify.com/actors/running/usage-and-resources#usage" target="_blank" rel="noopener noreferrer" class="">compute-unit explanation</a>. URL count and allocated memory do not establish billed consumption without runtime and other meters. The worksheet accepts supplied units; it does not predict retries, browser duration, traffic or future retention.</p>
<p>For historical reconciliation, use effective-period prices and billing records. Apify says historical run dollar displays use current service prices and are informational. Recalculating old meters at today's rates is not proof of an old invoice. <a href="https://docs.apify.com/actors/running/usage-and-resources#usage" target="_blank" rel="noopener noreferrer" class="">Source: usage billing elements</a>.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="charged-events-are-not-accepted-records">Charged events are not accepted records<a href="https://scrapingant.com/blog/apify-pricing#charged-events-are-not-accepted-records" class="hash-link" aria-label="Direct link to Charged events are not accepted records" title="Direct link to Charged events are not accepted records" translate="no">​</a></h3>
<p>Use already <strong>charged</strong> event counts, not requested event counts or validated output counts. Apify's event API can charge fewer events than requested when a limit is reached. Event limits are per run, and resources can continue briefly while a run stops. Do not treat an event cap as a guaranteed total monthly bill cap. <a href="https://docs.apify.com/actors/publishing/monetize/pay-per-event#respect-user-spending-limits" target="_blank" rel="noopener noreferrer" class="">Source: spending limits and chargedCount</a>.</p>
<p>The examples use the documented, configurable <code>apify-actor-start</code> default of <strong>$0.00005 per event</strong>. Its charged count depends on memory, and enabling it covers the first five seconds of compute. The examples supply charged counts and applicable billed CU directly; they do not derive starts or subtract that window again. No real Actor is assumed to use this default. <a href="https://docs.apify.com/actors/publishing/monetize/pay-per-event#use-apify-actor-start" target="_blank" rel="noopener noreferrer" class="">Source: start event</a>.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="apply-the-account-allowance-once">Apply the account allowance once<a href="https://scrapingant.com/blog/apify-pricing#apply-the-account-allowance-once" class="hash-link" aria-label="Direct link to Apply the account allowance once" title="Direct link to Apply the account allowance once" translate="no">​</a></h2>
<p>Let <code>W</code> be eligible charges for the selected Actors, <code>O</code> other eligible account usage, <code>A</code> the prepaid allowance, <code>F</code> the subscription fee, and <code>X</code> known cash charges outside prepaid eligibility.</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">eligible account usage = W + O</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">prepaid applied = min(W + O, A)</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">paid-plan overage = max(W + O - A, 0)</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">modeled cycle cash cost = F + overage + X</span><br></div></code></pre></div></div>
<p>This attributes a plan fee and usage overage to a cycle. Because overage is added to the next invoice, it is <strong>not a prediction of one invoice's payment amount</strong>. It excludes unknown adjustments, taxes, credits and unsupported billing arrangements. Do not add the prepaid allowance as a second charge or give each Actor its own account allowance.</p>
<p><img decoding="async" loading="lazy" alt="Seven-step Apify reconciliation flow: confirm events, determine run usage inclusion, add post-run charges, add other usage, apply one allowance, check unknowns and Free limits, then calculate cycle cash cost." src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI3MDAiIGhlaWdodD0iMTEwMCIgdmlld0JveD0iMCAwIDcwMCAxMTAwIiByb2xlPSJpbWciIGFyaWEtbGFiZWxsZWRieT0idGl0bGUgZGVzYyI+Cjx0aXRsZSBpZD0idGl0bGUiPkFwaWZ5IG1vbnRobHkgY2hhcmdlIHJlY29uY2lsaWF0aW9uIGZsb3c8L3RpdGxlPjxkZXNjIGlkPSJkZXNjIj5JZGVudGlmeSBldmVudCBwcmljZXMgYW5kIHJ1bi11c2FnZSBpbmNsdXNpb247IGFkZCBwb3N0LXJ1biBhbmQgb3RoZXIgZWxpZ2libGUgdXNhZ2U7IGFwcGx5IG9uZSBhbGxvd2FuY2UuIE1pc3NpbmcgaW5wdXRzIHByZXZlbnQgYSBwcmVjaXNlIGJpbGwuIEZyZWUgZXhjZXNzIGlzIGluZmVhc2libGUuIFBhaWQgY3ljbGUgY29zdCBpcyBwbGFuIGZlZSBwbHVzIG5leHQtaW52b2ljZSBvdmVyYWdlIGFuZCBrbm93biBleHRyYSBjYXNoIGNoYXJnZXMuPC9kZXNjPgo8c3R5bGU+dGV4dHtmb250LWZhbWlseTpBcmlhbCxzYW5zLXNlcmlmO2ZpbGw6IzE0MmM0M30uaGVhZGluZ3tmb250LXdlaWdodDo3MDB9Lm11dGVke2ZpbGw6IzQ4NjQ3N30uc21hbGx7Zm9udC1zaXplOjIwcHh9PC9zdHlsZT4KPHJlY3Qgd2lkdGg9IjcwMCIgaGVpZ2h0PSIxMTAwIiBmaWxsPSIjZjRmOGZiIi8+PHRleHQgeD0iMzIiIHk9IjUwIiBmb250LXNpemU9IjI5IiBjbGFzcz0iaGVhZGluZyI+UmVjb25jaWxlIGEgbW9udGhseSBBcGlmeSBiaWxsPC90ZXh0Pjx0ZXh0IHg9IjMyIiB5PSI4MiIgZm9udC1zaXplPSIyMSIgY2xhc3M9Im11dGVkIj5Nb2RlbCBzY29wZTogVVNELCBrbm93biBjeWNsZSB0b3RhbHM8L3RleHQ+PHJlY3QgeD0iMjIiIHk9IjExNiIgd2lkdGg9IjY1NiIgaGVpZ2h0PSI5NCIgcng9IjEyIiBmaWxsPSJ3aGl0ZSIgc3Ryb2tlPSIjYmZkMGRjIi8+PHRleHQgeD0iNDAiIHk9IjE1MSIgZm9udC1zaXplPSIyMyIgY2xhc3M9ImhlYWRpbmciPjEgIENvbmZpcm0gQWN0b3IgcHJpY2VzIGFuZCBjaGFyZ2VkIGV2ZW50czwvdGV4dD48dGV4dCB4PSI0MCIgeT0iMTgzIiBmb250LXNpemU9IjIwIiBjbGFzcz0iIj5SZWNvcmQgQWN0b3IvYnVpbGQsIHNvdXJjZSBkYXRlIGFuZCBldmVudCBkaXZpc29yLjwvdGV4dD48cGF0aCBkPSJNMzUwIDIxNHYyMG0tNi02IDYgNiA2LTYiIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzE2OGM4MiIgc3Ryb2tlLXdpZHRoPSIzIi8+PHJlY3QgeD0iMjIiIHk9IjI0MiIgd2lkdGg9IjY1NiIgaGVpZ2h0PSI5NCIgcng9IjEyIiBmaWxsPSJ3aGl0ZSIgc3Ryb2tlPSIjYmZkMGRjIi8+PHRleHQgeD0iNDAiIHk9IjI3NyIgZm9udC1zaXplPSIyMyIgY2xhc3M9ImhlYWRpbmciPjIgIERldGVybWluZSBBY3Rvci1ydW4gdXNhZ2UgaW5jbHVzaW9uPC90ZXh0Pjx0ZXh0IHg9IjQwIiB5PSIzMDkiIGZvbnQtc2l6ZT0iMjAiIGNsYXNzPSIiPkluY2x1ZGVkIOKGkiBhZGQgJDAuIEV4dHJhIC8gdXNhZ2Utb25seSDihpIgYWRkIG1ldGVycy48L3RleHQ+PHBhdGggZD0iTTM1MCAzNDB2MjBtLTYtNiA2IDYgNi02IiBmaWxsPSJub25lIiBzdHJva2U9IiMxNjhjODIiIHN0cm9rZS13aWR0aD0iMyIvPjxyZWN0IHg9IjIyIiB5PSIzNjgiIHdpZHRoPSI2NTYiIGhlaWdodD0iOTQiIHJ4PSIxMiIgZmlsbD0id2hpdGUiIHN0cm9rZT0iI2JmZDBkYyIvPjx0ZXh0IHg9IjQwIiB5PSI0MDMiIGZvbnQtc2l6ZT0iMjMiIGNsYXNzPSJoZWFkaW5nIj4zICBBZGQgcG9zdC1ydW4gc3RvcmFnZSBhbmQgdHJhbnNmZXI8L3RleHQ+PHRleHQgeD0iNDAiIHk9IjQzNSIgZm9udC1zaXplPSIyMCIgY2xhc3M9IiI+U2VwYXJhdGUgc2NvcGU7IGJpbGxhYmxlIGluIGV2ZXJ5IGNoYXJnaW5nIG1vZGVsLjwvdGV4dD48cGF0aCBkPSJNMzUwIDQ2NnYyMG0tNi02IDYgNiA2LTYiIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzE2OGM4MiIgc3Ryb2tlLXdpZHRoPSIzIi8+PHJlY3QgeD0iMjIiIHk9IjQ5NCIgd2lkdGg9IjY1NiIgaGVpZ2h0PSI5NCIgcng9IjEyIiBmaWxsPSJ3aGl0ZSIgc3Ryb2tlPSIjYmZkMGRjIi8+PHRleHQgeD0iNDAiIHk9IjUyOSIgZm9udC1zaXplPSIyMyIgY2xhc3M9ImhlYWRpbmciPjQgIEluY2x1ZGUgb3RoZXIgZWxpZ2libGUgYWNjb3VudCB1c2FnZTwvdGV4dD48dGV4dCB4PSI0MCIgeT0iNTYxIiBmb250LXNpemU9IjIwIiBjbGFzcz0iIj5FeGNsdWRlIHRoZXNlIEFjdG9yIHNjb3BlcyBmcm9tIHRoYXQgb3RoZXIgdG90YWwuPC90ZXh0PjxwYXRoIGQ9Ik0zNTAgNTkydjIwbS02LTYgNiA2IDYtNiIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjMTY4YzgyIiBzdHJva2Utd2lkdGg9IjMiLz48cmVjdCB4PSIyMiIgeT0iNjIwIiB3aWR0aD0iNjU2IiBoZWlnaHQ9Ijk0IiByeD0iMTIiIGZpbGw9IndoaXRlIiBzdHJva2U9IiNiZmQwZGMiLz48dGV4dCB4PSI0MCIgeT0iNjU1IiBmb250LXNpemU9IjIzIiBjbGFzcz0iaGVhZGluZyI+NSAgQXBwbHkgdGhlIGFjY291bnQgYWxsb3dhbmNlIG9uY2U8L3RleHQ+PHRleHQgeD0iNDAiIHk9IjY4NyIgZm9udC1zaXplPSIyMCIgY2xhc3M9IiI+UGFpZCBvdmVyYWdlID0gbWF4KGVsaWdpYmxlIHVzYWdlIOKIkiBhbGxvd2FuY2UsIDApLjwvdGV4dD48cGF0aCBkPSJNMzUwIDcxOHYyMG0tNi02IDYgNiA2LTYiIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzE2OGM4MiIgc3Ryb2tlLXdpZHRoPSIzIi8+PHJlY3QgeD0iMjIiIHk9Ijc0NiIgd2lkdGg9IjY1NiIgaGVpZ2h0PSI5NCIgcng9IjEyIiBmaWxsPSJ3aGl0ZSIgc3Ryb2tlPSIjYmZkMGRjIi8+PHRleHQgeD0iNDAiIHk9Ijc4MSIgZm9udC1zaXplPSIyMyIgY2xhc3M9ImhlYWRpbmciPjYgIENoZWNrIG1pc3NpbmcgaW5wdXRzIGFuZCBGcmVlIGZlYXNpYmlsaXR5PC90ZXh0Pjx0ZXh0IHg9IjQwIiB5PSI4MTMiIGZvbnQtc2l6ZT0iMjAiIGNsYXNzPSIiPk1pc3Npbmcg4oaSIHBhcnRpYWwuIEFib3ZlIEZyZWUgYWxsb3dhbmNlIOKGkiBpbmZlYXNpYmxlLjwvdGV4dD48cGF0aCBkPSJNMzUwIDg0NHYyMG0tNi02IDYgNiA2LTYiIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzE2OGM4MiIgc3Ryb2tlLXdpZHRoPSIzIi8+PHJlY3QgeD0iMjIiIHk9Ijg3MiIgd2lkdGg9IjY1NiIgaGVpZ2h0PSI5NCIgcng9IjEyIiBmaWxsPSJ3aGl0ZSIgc3Ryb2tlPSIjYmZkMGRjIi8+PHRleHQgeD0iNDAiIHk9IjkwNyIgZm9udC1zaXplPSIyMyIgY2xhc3M9ImhlYWRpbmciPjcgIEF0dHJpYnV0ZSBjeWNsZSBjYXNoIGNvc3Q8L3RleHQ+PHRleHQgeD0iNDAiIHk9IjkzOSIgZm9udC1zaXplPSIyMCIgY2xhc3M9IiI+UGFpZCBwbGFuIGZlZSArIG92ZXJhZ2UgKyBrbm93biBleHRyYSBjYXNoIGNoYXJnZXMuPC90ZXh0Pjx0ZXh0IHg9IjMyIiB5PSIxMDM2IiBmb250LXNpemU9IjIyIiBjbGFzcz0iaGVhZGluZyI+T3ZlcmFnZSBpcyBhZGRlZCB0byB0aGUgbmV4dCBpbnZvaWNlLjwvdGV4dD48dGV4dCB4PSIzMiIgeT0iMTA3MCIgZm9udC1zaXplPSIyMCIgY2xhc3M9Im11dGVkIj5SdWxlcyBjaGVja2VkIDIwMjYtMTAtMDIuIE5vdCBhbiBvYnNlcnZlZCBpbnZvaWNlLjwvdGV4dD48L3N2Zz4K" width="700" height="1100" class="img_ev3q"></p>
<p><em>Documented-rule flow, checked October 2, 2026. Unknown billable inputs stop an exact total; exceeding Free's allowance is infeasible in this model.</em> <a href="https://scrapingant.com/blog/assets/files/bill-flow-730a9158860d439a4f384c424ed8f829.svg" target="_blank" class="">Open the flow at full size</a>.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="reproduce-the-synthetic-ledgers">Reproduce the synthetic ledgers<a href="https://scrapingant.com/blog/apify-pricing#reproduce-the-synthetic-ledgers" class="hash-link" aria-label="Direct link to Reproduce the synthetic ledgers" title="Direct link to Reproduce the synthetic ledgers" translate="no">​</a></h2>
<p>Use the <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/fa9f7d878660eba9e3a053dd3fc6a780b7cb3412/examples/apify-pricing" target="_blank" rel="noopener noreferrer" class="">versioned offline worksheet</a>. With Git installed, clone and check out the tested commit:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">git clone https://github.com/ScrapingAnt/scrapingant-examples.git</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">cd scrapingant-examples</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">git checkout fa9f7d878660eba9e3a053dd3fc6a780b7cb3412</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">cd examples/apify-pricing</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">./run.sh</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3 model.py fixtures/s2-event-plus-usage.json --csv /tmp/apify-s2-lines.csv</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3 model.py fixtures/s4-missing-usage.json --csv /tmp/apify-s4-lines.csv</span><br></div></code></pre></div></div>
<p>Python <strong>3.10.2 and 3.12.11</strong> were tested. No API key or dependency installation is needed. <code>./run.sh</code> runs <strong>38 tests</strong>, compares <strong>15 JSON/CSV files</strong> for seven scenarios, and checks three deterministic SVGs. The two <code>model.py</code> commands print JSON and export line-item CSVs. They do not run an Actor.</p>
<p>The fully specified synthetic Actor scopes below have a run-resource valuation of <strong>$21.64</strong> and post-run charges of <strong>$0.584</strong>. S4 omits residential proxy quantity, so its complete run valuation is unknown. Event cases add 960 charged start events × $0.00005 = <strong>$0.048</strong>. The <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/fa9f7d878660eba9e3a053dd3fc6a780b7cb3412/examples/apify-pricing/expected_output" target="_blank" rel="noopener noreferrer" class="">fixtures and captured outputs</a> expose each meter. These are supplied quantities, not measured resource consumption.</p>
<table><thead><tr><th>Synthetic case</th><th style="text-align:right">Eligible workload charges</th><th style="text-align:right">Modeled account cycle cash</th><th>What it tests</th></tr></thead><tbody><tr><td>S1: events include run usage</td><td style="text-align:right">$0.632</td><td style="text-align:right">$19</td><td>Exclude the $21.64 run valuation; retain the plan floor</td></tr><tr><td>S2: events plus run usage</td><td style="text-align:right">$22.272</td><td style="text-align:right">$22.272</td><td>Add run usage once; $3.272 next-invoice overage</td></tr><tr><td>S3: usage-only</td><td style="text-align:right">$22.224</td><td style="text-align:right">$22.224</td><td>No developer event charge</td></tr><tr><td>S4: missing residential volume</td><td style="text-align:right">Known subtotal $14.272</td><td style="text-align:right">Unknown</td><td>Unknown is not zero</td></tr><tr><td>S5: zero accepted records</td><td style="text-align:right">$22.272</td><td style="text-align:right">$22.272</td><td>Unit cost is undefined</td></tr><tr><td>S6: two Actors, plus $15 other usage</td><td style="text-align:right">$44.544</td><td style="text-align:right">$59.544</td><td>Apply one shared allowance; $40.544 overage</td></tr><tr><td>S7: S2 workload on Free</td><td style="text-align:right">$22.272 requested</td><td style="text-align:right">Infeasible</td><td>$17.272 beyond allowance; no invented paid overage</td></tr></tbody></table>
<p><img decoding="async" loading="lazy" alt="Synthetic Starter comparison: S1 has $0.632 eligible usage but a $19 cycle cost; S2 has $22.272 usage and a $22.272 cycle cost comprising the $19 fee and $3.272 overage." src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI3NjAiIGhlaWdodD0iNzMwIiB2aWV3Qm94PSIwIDAgNzYwIDczMCIgcm9sZT0iaW1nIiBhcmlhLWxhYmVsbGVkYnk9InRpdGxlIGRlc2MiPgo8dGl0bGUgaWQ9InRpdGxlIj5TeW50aGV0aWMgQXBpZnkgU3RhcnRlciBiaWxsIGNhbGN1bGF0aW9uPC90aXRsZT48ZGVzYyBpZD0iZGVzYyI+UzEgaW5jbHVkZXMgcnVuIHVzYWdlOiBlbGlnaWJsZSBjaGFyZ2VzIDAuNjMyIGRvbGxhcnMsIG1vZGVsZWQgY3ljbGUgYmlsbCAxOSBkb2xsYXJzLCB1bnVzZWQgcHJlcGFpZCAxOC4zNjguIFMyIGFkZHMgcnVuIHVzYWdlOiBjaGFyZ2VzIGFuZCBjeWNsZSBiaWxsIDIyLjI3MiBkb2xsYXJzLCBuZXh0LWludm9pY2Ugb3ZlcmFnZSAzLjI3Mi4gQm90aCBhcmUgc3ludGhldGljIGFyaXRobWV0aWMgZXhhbXBsZXMuPC9kZXNjPgo8c3R5bGU+dGV4dHtmb250LWZhbWlseTpBcmlhbCxzYW5zLXNlcmlmO2ZpbGw6IzE0MmM0M30uaGVhZGluZ3tmb250LXdlaWdodDo3MDB9Lm11dGVke2ZpbGw6IzQ4NjQ3N30uc21hbGx7Zm9udC1zaXplOjIwcHh9PC9zdHlsZT4KPHJlY3Qgd2lkdGg9Ijc2MCIgaGVpZ2h0PSI3MzAiIGZpbGw9IiNmNGY4ZmIiLz48dGV4dCB4PSIzMiIgeT0iNDgiIGZvbnQtc2l6ZT0iMjkiIGNsYXNzPSJoZWFkaW5nIj5UaGUgbW9udGhseSBmbG9vciBzdXJ2aXZlcyB1bmRlcnVzZTwvdGV4dD48dGV4dCB4PSIzMiIgeT0iODAiIGZvbnQtc2l6ZT0iMjEiIGNsYXNzPSJtdXRlZCI+U1lOVEhFVElDIFN0YXJ0ZXIgZXhhbXBsZXMg4oCiIG5vdCBjdXN0b21lciBiaWxsczwvdGV4dD48cmVjdCB4PSIyMiIgeT0iMTE0IiB3aWR0aD0iNzE2IiBoZWlnaHQ9IjIzMiIgcng9IjEyIiBmaWxsPSJ3aGl0ZSIgc3Ryb2tlPSIjYmZkMGRjIi8+PHRleHQgeD0iNDAiIHk9IjE0OCIgZm9udC1zaXplPSIyNSIgY2xhc3M9ImhlYWRpbmciPlMxICBFdmVudCBwcmljZSBpbmNsdWRlcyBydW4gdXNhZ2U8L3RleHQ+PHRleHQgeD0iNDAiIHk9IjE3OSIgZm9udC1zaXplPSIyMiIgY2xhc3M9IiI+RWxpZ2libGUgY2hhcmdlczogJDAuNjMyPC90ZXh0PjxyZWN0IHg9IjQwIiB5PSIyMDMiIHdpZHRoPSI0NTYiIGhlaWdodD0iNDYiIGZpbGw9IiMzNTU2NmYiLz48dGV4dCB4PSI0MCIgeT0iMjg1IiBmb250LXNpemU9IjI1IiBjbGFzcz0iaGVhZGluZyI+Q3ljbGUgY2FzaCBjb3N0OiAkMTk8L3RleHQ+PHRleHQgeD0iNDAiIHk9IjMxOCIgZm9udC1zaXplPSIyMiIgY2xhc3M9IiI+VW51c2VkIHByZXBhaWQ6ICQxOC4zNjg8L3RleHQ+PHJlY3QgeD0iMjIiIHk9IjM2NCIgd2lkdGg9IjcxNiIgaGVpZ2h0PSIyMzIiIHJ4PSIxMiIgZmlsbD0id2hpdGUiIHN0cm9rZT0iI2JmZDBkYyIvPjx0ZXh0IHg9IjQwIiB5PSIzOTgiIGZvbnQtc2l6ZT0iMjUiIGNsYXNzPSJoZWFkaW5nIj5TMiAgRXZlbnRzIHBsdXMgcnVuIHVzYWdlPC90ZXh0Pjx0ZXh0IHg9IjQwIiB5PSI0MjkiIGZvbnQtc2l6ZT0iMjIiIGNsYXNzPSIiPkVsaWdpYmxlIGNoYXJnZXM6ICQyMi4yNzI8L3RleHQ+PHJlY3QgeD0iNDAiIHk9IjQ1MyIgd2lkdGg9IjQ1NiIgaGVpZ2h0PSI0NiIgZmlsbD0iIzM1NTY2ZiIvPjxyZWN0IHg9IjQ5NiIgeT0iNDUzIiB3aWR0aD0iNzguNTI4IiBoZWlnaHQ9IjQ2IiBmaWxsPSIjMTY4YzgyIi8+PHRleHQgeD0iNDAiIHk9IjUzNSIgZm9udC1zaXplPSIyNSIgY2xhc3M9ImhlYWRpbmciPkN5Y2xlIGNhc2ggY29zdDogJDIyLjI3MjwvdGV4dD48dGV4dCB4PSI0MCIgeT0iNTY4IiBmb250LXNpemU9IjIyIiBjbGFzcz0iIj5OZXh0LWludm9pY2Ugb3ZlcmFnZTogJDMuMjcyPC90ZXh0PjxyZWN0IHg9IjMyIiB5PSI2NDAiIHdpZHRoPSIyMCIgaGVpZ2h0PSIyMCIgZmlsbD0iIzM1NTY2ZiIvPjx0ZXh0IHg9IjY0IiB5PSI2NTgiIGZvbnQtc2l6ZT0iMjEiIGNsYXNzPSIiPk1vbnRobHkgcGxhbiBmZWUgKCQxOSk8L3RleHQ+PHJlY3QgeD0iMzg4IiB5PSI2NDAiIHdpZHRoPSIyMCIgaGVpZ2h0PSIyMCIgZmlsbD0iIzE2OGM4MiIvPjx0ZXh0IHg9IjQyMCIgeT0iNjU4IiBmb250LXNpemU9IjIxIiBjbGFzcz0iIj5PdmVyYWdlIGJleW9uZCBwcmVwYWlkPC90ZXh0Pjx0ZXh0IHg9IjMyIiB5PSI3MDAiIGZvbnQtc2l6ZT0iMjAiIGNsYXNzPSJtdXRlZCI+U3ludGhldGljIHF1YW50aXR5IGlucHV0czsgcHJpY2VzIGNoZWNrZWQgMjAyNi0xMC0wMi48L3RleHQ+PC9zdmc+Cg==" width="760" height="730" class="img_ev3q"></p>
<p><em>Synthetic arithmetic, not customer bills. Underusing prepaid usage does not refund the subscription; extra usage above it creates overage.</em> <a href="https://scrapingant.com/blog/assets/files/synthetic-costs-cf7cc0f76e9cf0d404c55e2f3c53758f.svg" target="_blank" class="">Open the chart at full size</a>.</p>
<p>The model retains subcent amounts. It displays cents only at the end using half-up rounding, so S2 displays <strong>$22.27</strong> while retaining <strong>$22.272</strong> internally. This is a worksheet convention; Apify's actual invoice-rounding rules were not verified.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="use-your-own-inputs-and-inspect-unknowns">Use your own inputs and inspect unknowns<a href="https://scrapingant.com/blog/apify-pricing#use-your-own-inputs-and-inspect-unknowns" class="hash-link" aria-label="Direct link to Use your own inputs and inspect unknowns" title="Direct link to Use your own inputs and inspect unknowns" translate="no">​</a></h2>
<p>Copy <code>fixtures/worksheet-template.json</code>, set <code>input_type</code> to <code>user_supplied</code>, name the worksheet and enter only known values. The <a href="https://github.com/ScrapingAnt/scrapingant-examples/blob/fa9f7d878660eba9e3a053dd3fc6a780b7cb3412/examples/apify-pricing/README.md" target="_blank" rel="noopener noreferrer" class="">README</a> defines every scope and unit. Use decimal strings or integer counts. Wrong units/currency/cadence, duplicate keys/scopes, fractional operation counts and invalid numbers are rejected.</p>
<p>Supply <code>other_account_usage_usd</code> <strong>excluding these Actor scopes</strong>. Do not enter the entire account total there and repeat the Actors. Supply outside-allowance charges separately. The code cannot detect the same actual usage entered into multiple scopes; reconcile their provenance yourself.</p>
<p>After filling your copied template, run <code>python3 model.py worksheet.json --csv /tmp/apify-worksheet-lines.csv</code>. The default <code>rates.json</code> snapshot is dated October 2, 2026. For a different effective period, supply an independently verified monthly snapshot with <code>--rates your-rates.json</code>; the model does not retrieve historical prices.</p>
<p>Keep unknown quantities as JSON <code>null</code>. CSV blanks also mean unknown. <code>status=partial</code> suppresses the precise account total and lists <code>unresolved</code>; missing included-run quantities appear as memo gaps. S4's known $19 cash lower bound assumes nonnegative charges and no adjustments; it is not a reconciled bill. A successful process exit alone does not mean the worksheet is complete. Invalid input exits with code 2.</p>
<p><code>unresolved_items</code> records add-ons, eligibility, adjustments and limit feasibility that have not been settled. An empty list is your completeness attestation. Dated official URL metadata documents provenance but does not authenticate a supplied rate. Confirm the Actor's effective inclusion setting and price yourself.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="choose-the-accepted-output-denominator">Choose the accepted-output denominator<a href="https://scrapingant.com/blog/apify-pricing#choose-the-accepted-output-denominator" class="hash-link" aria-label="Direct link to Choose the accepted-output denominator" title="Direct link to Choose the accepted-output denominator" translate="no">​</a></h3>
<p>Define what a valid record requires: for example, a unique product ID, required fields and the expected revision. Keep charged failed attempts in the numerator; count an accepted record once. The examples' acceptance counts are synthetic assumptions.</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">workload usage per accepted record = W / accepted records</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">account cash per accepted record = (F + overage + X) / accepted records</span><br></div></code></pre></div></div>
<p>S2 assumes 80 accepted records: both ratios are <strong>$0.2784</strong> here. S1 uses the same assumed count: workload usage per record is <strong>$0.0079</strong>, while full account cash per record is <strong>$0.2375</strong>. They answer different questions. Full account cash includes the subscription and unrelated usage/add-ons; it is not marginal workload cost. Zero or unknown accepted count makes either ratio undefined.</p>
<p>The output also separates incremental workload cash, assuming you retain the same plan and other usage without this workload. None of these ratios establishes a real acceptance rate, price per URL or cheapest vendor.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="observed-example-cost-per-accepted-page">Observed example: cost per accepted page<a href="https://scrapingant.com/blog/apify-pricing#observed-example-cost-per-accepted-page" class="hash-link" aria-label="Direct link to Observed example: cost per accepted page" title="Direct link to Observed example: cost per accepted page" translate="no">​</a></h2>
<p>An inexpensive run can still yield expensive usable output. On <strong>October 7, 2026</strong>, we ran Website Content Crawler build <strong>0.3.97</strong> on the same <strong>30 owned synthetic pages</strong> under three configurations. These are observed runs, separate from the invented worksheet ledgers above. Each configuration has one run; the observations do not establish a typical success rate or future price.</p>
<p>We kept the fixture bodies, assignments and original acceptance rule fixed. An accepted output had to correspond to one assigned URL, retain at least <strong>95% of its 20 required sentence IDs and complete whitespace-normalized sentences</strong>, contain at most <strong>one distinct boilerplate ID</strong>, and come from a natively <code>SUCCEEDED</code> run. Missing, duplicate and erroneous output was not accepted. The primary score used <code>text</code>; Markdown and formatting markers were diagnostic. See the <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/f79a916e324cb990110879bf00b0fd687753c314/fixtures/apify-study-v1" target="_blank" rel="noopener noreferrer" class="">pinned fixtures</a> and <a href="https://github.com/ScrapingAnt/scrapingant-examples/blob/f79a916e324cb990110879bf00b0fd687753c314/examples/apify-continuation-study/acceptance.py" target="_blank" rel="noopener noreferrer" class="">acceptance implementation</a>.</p>
<p>The runs used <code>playwright:firefox</code>, 8,192 MB allocated memory, a 120-second run timeout, maximum concurrency 1 and zero request retries. The original transformer was <code>readableTextIfPossible</code>. The second run added only <code>ignoreCanonicalUrl=true</code>; the third retained that setting and changed only <code>htmlTransformer</code> to <code>readableText</code>. Apify documents that the ignore option skips both ETag and canonical-URL checks; it should not be enabled indiscriminately. <a href="https://apify.com/apify/website-content-crawler/input-schema" target="_blank" rel="noopener noreferrer" class="">Source: WCC input settings</a>.</p>
<table><thead><tr><th>Configuration</th><th style="text-align:right">Assigned pages</th><th style="text-align:right">Raw output rows</th><th style="text-align:right">Accepted outputs</th><th style="text-align:right">Informational run USD</th><th style="text-align:right">USD per 1,000 accepted outputs in this batch</th></tr></thead><tbody><tr><td>Original input</td><td style="text-align:right">30</td><td style="text-align:right">3</td><td style="text-align:right">2</td><td style="text-align:right">$0.02385345</td><td style="text-align:right">$11.926727</td></tr><tr><td>Ignore ETag/canonical checks</td><td style="text-align:right">30</td><td style="text-align:right">30</td><td style="text-align:right">20</td><td style="text-align:right">$0.04011095</td><td style="text-align:right">$2.005547</td></tr><tr><td>Above, plus <code>readableText</code></td><td style="text-align:right">30</td><td style="text-align:right">30</td><td style="text-align:right">30</td><td style="text-align:right">$0.02807915</td><td style="text-align:right">$0.935972</td></tr></tbody></table>
<p>The numerator is each run's <strong>informational <code>usageTotalUsd</code> snapshot from October 7</strong>, including work that did not produce accepted output. It is not a reconciled cash payment. The ratios use full-precision amounts in the <a href="https://scrapingant.com/blog/assets/files/observed-output-costs-2026-10-07-92baa5d3c24e186a141c8db6e708f763.csv" target="_blank" class="">downloadable CSV</a>; the table rounds run amounts to eight decimals and ratios to six. The CSV contains aggregate observations for reproducing this arithmetic; it does not independently verify an invoice or reproduce the historical runs. Multiplying by 1,000 makes the batch ratio easier to compare; <strong>we did not run 1,000 pages in these cells</strong>, and this is not a volume quote.</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">batch USD per 1,000 accepted outputs = informational run USD × 1,000 / accepted outputs</span><br></div></code></pre></div></div>
<p>To reproduce the arithmetic without an API call, download the CSV and run this with Python 3:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3 - observed-output-costs-2026-10-07.csv &lt;&lt;'PY'</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">import csv, sys</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">from decimal import Decimal</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">with open(sys.argv[1], newline="", encoding="utf-8") as source:</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    for row in csv.DictReader(source):</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        ratio = Decimal(row["informational_run_usd"]) * 1000 / int(row["accepted_outputs"])</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        print(row["stage"], format(ratio, ".6f"))</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">PY</span><br></div></code></pre></div></div>
<p>Captured arithmetic output:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">original 11.926727</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">ignore_etag_canonical 2.005547</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">readable_text 0.935972</span><br></div></code></pre></div></div>
<p>The original run skipped <strong>27 pages</strong> because different fixture bodies shared an observed ETag: a hosting confound, not evidence that WCC generally drops distinct pages. Ignoring those checks removed the skips. Ten remaining outputs retained all 20 required sentences but contained eight boilerplate IDs; changing the transformer reduced those IDs to zero. The final native log still reported <strong>29 finished requests and one failed request</strong>, despite 30 accepted exported outputs. Export quality and error-free execution are separate checks.</p>
<p>These figures exclude allocation of the subscription fee, prepaid allowance, overage and later storage or data access. Historical dollar displays can change with current service prices, so a later snapshot of the same run must not be added as another charge. Account usage overlaps run usage; it is not an extra line item. <a href="https://docs.apify.com/actors/running/usage-and-resources#usage" target="_blank" rel="noopener noreferrer" class="">Source: informational usage billing elements</a>.</p>
<p>Use this example to choose the denominator for your own ledger: count outputs that satisfy your task, keep the costs of failed or unusable work, and reconcile account charges separately. It does not establish the cheapest Actor, general runtime reliability or comparative ScrapingAnt savings.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="when-the-apify-platform-is-worth-keeping">When the Apify platform is worth keeping<a href="https://scrapingant.com/blog/apify-pricing#when-the-apify-platform-is-worth-keeping" class="hash-link" aria-label="Direct link to When the Apify platform is worth keeping" title="Direct link to When the Apify platform is worth keeping" translate="no">​</a></h2>
<p>Keep Apify when your chosen Actor's extraction logic and hosted run/storage workflow meet requirements you would otherwise have to build. Reconcile the cost of that workflow before comparing it with a retrieval request. Apify's <a href="https://docs.apify.com/actors/running" target="_blank" rel="noopener noreferrer" class="">Actor running documentation</a> describes the hosted execution model; its <a href="https://docs.apify.com/storage/dataset" target="_blank" rel="noopener noreferrer" class="">dataset documentation</a> describes output storage.</p>
<p>If your application already owns orchestration and validation and needs page retrieval, a separate retrieval API may be worth evaluating. ScrapingAnt supports fetching without a headless browser with <code>browser=false</code> for static content; see its <a href="https://docs.scrapingant.com/request-response-format" target="_blank" rel="noopener noreferrer" class="">request format</a> and <a href="https://docs.scrapingant.com/headless-browser" target="_blank" rel="noopener noreferrer" class="">browser modes</a>. ScrapingAnt does not support localStorage operations; keep browser automation when setting, exporting or restoring that state is required.</p>
<p>For a compatible retrieval task, review <a href="https://scrapingant.com/#pricing" target="_blank" rel="noopener noreferrer" class="">current ScrapingAnt pricing</a> and <a href="https://app.scrapingant.com/signup" target="_blank" rel="noopener noreferrer" class="">create an account</a> if you want to evaluate it. This worksheet provides no evidence of comparative savings or a compatible replacement for a particular Actor.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">Crawlee for Python Tutorial with Examples</h3><p class="description_eu34">Learn how to use Crawlee for Python to build efficient web scrapers for data extraction. This tutorial covers setting up Crawlee, creating web scrapers, handling multiple URLs, implementing custom logic, and best practices.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/crawlee-python-tutorial">→ Read Article</a><span class="readTime_W_dz">5<!-- --> min</span></div></div></div>
<p><strong>Evidence and authorship:</strong> Official pricing sources and offline worksheet examples were checked/tested on <strong>October 2, 2026</strong>. The controlled WCC runs and informational charge snapshots above are dated <strong>October 7, 2026</strong>. AI agents assisted with the worksheet, synthetic fixtures, observations, visuals and drafting; separate AI agents performed calculation and editorial/factual review of the original worksheet article. ScrapingAnt is the publisher and Oleg Kulyk is the publication owner. No separate human review, actual invoice reconciliation or general runtime benchmark is claimed. Recheck official prices and the selected Actor's charging model before using the worksheet for a new period.</p>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="web scraping" term="web scraping"/>
        <category label="api" term="api"/>
        <category label="pricing" term="pricing"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Firecrawl Pricing: Budget for Accepted Results]]></title>
        <id>https://scrapingant.com/blog/firecrawl-pricing</id>
        <link href="https://scrapingant.com/blog/firecrawl-pricing"/>
        <updated>2026-10-02T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Current Firecrawl plans, a recurring-scrape budget model, and a 200-credit fixture study showing why accepted output matters more than HTTP 200.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Firecrawl Pricing: Budget for Accepted Results" src="https://scrapingant.com/blog/assets/images/firecrawl-pricing-19151abd2cd4c91e8d992e26f3d09d54.png" width="1200" height="630" class="img_ev3q"></p>
<p>Firecrawl's basic scrape costs one credit per page. Your useful unit is the result your application can accept: a complete record, rendered content, or a table with the correct cell associations. Budgeting from successful HTTP responses alone misses that difference.</p>
<p>This guide answers the plan-price question first, then models a recurring URL list and explains a <strong>200-credit controlled study</strong>. The study used free credits; it measured credit consumption, <strong>not cash expenditure</strong>. ScrapingAnt publishes this article and sells a competing retrieval API. There is no ScrapingAnt head-to-head test here.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="current-firecrawl-prices">Current Firecrawl prices<a href="https://scrapingant.com/blog/firecrawl-pricing#current-firecrawl-prices" class="hash-link" aria-label="Direct link to Current Firecrawl prices" title="Direct link to Current Firecrawl prices" translate="no">​</a></h2>
<p>Checked <strong>October 2, 2026</strong>, in USD against <a href="https://www.firecrawl.dev/pricing" target="_blank" rel="noopener noreferrer" class="">Firecrawl's official pricing</a> and its <a href="https://www.firecrawl.dev/pricing.md" target="_blank" rel="noopener noreferrer" class="">public pricing document</a>:</p>
<table><thead><tr><th>Plan</th><th style="text-align:right">Monthly credits</th><th style="text-align:right">Monthly bill</th><th style="text-align:right">Annual invoice</th><th style="text-align:right">Concurrency</th></tr></thead><tbody><tr><td>Free</td><td style="text-align:right">1,000</td><td style="text-align:right">$0</td><td style="text-align:right">—</td><td style="text-align:right">2</td></tr><tr><td>Hobby</td><td style="text-align:right">5,000</td><td style="text-align:right">$19</td><td style="text-align:right">$190/year</td><td style="text-align:right">5</td></tr><tr><td>Standard</td><td style="text-align:right">100,000</td><td style="text-align:right">$99</td><td style="text-align:right">$990/year</td><td style="text-align:right">25</td></tr><tr><td>Growth</td><td style="text-align:right">500,000</td><td style="text-align:right">$399</td><td style="text-align:right">$3,990/year</td><td style="text-align:right">50</td></tr><tr><td>Scale</td><td style="text-align:right">1,000,000</td><td style="text-align:right">$749</td><td style="text-align:right">$7,190/year</td><td style="text-align:right">100</td></tr><tr><td>Enterprise</td><td style="text-align:right">Custom</td><td style="text-align:right">Custom</td><td style="text-align:right">Custom</td><td style="text-align:right">Custom</td></tr></tbody></table>
<p>Annual invoices are paid yearly; they are not monthly subscription prices. Free credits refresh monthly. Paid top-ups come in $5 increments: Hobby gets 1,000 credits, Standard 2,000, Growth 2,500 and Scale 5,000 per increment. Check the billing spend cap before enabling automatic top-ups. Hobby, Standard and Growth plan credits do not roll over; Scale carries them for one month. Sources: <a href="https://www.firecrawl.dev/pricing" target="_blank" rel="noopener noreferrer" class="">official billing details</a>.</p>
<p>Basic Scrape and Crawl cost one credit per page. Search is two credits per ten results; Interact is two per browser minute. JSON format on Scrape/Crawl adds four credits per page. A delivered error page can be charged even when a no-result scrape is uncharged. These are documented rates, not measurements from our basic-scrape study. <a href="https://www.firecrawl.dev/pricing" target="_blank" rel="noopener noreferrer" class="">Source: current credit table and failure policy</a>.</p>
<p>Use the live pricing page for endpoint-specific limits and options before buying. This guide models basic retrieval; it does not price a mixed Agent, extraction, interaction or monitoring workload.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="model-a-recurring-scrape-bill">Model a recurring scrape bill<a href="https://scrapingant.com/blog/firecrawl-pricing#model-a-recurring-scrape-bill" class="hash-link" aria-label="Direct link to Model a recurring scrape bill" title="Direct link to Model a recurring scrape bill" translate="no">​</a></h2>
<p>Keep three quantities separate:</p>
<ol>
<li class=""><strong>Documented prices:</strong> the plan and top-up rules above.</li>
<li class=""><strong>Observed credits:</strong> the units consumed by recorded calls below.</li>
<li class=""><strong>Modeled cash:</strong> a hypothetical subscription bill using stated workload assumptions.</li>
</ol>
<p>For a known URL list, start with scheduled URLs × runs per month. Add billed retry attempts, then multiply by the credit cost of the selected operation. Count an accepted scheduled output once, even if it needed multiple attempts. If freshness matters, acceptance must also check the expected revision or maximum permissible age.</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">credits needed = (URLs × runs + billed retry attempts) × credits per attempt</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">modeled monthly bill = selected plan price + $5 × required top-up increments</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">cost per accepted output = modeled monthly bill ÷ accepted scheduled outputs</span><br></div></code></pre></div></div>
<p>This simplified model starts a billing period with the plan allowance and no existing purchased credits. It excludes taxes, add-ons, other endpoints and your engineering/storage costs. A plan may also be required for concurrency or rate limits even when its credit allowance is underused.</p>
<p>The <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/a684c9cfeafcadad3986c11349092d74ae4b688a/examples/firecrawl-pricing" target="_blank" rel="noopener noreferrer" class="">offline budget model and tests</a> let you change URL count, cadence, retries, credits per attempt, accepted output and selected plan. Here is its tested default scenario, <strong>not an observed workload or invoice</strong>:</p>
<table><thead><tr><th>Hypothetical workload</th><th style="text-align:right">Standard, monthly billing</th></tr></thead><tbody><tr><td>1,000 URLs × 30 runs, no retries</td><td style="text-align:right">30,000 credits</td></tr><tr><td>Assumed accepted outputs</td><td style="text-align:right">27,000</td></tr><tr><td>Modeled bill</td><td style="text-align:right">$99</td></tr><tr><td>Unused monthly plan allowance</td><td style="text-align:right">70,000 credits</td></tr><tr><td>Modeled cost per accepted output</td><td style="text-align:right">About $0.00367</td></tr></tbody></table>
<p>The assumed 27,000 outputs do not imply a measured 90% acceptance rate. Dividing $99 by the entire 100,000-credit allowance gives $0.00099 per credit at full utilization; it does not describe this partially used workload. At only 3,000 accepted one-credit outputs on Standard, the same $99 minimum means $0.033 each.</p>
<p>Compare plans with the same workload. For the 30,000-credit example, Hobby plus 25 top-ups models $144, while Standard models $99. At the Hobby allowance boundary, 5,000 credits model $19; 5,001 model $24 because top-ups are discrete. The scorer tests these boundaries, Free's allowance ceiling and <strong>zero accepted output, whose unit-cost ratio is undefined</strong>. These calculations do not establish which plan fits your concurrency needs.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-counts-as-an-accepted-result">What counts as an accepted result?<a href="https://scrapingant.com/blog/firecrawl-pricing#what-counts-as-an-accepted-result" class="hash-link" aria-label="Direct link to What counts as an accepted result?" title="Direct link to What counts as an accepted result?" translate="no">​</a></h2>
<p>Define the application contract before measuring cost. A source-preserving retrieval and a usable business record can be different outcomes:</p>
<table><thead><tr><th>Task</th><th>Acceptance check</th><th>Rejection example</th></tr></thead><tbody><tr><td>Catalog ingestion</td><td>Required identities, numeric prices and currency</td><td>Correctly preserved “price unavailable”</td></tr><tr><td>Delayed page</td><td>Expected rendered text and loaded marker</td><td>HTTP 200 with initial placeholder</td></tr><tr><td>Labeled table data</td><td>Correct header/cell associations and required visibility</td><td>Flattened spans or a hidden row included</td></tr><tr><td>Fresh recurring document</td><td>Expected revision or acceptable age</td><td>Cached content older than your contract allows</td></tr></tbody></table>
<p>The last row is a recommended contract, <strong>not a measured freshness result</strong>. Our origin content stayed unchanged. Do not count delivered 404s, duplicates or incomplete records as accepted fresh documents simply because a request completed.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="observed-credits-per-accepted-output">Observed credits per accepted output<a href="https://scrapingant.com/blog/firecrawl-pricing#observed-credits-per-accepted-output" class="hash-link" aria-label="Direct link to Observed credits per accepted output" title="Direct link to Observed credits per accepted output" translate="no">​</a></h2>
<p>On <strong>October 1, 2026</strong>, the study made <strong>24 exploratory calls plus 176 preregistered calls</strong> against six owned synthetic fixtures and one missing path on a single GitHub Pages origin. Every billed call requested basic proxy, Markdown and HTML, without AI extraction or premium formats, and reported one credit. Total observed consumption was <strong>200 credits</strong>. One extra connector-catalog rejection happened before transport and had a separately verified zero debit; it had no reported credit field and was not a returned fixture result.</p>
<p>The expansion comprised 112 fresh run slots across eight settings, 32 delayed-content pair calls, 16 permanent-404 diagnostic calls and 16 fresh/cache calls. The pilot stays separate from the balanced setting results. The <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/a684c9cfeafcadad3986c11349092d74ae4b688a/examples/firecrawl-pricing" target="_blank" rel="noopener noreferrer" class="">public packet</a> includes the dated design, sanitized attempt CSV, exact output strings, pinned fixture sources and offline acceptance checks.</p>
<p><img decoding="async" loading="lazy" alt="Observed credits per accepted output: complete catalog 14/14 equals 1; delayed content with wait 14/14 equals 1; corrective pairs 16/8 equals 2; same-setting pairs 16/0 is undefined." src="https://scrapingant.com/blog/assets/images/credits-2d8e58a402c2d5821d7566aceef44faa.png" width="1520" height="2388" class="img_ev3q"></p>
<p><em>Figure 1. Separate observed subsets, not a combined accuracy rate. Both attempts in each retry pair stay in the numerator. Source: <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/a684c9cfeafcadad3986c11349092d74ae4b688a/examples/firecrawl-pricing/visuals" target="_blank" rel="noopener noreferrer" class="">chart data and renderer</a>.</em></p>
<table><thead><tr><th>Observed subset and contract</th><th style="text-align:right">Credits</th><th style="text-align:right">Accepted outputs</th><th style="text-align:right">Credits per accepted output</th></tr></thead><tbody><tr><td>Complete catalog, basic preservation</td><td style="text-align:right">14</td><td style="text-align:right">14</td><td style="text-align:right">1</td></tr><tr><td>Delayed content, 2,500 ms wait</td><td style="text-align:right">14</td><td style="text-align:right">14</td><td style="text-align:right">1</td></tr><tr><td>Corrective retry pairs, rendered content</td><td style="text-align:right">16</td><td style="text-align:right">8</td><td style="text-align:right">2</td></tr><tr><td>Same-setting retry pairs, rendered content</td><td style="text-align:right">16</td><td style="text-align:right">0</td><td style="text-align:right">Undefined</td></tr></tbody></table>
<p>These are different fixture contracts, not a provider-wide success rate. The missing-price catalog passed basic source-preservation checks in 14/14 balanced runs while supplying zero outputs meeting the separate two-numeric-prices requirement. Rejecting those records is an application decision; Firecrawl did not invent the missing price.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="a-billed-http-200-can-be-a-placeholder">A billed HTTP 200 can be a placeholder<a href="https://scrapingant.com/blog/firecrawl-pricing#a-billed-http-200-can-be-a-placeholder" class="hash-link" aria-label="Direct link to A billed HTTP 200 can be a placeholder" title="Direct link to A billed HTTP 200 can be a placeholder" translate="no">​</a></h3>
<p>The delayed fixture changes its DOM after 1,500 ms. Default requests omitted <code>waitFor</code>. In eight same-setting pairs, the first and second calls both missed the rendered-content contract. In eight corrective pairs, failed first calls were followed by <code>waitFor: 2500</code>; all eight second calls passed. The corrective strategy consumed <strong>16 credits for 8 accepted outputs</strong>, or <strong>2 credits each</strong>, including the first failures.</p>
<p><img decoding="async" loading="lazy" alt="Saved default-wait HTML contains Web Scraping is hard despite target HTTP 200; saved waitFor 2500 HTML contains the expected delayed text and loaded marker, also HTTP 200." src="https://scrapingant.com/blog/assets/images/delayed-7eca5355e660011e2d8d0bb3567b4dde.png" width="1520" height="1780" class="img_ev3q"></p>
<p><em>Figure 2. Faithful offline rendering of saved outputs 56 and 57, with display styling added. Both reported target status 200 and one credit. Source: <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/a684c9cfeafcadad3986c11349092d74ae4b688a/examples/firecrawl-pricing" target="_blank" rel="noopener noreferrer" class="">saved captures and rendering code</a>. These are presentations of stored HTML, not screenshots taken during the API calls.</em></p>
<p>Text equivalent: the default output says <code>Web Scraping is hard</code>. The corrected output says <code>I ❤️ ScrapingAnt (after 1500 ms)</code> and <code>loaded</code>. Those strings belong to the synthetic fixture. They are not customer feedback or endorsements.</p>
<p>Set readiness to match your page and validate the result. The fixture does not establish a universal wait duration or retry probability. Repeating identical settings consumed 16 credits with zero accepted outputs; its ratio is undefined, not zero.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="html-and-markdown-can-serve-different-table-contracts">HTML and Markdown can serve different table contracts<a href="https://scrapingant.com/blog/firecrawl-pricing#html-and-markdown-can-serve-different-table-contracts" class="hash-link" aria-label="Direct link to HTML and Markdown can serve different table contracts" title="Direct link to HTML and Markdown can serve different table contracts" translate="no">​</a></h3>
<p>Both balanced table modes preserved all <strong>12 HTML table structures</strong> in 14/14 runs each, counting the nested table. Markdown retained tested ordinary values but had uneven pipe widths in the multirow-header, body-span and nested sections. It included the hidden <code>AF-2</code> row in every balanced run, so each mode passed the strict visible-only Markdown contract in <strong>0/14</strong> runs.</p>
<p><img decoding="async" loading="lazy" alt="Returned HTML retains Region spanning two rows and each year spanning Q1 and Q2; returned Markdown shows unequal pipe-row cell counts." src="https://scrapingant.com/blog/assets/images/table-c3a04ecd0ad0297bea6fc0b04028627d.png" width="1520" height="2436" class="img_ev3q"></p>
<p><em>Figure 3. Exact two-row-header extract from saved output 36: rendered HTML above, unchanged Markdown text below. Styling is added; cells and values are unchanged. Source: <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/a684c9cfeafcadad3986c11349092d74ae4b688a/examples/firecrawl-pricing" target="_blank" rel="noopener noreferrer" class="">table render and stored outputs</a>.</em></p>
<p>Text equivalent: the HTML header associates <code>2025</code> with Q1/Q2 and <code>2026</code> with Q1/Q2; North has values 10, 12, 14 and 16, and South has 7, 8, 9 and 11. The Markdown's first header row has three cells, its quarter row four, and its data rows five. Preserve HTML and reconstruct labels when your ingestion contract requires those associations.</p>
<p>Pipe Markdown has no native span syntax. These observations do not establish a Firecrawl-specific defect, and <code>onlyMainContent</code> is not a visible-only output guarantee. Define which format you need before counting an output as accepted.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="caching-and-error-pages-still-need-acceptance-checks">Caching and error pages still need acceptance checks<a href="https://scrapingant.com/blog/firecrawl-pricing#caching-and-error-pages-still-need-acceptance-checks" class="hash-link" aria-label="Direct link to Caching and error pages still need acceptance checks" title="Direct link to Caching and error pages still need acceptance checks" translate="no">​</a></h3>
<p>Eight fresh/cache pairs produced eight reported cache hits with identical content; each cache repeat still consumed one credit. The source never changed, so this tests neither invalidation nor freshness after an edit. Sixteen expansion requests to the permanently missing path returned billed 404 pages. They provided no requested content; repeating a permanent 404 was a diagnostic exercise, not a recommended retry policy.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="reproduce-the-calculations-without-api-credits">Reproduce the calculations without API credits<a href="https://scrapingant.com/blog/firecrawl-pricing#reproduce-the-calculations-without-api-credits" class="hash-link" aria-label="Direct link to Reproduce the calculations without API credits" title="Direct link to Reproduce the calculations without API credits" translate="no">​</a></h2>
<p>Download the <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/a684c9cfeafcadad3986c11349092d74ae4b688a/examples/firecrawl-pricing" target="_blank" rel="noopener noreferrer" class="">pinned packet</a>, open its directory and run the README commands. You need Python 3.10 or newer. Installing the pinned dependency needs package-download access; replay requires no credentials and makes no provider calls.</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3 -m venv .venv</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">source .venv/bin/activate</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3 -m pip install -r requirements.txt</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">./run.sh</span><br></div></code></pre></div></div>
<p>The stored replay output is:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">Offline replay: 200 billed calls, 200 credits; 24 pilot + 176 preregistered; 201 connector attempts.</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">delayed-corrective-pair: 16 credits / 8 accepted = 2.0</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">delayed-same-pair: 16 credits / 0 accepted = undefined</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Eight balanced settings: 14/14 basic preservation per setting; stricter contracts reported separately.</span><br></div></code></pre></div></div>
<p>The script writes the recomputed CSV and JSON under <code>expected_output/</code>. If an input hash or count fails, stop and obtain the unchanged pinned packet; do not interpret a partial replay as the full study. To model your own hypothetical monthly workload:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3 budget.py --plan Standard --urls 1000 --runs 30 --retry-attempts 0 --credits-per-attempt 1 --accepted 27000</span><br></div></code></pre></div></div>
<p>Use your own observed accepted count when available. Do not substitute our fixture results for your production acceptance rate.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="when-to-keep-firecrawl-and-where-scrapingant-fits">When to keep Firecrawl, and where ScrapingAnt fits<a href="https://scrapingant.com/blog/firecrawl-pricing#when-to-keep-firecrawl-and-where-scrapingant-fits" class="hash-link" aria-label="Direct link to When to keep Firecrawl, and where ScrapingAnt fits" title="Direct link to When to keep Firecrawl, and where ScrapingAnt fits" translate="no">​</a></h2>
<p>Firecrawl's <a href="https://docs.firecrawl.dev/features/scrape" target="_blank" rel="noopener noreferrer" class="">Scrape documentation</a> describes Markdown, HTML and structured output alongside rendering and caching controls. On the article fixture, main-content mode removed the tested cookie, aside and footer markers in 14/14 balanced runs, while whole-page mode retained them; both preserved the tested article content. That is a useful bounded strength for document ingestion.</p>
<p>Keep Firecrawl when its output meets your contract and its broader workflow fits your application. A low retrieval-only unit price does not replace required search, crawl orchestration, interaction or extraction. This page owns the <strong>budgeting decision</strong>; choosing an alternative also requires compatibility, freshness, migration and rollback tests that this study did not run.</p>
<p>For a known-URL retrieval task, you can evaluate ScrapingAnt against your own acceptance checks. For non-Google targets, its documented datacenter request costs are <strong>1 credit without a browser</strong> and <strong>10 credits with JavaScript rendering</strong>. Those are ScrapingAnt units, not interchangeable Firecrawl credits. See <a href="https://docs.scrapingant.com/credits-cost" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt's credit costs</a>. No matched ScrapingAnt costs or outputs were measured here, so there is no supported savings or superiority claim.</p>
<p>The <a href="https://scrapingant.com/firecrawl-alternative" target="_blank" rel="noopener noreferrer" class="">Firecrawl alternative comparison</a> puts both price lists side by side, shows what 1,000 pages cost by request mode, and lists the Firecrawl features ScrapingAnt does not replace.</p>
<p>If that retrieval scope fits, <a href="https://app.scrapingant.com/signup" target="_blank" rel="noopener noreferrer" class="">create a ScrapingAnt account</a> and validate a small compatible URL sample before committing a recurring workload. Keep Firecrawl when it already does the job; ScrapingAnt is not needed to replay this packet or calculate Firecrawl's bill.</p>
<p>For the broader planning question, see <a class="" href="https://scrapingant.com/blog/cost-efficient-web-scraping-api">cost-efficient web scraping</a>.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">The Benefits of Using Markdown for Efficient Data Extraction</h3><p class="description_eu34">Markdown is a popular choice for web scraping and data extraction due to its simplicity and versatility. This article explores the benefits of using Markdown for efficient data extraction and how it can help you streamline your web scraping projects.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/markdown-efficient-data-extraction">→ Read Article</a><span class="readTime_W_dz">20<!-- --> min</span></div></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="study-limitations-and-maintenance">Study limitations and maintenance<a href="https://scrapingant.com/blog/firecrawl-pricing#study-limitations-and-maintenance" class="hash-link" aria-label="Direct link to Study limitations and maintenance" title="Direct link to Study limitations and maintenance" translate="no">​</a></h2>
<p>All targets were synthetic, owned and on one origin. Repeated sequential requests shared one account and a short session; more repetitions do not add website diversity or establish statistical independence. The eight balanced settings passed their distinct basic checklists in 14/14 slots each; pooling them would create a misleading overall accuracy number.</p>
<p>There was no production anti-bot corpus, changed-origin freshness test, ScrapingAnt comparison, AI extraction test, throughput/SLA measurement or engine benchmark. Receipt, API HTTP-status and engine-timing fields were not surfaced. Target status and connector completion are distinct from useful output. Free-credit consumption is evidence about units, not an invoice. Prices and service behavior can change; recheck official billing inputs before making a purchase decision.</p>
<p>Offline examples tested on <strong>2026-10-02</strong> with <strong>Python 3.10.2 and lxml 6.1.1</strong>; stored retrievals were collected on <strong>2026-10-01</strong>. Code and data: <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/a684c9cfeafcadad3986c11349092d74ae4b688a/examples/firecrawl-pricing" target="_blank" rel="noopener noreferrer" class="">versioned public evidence packet</a>.</p>
<p><em>AI agents assisted with drafting, export code, offline analysis, figures and automated editorial/factual review. ScrapingAnt is the publisher and Oleg Kulyk is the publication owner. No separate human review is claimed.</em></p>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="web scraping" term="web scraping"/>
        <category label="api" term="api"/>
        <category label="pricing" term="pricing"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Find Elements with Playwright in Python: Extract Validated Records]]></title>
        <id>https://scrapingant.com/blog/playwright-find-elements</id>
        <link href="https://scrapingant.com/blog/playwright-find-elements"/>
        <updated>2026-09-28T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Find one or many elements with Python Playwright locators, wait for complete records, and extract text, attributes and JSON with tested examples.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Find Elements with Playwright in Python: Extract Validated Records" src="https://scrapingant.com/blog/assets/images/playwright-find-elements-1f03f2a9ead373be83a6f8298567cfef.png" width="1200" height="630" class="img_ev3q"></p>
<p>Use <code>page.locator("#catalog &gt; .product")</code> to find repeated elements in Python Playwright. Narrow to an intended record before reading a single value, or call <code>all()</code> to iterate the matches after the list has finished loading. To build records in the browser, use <code>evaluate_all()</code> with a JavaScript projection.</p>
<p>Finding a match is only part of extraction. In the examples below, <code>.first</code> returned a sponsored product, while an early <code>count()</code> found a loading placeholder. Both operations worked; both selected the wrong data. The runnable example waits for the catalog's ready state and checks every SKU, title, currency, price and link against independently specified records.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="choose-a-locator-and-decide-how-many-matches-you-expect">Choose a locator and decide how many matches you expect<a href="https://scrapingant.com/blog/playwright-find-elements#choose-a-locator-and-decide-how-many-matches-you-expect" class="hash-link" aria-label="Direct link to Choose a locator and decide how many matches you expect" title="Direct link to Choose a locator and decide how many matches you expect" translate="no">​</a></h2>
<p>These are the queries used by the local catalog example. They assume <code>page</code> has loaded the fixture and its catalog is ready.</p>
<table><thead><tr><th>Task</th><th>Python Playwright expression</th></tr></thead><tbody><tr><td>Find all catalog records by CSS</td><td><code>page.locator("#catalog &gt; .product")</code></td></tr><tr><td>Find the named list and its items by role</td><td><code>page.get_by_role("list", name="Catalog", exact=True).get_by_role("listitem")</code></td></tr><tr><td>Find an exact title</td><td><code>page.get_by_text("Café Mug", exact=True)</code></td></tr><tr><td>Find a record with an explicit attribute</td><td><code>page.locator('#catalog &gt; .product[data-badge="featured"]')</code></td></tr><tr><td>Find the catalog items with XPath</td><td><code>page.locator('xpath=//ul[@id="catalog"]/li[@class="product"]')</code></td></tr><tr><td>Enter the fixture's frame</td><td><code>page.frame_locator("#catalog-frame").locator(".product")</code></td></tr></tbody></table>
<p>A locator describes how to find elements. A singular operation such as <code>inner_text()</code> requires an unambiguous match. In the test catalog, this read matched four titles and raised a strict-mode error:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">target </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">locator</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"#main .product .title"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">target</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">inner_text</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><br></div></code></pre></div></div>
<p>The matching records were <code>AD-000</code>, <code>K-101</code>, <code>M-202</code> and <code>T-303</code>. The first was an advertisement outside the catalog. Changing the query to <code>.first</code> suppressed the ambiguity but selected that advertisement:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">locator</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"#main .product"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">first</span><br></div></code></pre></div></div>
<p>Its extracted title was <code>Sponsored kettle</code>, with price <code>999.00</code> in <code>USD</code>. Narrowing the parent scope to <code>#catalog &gt; .product</code> produced the intended records instead. Treat <code>.first</code> as an explicit choice of position, not a repair for a selector that matches too much. Playwright's <a href="https://playwright.dev/python/docs/locators#strictness" target="_blank" rel="noopener noreferrer" class="">locator guide</a> explains singular-operation strictness.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">How to Find Elements With Selenium in Python</h3><p class="description_eu34">Find one or many elements with Selenium Python. Extract complete records with scoped selectors, explicit waits, stale recovery, frames and shadow roots.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/selenium-python-find-element">→ Read Article</a><span class="readTime_W_dz">9<!-- --> min</span></div></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="run-a-complete-locator-to-record-example">Run a complete locator-to-record example<a href="https://scrapingant.com/blog/playwright-find-elements#run-a-complete-locator-to-record-example" class="hash-link" aria-label="Direct link to Run a complete locator-to-record example" title="Direct link to Run a complete locator-to-record example" translate="no">​</a></h2>
<p>The example repository includes the HTML fixture, a loopback HTTP server, extraction code and a literal record validator. It needs no API key and does not scrape an external site.</p>
<p>Use Python 3.12. Clone the repository and select the tested source revision:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">git clone https://github.com/ScrapingAnt/scrapingant-examples.git</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">cd scrapingant-examples</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">git checkout 74638f75381bdf2ff0652f1f07bd98f65186b40d</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">cd examples/playwright-find-elements</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3.12 -m venv .venv</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">source .venv/bin/activate</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python -m pip install -r requirements.txt</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python -m playwright install chromium firefox</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python quickstart.py</span><br></div></code></pre></div></div>
<p>The dependency lock pins Playwright to <code>1.63.0</code>. Browser installation downloads the corresponding binaries; Linux may also need the <a href="https://playwright.dev/python/docs/browsers#install-system-dependencies" target="_blank" rel="noopener noreferrer" class="">documented browser system dependencies</a>.</p>
<p>The runnable <a href="https://github.com/ScrapingAnt/scrapingant-examples/blob/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/playwright-find-elements/quickstart.py" target="_blank" rel="noopener noreferrer" class=""><code>quickstart.py</code></a> starts the fixture server, launches Chromium, waits for the catalog, and validates the returned records. Its extraction and cleanup are:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token keyword" style="color:rgb(86, 156, 214)">with</span><span class="token plain"> fixture_server</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">as</span><span class="token plain"> origin</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> sync_playwright</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">as</span><span class="token plain"> playwright</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    browser </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> playwright</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">chromium</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">launch</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">headless</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token boolean">True</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> chromium_sandbox</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token boolean">True</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token keyword" style="color:rgb(86, 156, 214)">try</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        page </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> browser</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">new_page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">goto</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">origin </span><span class="token operator" style="color:rgb(212, 212, 212)">+</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/catalog.html"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> wait_until</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token string" style="color:rgb(206, 145, 120)">"load"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        wait_for_catalog</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        records </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">locator</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"#catalog &gt; .product"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">evaluate_all</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">PROJECTION</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        validate_records</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">records</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> CATALOG</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token keyword" style="color:rgb(86, 156, 214)">finally</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        browser</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">close</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><br></div></code></pre></div></div>
<p>This is an excerpt from the complete file, not a standalone replacement for it: <code>fixture_server</code>, <code>wait_for_catalog</code>, <code>PROJECTION</code> and the independent <code>CATALOG</code> oracle come from the modules included beside it. The server context also closes its listener when the program exits. Chromium's sandbox stays enabled.</p>
<p>The captured output was:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"sku"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"K-101"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"title"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Copper Kettle"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"currency"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"USD"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"24.50"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"href"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/products/kettle"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"sku"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"M-202"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"title"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Café Mug"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"currency"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"EUR"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"12.00"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"href"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/products/mug"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"sku"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"T-303"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"title"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Tea &amp; Honey"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"currency"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"GBP"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"8.75"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"href"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/products/tea"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><br></div></code></pre></div></div>
<p>The prices stay as decimal strings. Parsing real prices needs an explicit contract for the site's formatting; this fixture does not test a general currency parser.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="wait-for-the-dataset-before-calling-count-or-all">Wait for the dataset before calling count or all<a href="https://scrapingant.com/blog/playwright-find-elements#wait-for-the-dataset-before-calling-count-or-all" class="hash-link" aria-label="Direct link to Wait for the dataset before calling count or all" title="Direct link to Wait for the dataset before calling count or all" translate="no">​</a></h2>
<p>The fixture initially contains one placeholder record. A controlled load then replaces it with the catalog. While loading was deliberately held open, <code>count()</code> returned <code>1</code>, <code>len(rows.all())</code> returned <code>1</code>, and extraction returned this record:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"sku"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"PENDING"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"title"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Loading catalog"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"currency"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"XXX"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"0.00"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"href"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/pending"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><br></div></code></pre></div></div>
<p>That is a concrete reason to separate the current match count from dataset readiness. Playwright documents that <a href="https://playwright.dev/python/docs/api/class-locator#locator-all" target="_blank" rel="noopener noreferrer" class=""><code>locator.all()</code> returns the matches currently present</a>, without waiting for a changing list to finish loading.</p>
<p>The fixture's readiness helper is:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token keyword" style="color:rgb(86, 156, 214)">def</span><span class="token plain"> </span><span class="token function" style="color:rgb(220, 220, 170)">wait_for_catalog</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">get_by_role</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"button"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> name</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token string" style="color:rgb(206, 145, 120)">"Load catalog"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> exact</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token boolean">True</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">click</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token comment" style="color:rgb(106, 153, 85)"># This signal belongs to the fixture application, and means all three rows were installed.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    expect</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">locator</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'#catalog[data-state="ready"]'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">to_have_count</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token number" style="color:rgb(181, 206, 168)">1</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    expect</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">locator</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"#catalog"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">to_have_attribute</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"aria-busy"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"false"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    expect</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">locator</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"#catalog &gt; .product"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">to_have_count</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token number" style="color:rgb(181, 206, 168)">3</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><br></div></code></pre></div></div>
<p>Here, <code>data-state="ready"</code> is meaningful because the fixture sets it after installing the complete list. Choose an equivalent application signal on your target. Use a known record count only when the application or your input specifies it; an arbitrary count cannot prove an infinite or virtualized list is complete.</p>
<p>After readiness, the row-by-row approach in <code>extract.py</code> reads the fields through locators:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token keyword" style="color:rgb(86, 156, 214)">def</span><span class="token plain"> </span><span class="token function" style="color:rgb(220, 220, 170)">locator_records</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">rows</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token keyword" style="color:rgb(86, 156, 214)">return</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token string" style="color:rgb(206, 145, 120)">"sku"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> row</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">get_attribute</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"data-sku"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token string" style="color:rgb(206, 145, 120)">"title"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> row</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">locator</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">".title"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">inner_text</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">strip</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token string" style="color:rgb(206, 145, 120)">"currency"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> row</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">locator</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">".price"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">get_attribute</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"data-currency"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token string" style="color:rgb(206, 145, 120)">"price"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> row</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">locator</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">".price"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">inner_text</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">strip</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token string" style="color:rgb(206, 145, 120)">"href"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> row</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">locator</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">".title"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">get_attribute</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"href"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">for</span><span class="token plain"> row </span><span class="token keyword" style="color:rgb(86, 156, 214)">in</span><span class="token plain"> rows</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token builtin" style="color:rgb(86, 156, 214)">all</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><br></div></code></pre></div></div>
<p>This returned the same three records as the quickstart. Validate the result after reading it: the supplied validator rejects missing records, duplicate SKUs, altered prices, unknown fields and invalid value types. In a real scraper, replace the fixture's exact record oracle with the dataset invariants you can actually establish.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="project-all-matched-elements-into-json-compatible-records">Project all matched elements into JSON-compatible records<a href="https://scrapingant.com/blog/playwright-find-elements#project-all-matched-elements-into-json-compatible-records" class="hash-link" aria-label="Direct link to Project all matched elements into JSON-compatible records" title="Direct link to Project all matched elements into JSON-compatible records" translate="no">​</a></h2>
<p>The quickstart uses <code>evaluate_all()</code> to pass the matched nodes to one JavaScript function. This is the exact projection from <code>extract.py</code>:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">PROJECTION </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token triple-quoted-string string" style="color:rgb(206, 145, 120)">"""rows =&gt; rows.map(row =&gt; ({</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token triple-quoted-string string" style="color:rgb(206, 145, 120)">    sku: row.getAttribute('data-sku'),</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token triple-quoted-string string" style="color:rgb(206, 145, 120)">    title: row.querySelector('.title').innerText.trim(),</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token triple-quoted-string string" style="color:rgb(206, 145, 120)">    currency: row.querySelector('.price').getAttribute('data-currency'),</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token triple-quoted-string string" style="color:rgb(206, 145, 120)">    price: row.querySelector('.price').innerText.trim(),</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token triple-quoted-string string" style="color:rgb(206, 145, 120)">    href: row.querySelector('.title').getAttribute('href')</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token triple-quoted-string string" style="color:rgb(206, 145, 120)">}))"""</span><br></div></code></pre></div></div>
<p><code>page.locator(...)</code> creates the Playwright locator. Inside the projection, <code>row.querySelector(...)</code> is a DOM query running in the browser. The expression uses JavaScript methods such as <code>getAttribute</code>, not Python's <code>get_attribute</code>. Browser evaluation and the Python program have <a href="https://playwright.dev/python/docs/evaluating" target="_blank" rel="noopener noreferrer" class="">separate execution environments</a>.</p>
<p>The projection ran after the same readiness checks as the locator loop and returned the same full record tuples. We did not measure a speed advantage. Use it when a browser-side mapping expresses your extraction clearly; it does not replace readiness or result validation.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">Web Scraping with Playwright Series Part 2 - Building a Scraper</h3><p class="description_eu34">Build a Python Playwright scraper with DOM selection, scrolling and network responses. Learn the limits of loading heuristics and resource blocking.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/web-scraping-playwright-python-part-2">→ Read Article</a><span class="readTime_W_dz">10<!-- --> min</span></div></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="understand-filters-replacement-nodes-and-text-reads">Understand filters, replacement nodes and text reads<a href="https://scrapingant.com/blog/playwright-find-elements#understand-filters-replacement-nodes-and-text-reads" class="hash-link" aria-label="Direct link to Understand filters, replacement nodes and text reads" title="Direct link to Understand filters, replacement nodes and text reads" translate="no">​</a></h2>
<p>Three small distinctions caused or exposed wrong assumptions in the fixture.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="write-has-filters-relative-to-each-row">Write has filters relative to each row<a href="https://scrapingant.com/blog/playwright-find-elements#write-has-filters-relative-to-each-row" class="hash-link" aria-label="Direct link to Write has filters relative to each row" title="Direct link to Write has filters relative to each row" translate="no">​</a></h3>
<p>The first attempt used a title locator starting at <code>#catalog</code> inside <code>rows.filter(has=...)</code>. It returned no records: each candidate row has no descendant catalog container. That failed capture is preserved with the code.</p>
<p>The corrected query was:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">title </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">get_by_text</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"Café Mug"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> exact</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token boolean">True</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">rows</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token builtin" style="color:rgb(86, 156, 214)">filter</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">has</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain">title</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><br></div></code></pre></div></div>
<p><code>rows</code> is the scoped catalog locator. The inner locator is evaluated relative to each candidate, so this selected the <code>M-202</code> record. The <a href="https://playwright.dev/python/docs/api/class-locator#locator-filter" target="_blank" rel="noopener noreferrer" class="">filter documentation</a> describes this relative scope.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="a-reused-locator-can-resolve-a-replacement-node">A reused locator can resolve a replacement node<a href="https://scrapingant.com/blog/playwright-find-elements#a-reused-locator-can-resolve-a-replacement-node" class="hash-link" aria-label="Direct link to A reused locator can resolve a replacement node" title="Direct link to A reused locator can resolve a replacement node" translate="no">​</a></h3>
<p>The replacement case saved a locator for the kettle title, read <code>Copper Kettle (archived)</code>, replaced the record's DOM node, and read the same locator again. It returned <code>Copper Kettle</code>. The identity check confirmed that the underlying node had changed.</p>
<p>This is useful when an application replaces its markup. Still wait for the relevant state transition: in the test, the second read followed an assertion that the catalog had reached <code>data-state="replaced"</code>.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="choose-text-and-attribute-methods-intentionally">Choose text and attribute methods intentionally<a href="https://scrapingant.com/blog/playwright-find-elements#choose-text-and-attribute-methods-intentionally" class="hash-link" aria-label="Direct link to Choose text and attribute methods intentionally" title="Direct link to Choose text and attribute methods intentionally" translate="no">​</a></h3>
<p>The diagnostic reads produced these exact differences:</p>
<table><thead><tr><th>Read</th><th>Captured value</th></tr></thead><tbody><tr><td>Title <code>inner_text()</code></td><td><code>Tea &amp; Honey</code></td></tr><tr><td>Title <code>text_content()</code></td><td><code>Tea &amp; HoneyHIDDEN</code></td></tr><tr><td>Link <code>get_attribute("href")</code></td><td><code>/products/kettle</code></td></tr><tr><td>Link DOM <code>href</code> property</td><td><code>{origin}/products/kettle</code></td></tr><tr><td>Present <code>data-badge</code> attribute</td><td><code>featured</code></td></tr><tr><td>Missing <code>data-badge</code> attribute</td><td><code>None</code> in Python, <code>null</code> in the captured JSON</td></tr></tbody></table>
<p>The title contained a hidden descendant with the text <code>HIDDEN</code>. The URL property resolved the relative link; the captured output substitutes <code>{origin}</code> for the fixture server's random local origin. Decide whether your output needs visible text, all descendant text, the raw attribute, or its resolved property.</p>
<p>The selector families also selected different sets. Role lookup excluded the hidden offer button by default; <code>include_hidden=True</code> included it. CSS and XPath selected both offer buttons. An exact text query matched the fixture's whitespace-normalized <code>Spaced title</code> example. These are reasons to choose a query by its meaning, rather than swapping locator syntax and assuming identical results.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="enter-frames-and-distinguish-shadow-root-queries">Enter frames and distinguish shadow-root queries<a href="https://scrapingant.com/blog/playwright-find-elements#enter-frames-and-distinguish-shadow-root-queries" class="hash-link" aria-label="Direct link to Enter frames and distinguish shadow-root queries" title="Direct link to Enter frames and distinguish shadow-root queries" translate="no">​</a></h2>
<p>The frame case used <code>page.frame_locator("#catalog-frame").locator(".product")</code> and extracted <code>F-404</code>, the <code>Frame spoon</code> record. The main-document selection contained the four light-DOM records, including the decoy, and did not include that framed record.</p>
<p>For the open shadow root, <code>page.locator("#shadow-host .product")</code> extracted <code>S-505</code>, <code>Shadow strainer</code>. Raw <code>document.querySelectorAll('.product')</code> returned the four light-DOM SKUs instead; scoped XPath returned no shadow matches. Playwright's <a href="https://playwright.dev/python/docs/locators#locate-in-shadow-dom" target="_blank" rel="noopener noreferrer" class="">shadow-DOM documentation</a> specifies the XPath exception and excludes closed shadow roots.</p>
<p>This matters when moving extraction code into <code>evaluate_all()</code> or another browser-JavaScript interface: ordinary DOM selectors do not acquire Playwright's locator behavior. The supplied fixture uses an open root and a same-origin frame. It does not establish cross-origin frame or closed-root behavior.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-the-checks-covered">What the checks covered<a href="https://scrapingant.com/blog/playwright-find-elements#what-the-checks-covered" class="hash-link" aria-label="Direct link to What the checks covered" title="Direct link to What the checks covered" translate="no">​</a></h2>
<p>Run the full local matrix from the example directory:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">./run.sh</span><br></div></code></pre></div></div>
<p>It writes rerun artifacts to ignored <code>run_output/</code>, leaving the captured <code>expected_output/</code> files intact. The tested environment was Python <code>3.12.10</code>, Playwright <code>1.63.0</code>, Chromium <code>153.0.8010.12</code> and Firefox <code>155.0</code> on macOS <code>26.6.2</code> arm64.</p>
<table><thead><tr><th>Check group</th><th style="text-align:right">Captured denominator</th><th>Result</th></tr></thead><tbody><tr><td>Selection/extraction cases, including wrong controls</td><td style="text-align:right">12 cases × 3 rounds × 2 browsers = 72 observations</td><td>Every observation matched its declared expected outcome</td></tr><tr><td>Separate method/error diagnostics</td><td style="text-align:right">4 cases × 3 rounds × 2 browsers = 24 observations</td><td>Every diagnostic matched its expected outcome</td></tr><tr><td>Offline record, summary and artifact-integrity regression tests</td><td style="text-align:right">25 test methods</td><td>Passed</td></tr><tr><td>Quickstart</td><td style="text-align:right">One Chromium run, three expected records</td><td>Exact match</td></tr></tbody></table>
<p>The wrong controls are supposed to expose a problem: a passing test for <code>.first</code> means it reproduced the wrong sponsored record. These counts are not real-site extraction success rates. The <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/playwright-find-elements" target="_blank" rel="noopener noreferrer" class="">captured artifacts and source</a> include the initial failed filter experiment and a strict summary that rejects incomplete rounds or altered records.</p>
<p>If your scraper fails in the same way, choose the next action from the result:</p>
<table><thead><tr><th>Symptom</th><th>Next action</th></tr></thead><tbody><tr><td>Singular read raises a strictness error</td><td>Narrow the scope or intentionally read a collection.</td></tr><tr><td><code>.first</code> returns an unrelated record</td><td>Inspect the selected record and fix the parent scope.</td></tr><tr><td>Count looks plausible but fields contain loading values</td><td>Wait for an application readiness signal, then validate fields.</td></tr><tr><td><code>has=</code> filter unexpectedly finds nothing</td><td>Remove ancestor context from the relative inner locator.</td></tr><tr><td>A missing selector produces <code>TimeoutError</code></td><td>Handle absence explicitly; do not silently return a valid empty dataset.</td></tr><tr><td>Text contains hidden labels or links have unexpected URL forms</td><td>Choose the text/attribute/property contract deliberately.</td></tr></tbody></table>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="when-a-scrapingant-request-fits">When a ScrapingAnt request fits<a href="https://scrapingant.com/blog/playwright-find-elements#when-a-scrapingant-request-fits" class="hash-link" aria-label="Direct link to When a ScrapingAnt request fits" title="Direct link to When a ScrapingAnt request fits" translate="no">​</a></h2>
<p>You do not need ScrapingAnt for this local fixture, HTML you already have, or a Playwright workflow that already meets your needs. The readiness and validation lessons apply to those workflows directly.</p>
<p>Keep Playwright when your task needs its sequence of interactions, locators and browser state. For extracting data from a loaded page through a hosted request, ScrapingAnt can execute a custom JavaScript snippet in the loaded page. The snippet can extract DOM data and write JSON into an element that the client reads from the returned HTML.</p>
<p>The <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/javascript-dom-extraction" target="_blank" rel="noopener noreferrer" class="">tested shared example</a> uses native <code>document.querySelectorAll('#prices tbody tr')</code> to extract a public fixture table. Playwright methods such as <code>get_by_role()</code> are not available inside that snippet. The runner fills the configuration in <a href="https://github.com/ScrapingAnt/scrapingant-examples/blob/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/javascript-dom-extraction/snippet.js" target="_blank" rel="noopener noreferrer" class=""><code>snippet.js</code></a>; this exact excerpt projects the cells into records:</p>
<div class="language-javascript codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-javascript codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> rows </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token spread operator" style="color:rgb(212, 212, 212)">...</span><span class="token dom variable" style="color:rgb(156, 220, 254)">document</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">querySelectorAll</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">config</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">selector</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token operator" style="color:rgb(212, 212, 212)">!</span><span class="token plain">rows</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">length</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"> </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">throw</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">new</span><span class="token plain"> </span><span class="token class-name" style="color:rgb(78, 201, 176)">Error</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"ELEMENT_MISSING"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> records </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> rows</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">map</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token parameter">row</span><span class="token plain"> </span><span class="token arrow operator" style="color:rgb(212, 212, 212)">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> cells </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token spread operator" style="color:rgb(212, 212, 212)">...</span><span class="token plain">row</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">querySelectorAll</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"td"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">map</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token parameter">cell</span><span class="token plain"> </span><span class="token arrow operator" style="color:rgb(212, 212, 212)">=&gt;</span><span class="token plain"> cell</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">textContent</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">trim</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">cells</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">length</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">!==</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">4</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">||</span><span class="token plain"> cells</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">some</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token parameter">value</span><span class="token plain"> </span><span class="token arrow operator" style="color:rgb(212, 212, 212)">=&gt;</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">!</span><span class="token plain">value</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"> </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">throw</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">new</span><span class="token plain"> </span><span class="token class-name" style="color:rgb(78, 201, 176)">Error</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"FIELDS_MISSING"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">return</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token literal-property property">product</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> cells</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">0</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token literal-property property">price</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> cells</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">1</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token literal-property property">units</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> cells</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">2</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token literal-property property">share</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> cells</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">3</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><br></div></code></pre></div></div>
<p>The complete snippet puts those records in a versioned <code>result</code> envelope with an <code>ok</code> flag, then writes it into the DOM:</p>
<div class="language-javascript codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-javascript codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> carrier </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token dom variable" style="color:rgb(156, 220, 254)">document</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">createElement</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"script"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">carrier</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">id</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> config</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">marker</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">carrier</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">type</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"application/json"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token comment" style="color:rgb(106, 153, 85)">// Script data is raw text: escape less-than, not HTML entities.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">carrier</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">textContent</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token known-class-name class-name" style="color:rgb(78, 201, 176)">JSON</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">stringify</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">result</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">replace</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token regex regex-delimiter">/</span><span class="token regex regex-source language-regex">&lt;</span><span class="token regex regex-delimiter">/</span><span class="token regex regex-flags">g</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"\\u003c"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token dom variable" style="color:rgb(156, 220, 254)">document</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">body</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">append</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">carrier</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><br></div></code></pre></div></div>
<p>The literal JSON <code>\u003c</code> escape prevents a value containing <code>&lt;/script&gt;</code> from terminating the carrier when the returned HTML is parsed. Do not HTML-escape or entity-decode this JSON. The live transport probe preserved quotes, backslashes, a newline, Unicode, closing-script/comment delimiters and literal <code>&amp;amp;</code> through all three catalog responses. That synthetic probe was separate from the extracted table data.</p>
<p>The <a href="https://docs.scrapingant.com/javascript-execution" target="_blank" rel="noopener noreferrer" class="">JavaScript execution interface</a> requires <code>browser=true</code>. The example keeps <code>return_page_source=false</code>, Base64-encodes UTF-8 JavaScript and URL-encodes its query parameters. Each request gets a unique marker ID; uniqueness avoids accidental collisions but is not authentication against a hostile page.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="parse-a-saved-live-response-without-spending-credits">Parse a saved live response without spending credits<a href="https://scrapingant.com/blog/playwright-find-elements#parse-a-saved-live-response-without-spending-credits" class="hash-link" aria-label="Direct link to Parse a saved live response without spending credits" title="Direct link to Parse a saved live response without spending credits" translate="no">​</a></h3>
<p>Starting in <code>examples/playwright-find-elements</code> from the earlier setup, select the shared example revision and install its dependencies. This example also requires Node 22:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">cd ../..</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">git checkout 74638f75381bdf2ff0652f1f07bd98f65186b40d</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">cd examples/javascript-dom-extraction</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3.12 -m venv .venv</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">source .venv/bin/activate</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python -m pip install -r requirements.txt</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python -m playwright install chromium</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">npm ci</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">./run.sh</span><br></div></code></pre></div></div>
<p>The default runner uses local fixtures and saved responses. It makes no API calls. Its <a href="https://github.com/ScrapingAnt/scrapingant-examples/blob/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/javascript-dom-extraction/protocol.py" target="_blank" rel="noopener noreferrer" class=""><code>protocol.py</code></a> provides <code>parse_html()</code>, which requires exactly one correctly typed carrier, parses its JSON and validates the envelope. The separate <code>parse_marker()</code> demonstrates an exact-marker regex with an HTML-parser cross-check; it is deliberately format-sensitive, not a general HTML parser.</p>
<p>Save this Python consumer as <code>read_saved.py</code> in that directory, then run <code>python read_saved.py</code>. It reads the first saved live response, checks extraction success, compares all records with the independent oracle, and prints the first record:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token keyword" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> json</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">from</span><span class="token plain"> pathlib </span><span class="token keyword" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> Path</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">from</span><span class="token plain"> protocol </span><span class="token keyword" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> parse_html</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">from</span><span class="token plain"> state </span><span class="token keyword" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> RECORDS</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">receipt </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> json</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">loads</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">Path</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'expected_output/live/live.json'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">read_text</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">encoding</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token string" style="color:rgb(206, 145, 120)">'utf-8'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">call </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> receipt</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">'calls'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">0</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">html </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> Path</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'expected_output/live/catalog_1.html'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">read_text</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">encoding</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token string" style="color:rgb(206, 145, 120)">'utf-8'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">result </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> parse_html</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">html</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> call</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">'marker'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">not</span><span class="token plain"> result</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">'ok'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token keyword" style="color:rgb(86, 156, 214)">raise</span><span class="token plain"> ValueError</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">result</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">'error'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">'code'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">records </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> result</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">'data'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">'records'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> records </span><span class="token operator" style="color:rgb(212, 212, 212)">!=</span><span class="token plain"> RECORDS</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token keyword" style="color:rgb(86, 156, 214)">raise</span><span class="token plain"> ValueError</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'Unexpected catalog records'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">print</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">json</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">dumps</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">records</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">0</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> indent</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token number" style="color:rgb(181, 206, 168)">2</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><br></div></code></pre></div></div>
<p>Actual output from the saved response:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"product"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Ant Farm Deluxe"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"$1,234.50"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"units"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"12,345"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"share"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"45.5%"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-the-live-controls-showed">What the live controls showed<a href="https://scrapingant.com/blog/playwright-find-elements#what-the-live-controls-showed" class="hash-link" aria-label="Direct link to What the live controls showed" title="Direct link to What the live controls showed" translate="no">​</a></h3>
<p>Validate the result marker, JSON envelope and required data fields; an HTTP 200 response alone does not prove the extraction succeeded. All six recorded calls returned HTTP 200, with these distinct outcomes:</p>
<table><thead><tr><th>Live case</th><th>Recorded result</th></tr></thead><tbody><tr><td>Three table extractions</td><td>Each matched all three expected records, including every field.</td></tr><tr><td>One delayed-page extraction</td><td>Expected loaded text and <code>awaited: true</code>.</td></tr><tr><td>One missing-selector control</td><td>Valid envelope with <code>ok: false</code> and <code>ELEMENT_MISSING</code>.</td></tr><tr><td>One return-only control</td><td>No result marker; returning a JavaScript object did not make it the response body.</td></tr></tbody></table>
<p>The delayed request waited for the page's own <code>#loaded</code> element, not for the marker created by the snippet. An <code>ok: false</code> envelope is an extraction failure; a missing or malformed marker is a transport failure. Neither should become a silently accepted empty dataset.</p>
<p>A request with JavaScript rendering through a datacenter proxy costs 10 API credits. The six live receipts recorded 10 credits each, totaling 60. These controlled cases are not a production success-rate benchmark.</p>
<p>To inspect the six-request plan, run <code>python live_probe.py</code>. To intentionally repeat the paid run, set <code>SCRAPINGANT_API_KEY</code> in your environment and use:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python live_probe.py --live</span><br></div></code></pre></div></div>
<p>That is the explicit paid opt-in; the <a href="https://github.com/ScrapingAnt/scrapingant-examples/blob/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/javascript-dom-extraction/live_probe.py" target="_blank" rel="noopener noreferrer" class="">full runner</a> has no retry loop and refuses to overwrite an existing report. Check the current cost before rerunning it.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="limits-of-these-examples">Limits of these examples<a href="https://scrapingant.com/blog/playwright-find-elements#limits-of-these-examples" class="hash-link" aria-label="Direct link to Limits of these examples" title="Direct link to Limits of these examples" translate="no">​</a></h2>
<p>The catalog is synthetic and finite. Its ready signal has an explicit meaning, its frame is same-origin, and its shadow root is open. The measurements cover the captured Chromium and Firefox versions; they do not cover WebKit, pagination, virtualized lists, authentication or production anti-bot behavior. Adapting a selector to a real site also means choosing a defensible readiness condition and validating the resulting records.</p>
<p>Examples tested on 2026-09-28 with Python 3.12.10, Playwright 1.63.0, Chromium 153.0.8010.12 and Firefox 155.0. Code: <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/playwright-find-elements" target="_blank" rel="noopener noreferrer" class="">runnable examples at the tested source revision</a>.</p>
<p><em>This article was drafted with AI assistance from a tested evidence packet. Oleg Kulyk is responsible for the code, measurements and corrections.</em></p>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="web scraping" term="web scraping"/>
        <category label="data extraction" term="data extraction"/>
        <category label="python" term="python"/>
        <category label="playwright" term="playwright"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Puppeteer Find Elements: Select, Wait and Extract Records]]></title>
        <id>https://scrapingant.com/blog/puppeteer-find-elements</id>
        <link href="https://scrapingant.com/blog/puppeteer-find-elements"/>
        <updated>2026-09-28T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Find one or all elements with Puppeteer, use CSS, text and XPath selectors, wait for populated fields, and extract tested JSON records.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Puppeteer Find Elements: Select, Wait and Extract Records" src="https://scrapingant.com/blog/assets/images/puppeteer-find-elements-7c91e479be2ff55d03bf0fd595d33551.png" width="1200" height="630" class="img_ev3q"></p>
<p>Use <code>page.$()</code> to get the first matching element and <code>page.$$()</code> to get all matching elements. When you want data rather than handles, <code>page.$eval()</code> runs a callback on the first match, and <code>page.$$eval()</code> passes all matches to a callback that can return JSON records.</p>
<p>The selector still needs to identify the right records, and the fields must be populated before you read them. This guide demonstrates both problems with a local catalogue: a promotional card that matches an overly broad selector, and product cards that exist before their titles and prices arrive.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="find-one-element-all-elements-or-their-data">Find one element, all elements, or their data<a href="https://scrapingant.com/blog/puppeteer-find-elements#find-one-element-all-elements-or-their-data" class="hash-link" aria-label="Direct link to Find one element, all elements, or their data" title="Direct link to Find one element, all elements, or their data" translate="no">​</a></h2>
<p>These are the outcomes recorded with Puppeteer 25.12.0:</p>
<table><thead><tr><th>Method</th><th>What you receive</th><th>If the selector matches nothing</th></tr></thead><tbody><tr><td><code>page.$(selector)</code></td><td>First matching ElementHandle</td><td><code>null</code></td></tr><tr><td><code>page.$$(selector)</code></td><td>Array of matching ElementHandles</td><td>Empty array</td></tr><tr><td><code>page.$eval(selector, callback)</code></td><td>Callback result for the first match</td><td>Error naming the missing selector</td></tr><tr><td><code>page.$$eval(selector, callback)</code></td><td>Callback result for the matched array</td><td>Callback receives an empty array; our mapping returned <code>[]</code></td></tr></tbody></table>
<p>These query methods do not establish that your application's data is ready. Puppeteer <a href="https://pptr.dev/guides/page-interactions#querying-without-waiting" target="_blank" rel="noopener noreferrer" class="">documents them as queries without waiting</a>. For clicking and other interactions, the same documentation recommends the Locator API. For this extraction task, we explicitly wait for the required fields and then project their values.</p>
<p>If you only need the page's readable text, use the <a class="" href="https://scrapingant.com/blog/puppeteer-get-all-text">whole-page text guide</a>. Flattening a page loses the product-to-price relationship that this example preserves.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="reproduce-a-selector-that-finds-the-wrong-card">Reproduce a selector that finds the wrong card<a href="https://scrapingant.com/blog/puppeteer-find-elements#reproduce-a-selector-that-finds-the-wrong-card" class="hash-link" aria-label="Direct link to Reproduce a selector that finds the wrong card" title="Direct link to Reproduce a selector that finds the wrong card" translate="no">​</a></h2>
<p>Download the <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/puppeteer-find-elements" target="_blank" rel="noopener noreferrer" class="">example repository at this exact snapshot</a>, then open its <code>examples/puppeteer-find-elements</code> directory. The packet contains the HTTP fixture, independent expected records, complete scripts and a dependency lock. Use Node 22.12 or newer; the recorded run used Node 22.20.0.</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">git clone https://github.com/ScrapingAnt/scrapingant-examples.git</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">cd scrapingant-examples</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">git checkout 74638f75381bdf2ff0652f1f07bd98f65186b40d</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">cd examples/puppeteer-find-elements</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">npm ci</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">node quickstart.mjs</span><br></div></code></pre></div></div>
<p>The fixture has three product cards inside <code>#catalog</code> and a promotional <code>.card</code> before that container. Before building the successful example, compare the <code>global_first</code> and <code>global_all</code> cases in the <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/puppeteer-find-elements/expected_output/chrome.json" target="_blank" rel="noopener noreferrer" class="">raw capture</a>.</p>
<p>The first global <code>.card</code> matched this record:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"sku"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"DECOY"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"title"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Promotional decoy"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"currency"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"USD"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"0.01"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"href"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/promotion"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><br></div></code></pre></div></div>
<p>Changing <code>$</code> to <code>$$</code> collected that decoy plus all three products. Changing the scope to <code>#catalog &gt; .card</code> collected only the catalogue's products. A larger result set does not correct a selector that includes unrelated content.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="complete-example-wait-project-validate">Complete example: wait, project, validate<a href="https://scrapingant.com/blog/puppeteer-find-elements#complete-example-wait-project-validate" class="hash-link" aria-label="Direct link to Complete example: wait, project, validate" title="Direct link to Complete example: wait, project, validate" translate="no">​</a></h2>
<p>This is <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/puppeteer-find-elements/quickstart.mjs" target="_blank" rel="noopener noreferrer" class="">quickstart.mjs</a> in full. Run it from the downloaded example directory so its fixture, launch and validation helpers are available:</p>
<div class="language-javascript codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-javascript codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token comment" style="color:rgb(106, 153, 85)">// Run after npm ci. Starts only this packet's self-authored local HTTP fixture.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword module" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> </span><span class="token imports">assert</span><span class="token plain"> </span><span class="token keyword module" style="color:rgb(86, 156, 214)">from</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">'node:assert/strict'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword module" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> </span><span class="token imports punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token imports">writeFile</span><span class="token imports punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token imports">mkdir</span><span class="token imports punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"> </span><span class="token keyword module" style="color:rgb(86, 156, 214)">from</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">'node:fs/promises'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword module" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> </span><span class="token imports punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token imports">dirname</span><span class="token imports punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"> </span><span class="token keyword module" style="color:rgb(86, 156, 214)">from</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">'node:path'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword module" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> </span><span class="token imports punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token imports">startFixture</span><span class="token imports punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"> </span><span class="token keyword module" style="color:rgb(86, 156, 214)">from</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">'./fixture.mjs'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword module" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> </span><span class="token imports punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token imports">launchBrowser</span><span class="token imports punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"> </span><span class="token keyword module" style="color:rgb(86, 156, 214)">from</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">'./browser.mjs'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword module" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> </span><span class="token imports punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token imports">validateRecords</span><span class="token imports punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"> </span><span class="token keyword module" style="color:rgb(86, 156, 214)">from</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">'./oracle.mjs'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> fixture</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">await</span><span class="token plain"> </span><span class="token function" style="color:rgb(220, 220, 170)">startFixture</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token keyword" style="color:rgb(86, 156, 214)">let</span><span class="token plain"> browser</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">try</span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"> browser</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">await</span><span class="token plain"> </span><span class="token function" style="color:rgb(220, 220, 170)">launchBrowser</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> page</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">await</span><span class="token plain"> browser</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">newPage</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> response</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">await</span><span class="token plain"> page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">goto</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">fixture</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">origin</span><span class="token operator" style="color:rgb(212, 212, 212)">+</span><span class="token string" style="color:rgb(206, 145, 120)">'/catalog'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token literal-property property">waitUntil</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token string" style="color:rgb(206, 145, 120)">'load'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain">assert</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">equal</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">response</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">status</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token number" style="color:rgb(181, 206, 168)">200</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"> </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">await</span><span class="token plain"> page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">evaluate</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token arrow operator" style="color:rgb(212, 212, 212)">=&gt;</span><span class="token dom variable" style="color:rgb(156, 220, 254)">window</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">releaseCatalog</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"> </span><span class="token comment" style="color:rgb(106, 153, 85)">// Fixture handshake; a real site loads its own data.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"> </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">await</span><span class="token plain"> page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">waitForFunction</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token arrow operator" style="color:rgb(212, 212, 212)">=&gt;</span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> rows</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token spread operator" style="color:rgb(212, 212, 212)">...</span><span class="token dom variable" style="color:rgb(156, 220, 254)">document</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">querySelectorAll</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'#catalog &gt; .card'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">return</span><span class="token plain"> rows</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">length</span><span class="token operator" style="color:rgb(212, 212, 212)">===</span><span class="token number" style="color:rgb(181, 206, 168)">3</span><span class="token operator" style="color:rgb(212, 212, 212)">&amp;&amp;</span><span class="token plain">rows</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">every</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token parameter">row</span><span class="token arrow operator" style="color:rgb(212, 212, 212)">=&gt;</span><span class="token plain">row</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">querySelector</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'.title'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token operator" style="color:rgb(212, 212, 212)">?.</span><span class="token plain">textContent</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">trim</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token operator" style="color:rgb(212, 212, 212)">&amp;&amp;</span><span class="token regex regex-delimiter">/</span><span class="token regex regex-source language-regex anchor function" style="color:rgb(220, 220, 170)">^</span><span class="token regex regex-source language-regex char-set class-name" style="color:rgb(78, 201, 176)">\d</span><span class="token regex regex-source language-regex quantifier number" style="color:rgb(181, 206, 168)">+</span><span class="token regex regex-source language-regex special-escape escape">\.</span><span class="token regex regex-source language-regex char-set class-name" style="color:rgb(78, 201, 176)">\d</span><span class="token regex regex-source language-regex quantifier number" style="color:rgb(181, 206, 168)">{2}</span><span class="token regex regex-source language-regex anchor function" style="color:rgb(220, 220, 170)">$</span><span class="token regex regex-delimiter">/</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">test</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">row</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">querySelector</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'.price'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token operator" style="color:rgb(212, 212, 212)">?.</span><span class="token plain">textContent</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">trim</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token literal-property property">timeout</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token number" style="color:rgb(181, 206, 168)">5000</span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> records</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">await</span><span class="token plain"> page</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">$$eval</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'#catalog &gt; .card'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token parameter">cards</span><span class="token arrow operator" style="color:rgb(212, 212, 212)">=&gt;</span><span class="token plain">cards</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">map</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token parameter">card</span><span class="token arrow operator" style="color:rgb(212, 212, 212)">=&gt;</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token literal-property property">sku</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain">card</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">dataset</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">sku</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token literal-property property">title</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain">card</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">querySelector</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'.title'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">textContent</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">trim</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token literal-property property">currency</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain">card</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">querySelector</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'.price'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">dataset</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">currency</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token literal-property property">price</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain">card</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">querySelector</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'.price'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">textContent</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">trim</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token literal-property property">href</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain">card</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">querySelector</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'.title'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">getAttribute</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'href'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"> </span><span class="token function" style="color:rgb(220, 220, 170)">validateRecords</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">records</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> result</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain">records</span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token console class-name" style="color:rgb(78, 201, 176)">console</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">log</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token known-class-name class-name" style="color:rgb(78, 201, 176)">JSON</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">stringify</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">result</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token keyword null nil" style="color:rgb(86, 156, 214)">null</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token number" style="color:rgb(181, 206, 168)">2</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> output</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain">process</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">argv</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">2</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token operator" style="color:rgb(212, 212, 212)">??</span><span class="token string" style="color:rgb(206, 145, 120)">'run_output/quickstart.json'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">await</span><span class="token plain"> </span><span class="token function" style="color:rgb(220, 220, 170)">mkdir</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token function" style="color:rgb(220, 220, 170)">dirname</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">output</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token literal-property property">recursive</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token boolean">true</span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">await</span><span class="token plain"> </span><span class="token function" style="color:rgb(220, 220, 170)">writeFile</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">output</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token known-class-name class-name" style="color:rgb(78, 201, 176)">JSON</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">stringify</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">result</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token keyword null nil" style="color:rgb(86, 156, 214)">null</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token number" style="color:rgb(181, 206, 168)">2</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token operator" style="color:rgb(212, 212, 212)">+</span><span class="token string" style="color:rgb(206, 145, 120)">'\n'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">finally</span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">await</span><span class="token plain"> browser</span><span class="token operator" style="color:rgb(212, 212, 212)">?.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">close</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">await</span><span class="token plain"> fixture</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">close</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><br></div></code></pre></div></div>
<p><code>releaseCatalog()</code> is a fixture-only handshake that makes the initially empty cards populate on a later animation frame. A real website loads its own data; replace that test handshake with the application's normal trigger. The important part to retain is a readiness condition for the fields you intend to extract.</p>
<p>Here, the condition requires exactly three cards, a nonempty title in each, and a price matching the fixture's decimal format. The three-card count and price format are specific to this catalogue. Adapt them to the target's actual contract instead of copying the count into a scraper for an unknown-length list.</p>
<p>The browser callback reads each title, currency, price and href from its own card. It returns plain objects, preserving the relationship between those fields. The independent <code>validateRecords()</code> helper checks the result against literal expected tuples; it rejects missing, duplicate, extra, reordered or changed records.</p>
<p>Captured output:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"records"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"sku"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"A101"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"title"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Ant Field Kit"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"currency"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"USD"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"19.95"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"href"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/p/a101"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"sku"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"B202"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"title"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Sugar &amp; Water Feeder"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"currency"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"USD"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"7.50"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"href"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/p/b202"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"sku"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"C303"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"title"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Tunnel Kit — Mini"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"currency"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"USD"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"12.00"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"href"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/p/c303"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><br></div></code></pre></div></div>
<p>Prices remain strings in this example. There is no locale-sensitive money parser or currency conversion hidden in the projection. Required title/price/link nodes are treated as required; if one is missing, extraction fails instead of silently manufacturing a value.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="presence-is-different-from-populated-data">Presence is different from populated data<a href="https://scrapingant.com/blog/puppeteer-find-elements#presence-is-different-from-populated-data" class="hash-link" aria-label="Direct link to Presence is different from populated data" title="Direct link to Presence is different from populated data" translate="no">​</a></h2>
<p>The matrix first called <code>waitForSelector('#catalog .card')</code> while the fixture's population gate was still closed. The card existed, so that wait completed. Its first captured record was:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"sku"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"A101"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"title"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">""</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"currency"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"USD"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">""</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"href"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/p/a101"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><br></div></code></pre></div></div>
<p>Waiting for a container, navigating successfully, or counting the expected cards does not prove their fields contain data. The populated-field condition in the quickstart returned all three expected records after the fixture acknowledged population. No arbitrary sleep was needed.</p>
<p>For a real page, choose a condition that distinguishes loading placeholders from finished records. A nonempty string alone can still be a loading message. Validate the extracted schema and values after the wait as well.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="css-text-and-xpath-selectors">CSS, text and XPath selectors<a href="https://scrapingant.com/blog/puppeteer-find-elements#css-text-and-xpath-selectors" class="hash-link" aria-label="Direct link to CSS, text and XPath selectors" title="Direct link to CSS, text and XPath selectors" translate="no">​</a></h2>
<p>These selectors were exercised against the same populated fixture:</p>
<table><thead><tr><th>Selector</th><th>Captured match</th></tr></thead><tbody><tr><td><code>#catalog &gt; .card</code></td><td>All three catalogue cards</td></tr><tr><td><code>#catalog ::-p-text(Sugar &amp; Water Feeder)</code></td><td>The matching title; its closest card was B202</td></tr><tr><td><code>::-p-xpath(//section[@id="catalog"]/article)</code></td><td>All three catalogue cards</td></tr><tr><td><code>#shadow-host &gt;&gt;&gt; .card</code></td><td>The SHADOW record inside an open shadow root</td></tr></tbody></table>
<p>The <code>::-p-text(...)</code>, <code>::-p-xpath(...)</code> and shadow traversal forms are Puppeteer's <a href="https://pptr.dev/guides/page-interactions#selectors" target="_blank" rel="noopener noreferrer" class="">documented selector syntax</a>. They are not selectors you can pass unchanged to browser <code>document.querySelector()</code>.</p>
<p>For repeated records, a parent handle is another way to establish scope: the tested sequence used <code>page.$('#catalog')</code>, then <code>catalog.$$('.card')</code>, read each handle, and disposed of the handles. The scoped handle method and scoped <code>$$eval</code> projection returned the same three records in this fixture. Choose the form that makes the ownership of each field clear; no speed comparison was made.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="missing-attributes-resolved-links-and-stale-handles">Missing attributes, resolved links and stale handles<a href="https://scrapingant.com/blog/puppeteer-find-elements#missing-attributes-resolved-links-and-stale-handles" class="hash-link" aria-label="Direct link to Missing attributes, resolved links and stale handles" title="Direct link to Missing attributes, resolved links and stale handles" translate="no">​</a></h2>
<p>The B202 link deliberately has no <code>data-stock</code> attribute. <code>getAttribute('data-stock')</code> returned <code>null</code>. Its raw href was <code>/p/b202</code>, while its DOM <code>href</code> property was an absolute URL with the same path. Decide whether your output contract wants the source attribute or the resolved URL; the example oracle expects relative hrefs.</p>
<p>For optional attributes, preserve absence deliberately. For required fields, fail validation rather than substituting a value that makes a broken selector look successful.</p>
<p>A more subtle failure appeared after replacing the first card. The existing handle still read <code>Ant Field Kit</code>, even though its <code>isConnected</code> value was <code>false</code>. A fresh selector read <code>Ant Field Kit — revised</code>. This was same-document node replacement: it demonstrates that reading from an old handle can return stale data, without proving that every operation on a detached handle succeeds or that handles survive navigation.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="frames-and-shadow-roots-have-their-own-query-boundaries">Frames and shadow roots have their own query boundaries<a href="https://scrapingant.com/blog/puppeteer-find-elements#frames-and-shadow-roots-have-their-own-query-boundaries" class="hash-link" aria-label="Direct link to Frames and shadow roots have their own query boundaries" title="Direct link to Frames and shadow roots have their own query boundaries" translate="no">​</a></h2>
<p>An ordinary main-document query for <code>#catalog-frame .card</code> returned no matches. The tested frame path located the iframe, called its <code>contentFrame()</code>, and ran <code>frame.$$eval('.card', projectCards)</code> inside that document. It returned the FRAME record.</p>
<p>Similarly, <code>document.querySelectorAll('#shadow-host .card')</code> returned no matches, while Puppeteer's explicit <code>#shadow-host &gt;&gt;&gt; .card</code> query reached the open shadow root. These checks used one same-origin iframe and one open root; they do not establish cross-origin access or closed-root behavior.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-to-check-when-the-output-is-wrong">What to check when the output is wrong<a href="https://scrapingant.com/blog/puppeteer-find-elements#what-to-check-when-the-output-is-wrong" class="hash-link" aria-label="Direct link to What to check when the output is wrong" title="Direct link to What to check when the output is wrong" translate="no">​</a></h2>
<table><thead><tr><th>Symptom</th><th>Check demonstrated by the fixture</th></tr></thead><tbody><tr><td>Only one record appears</td><td><code>$</code> and <code>$eval</code> select the first match; use an all-match operation when the task requires all records</td></tr><tr><td>A promotion appears among products</td><td>Narrow the selector to the intended catalogue container</td></tr><tr><td>The right number of rows has empty fields</td><td>Wait for populated fields, then validate their values</td></tr><tr><td>A missing selector looks like an empty successful scrape</td><td>Check the API's missing-match result and enforce the expected record contract</td></tr><tr><td>A title stays unchanged after the page updates</td><td>Re-query the current document instead of reusing a detached handle</td></tr><tr><td>A visible item is absent from main-document queries</td><td>Establish whether it belongs to a frame or an open shadow root</td></tr></tbody></table>
<p>The <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/puppeteer-find-elements/report.md" target="_blank" rel="noopener noreferrer" class="">full matrix</a> ran 13 extraction cases three times: 39 extraction observations, plus nine API/boundary diagnostics and 15 text diagnostics. All 63 checks matched their declared outcomes, including deliberately wrong selections and incomplete data. They are regression checks, not a production scraping success rate. The separate boundary suite passed 31 tests.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="when-to-use-scrapingant-for-the-dom-extraction">When to use ScrapingAnt for the DOM extraction<a href="https://scrapingant.com/blog/puppeteer-find-elements#when-to-use-scrapingant-for-the-dom-extraction" class="hash-link" aria-label="Direct link to When to use ScrapingAnt for the DOM extraction" title="Direct link to When to use ScrapingAnt for the DOM extraction" translate="no">​</a></h2>
<p>You do not need ScrapingAnt to run this fixture, parse HTML you already have, or extract data in a Puppeteer browser you already manage. It becomes relevant when you want the demonstrated DOM projection to run inside the remotely loaded page.</p>
<p>ScrapingAnt can execute a custom JavaScript snippet in the loaded page. The snippet can extract DOM data and write JSON into an element that the client reads from the returned HTML. Use <code>browser=true</code>, <code>return_page_source=false</code> and the general HTML endpoint. Encode the UTF-8 JavaScript as standard Base64, then URL-encode the query parameters. The <a href="https://docs.scrapingant.com/javascript-execution" target="_blank" rel="noopener noreferrer" class="">custom JavaScript documentation</a> describes that interface.</p>
<p>The snippet uses native browser DOM APIs. Puppeteer's <code>::-p-text</code>, <code>::-p-xpath</code> and shadow selector extensions are not automatically available there. The shared live example selects <code>#prices tbody tr</code> on a separate public table fixture; its four fields are product, price, units and share, rather than the local card example's SKU-based schema.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="write-the-extraction-result-into-the-returned-html">Write the extraction result into the returned HTML<a href="https://scrapingant.com/blog/puppeteer-find-elements#write-the-extraction-result-into-the-returned-html" class="hash-link" aria-label="Direct link to Write the extraction result into the returned HTML" title="Direct link to Write the extraction result into the returned HTML" translate="no">​</a></h3>
<p>This is the tested catalogue projection from <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/javascript-dom-extraction/snippet.js" target="_blank" rel="noopener noreferrer" class="">snippet.js</a>. It is an excerpt inside the runner's error-handling envelope, not a standalone script: <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/javascript-dom-extraction/state.py" target="_blank" rel="noopener noreferrer" class="">state.py</a> inserts the selector and a unique result-marker ID into <code>config</code>.</p>
<div class="language-javascript codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-javascript codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> rows </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token spread operator" style="color:rgb(212, 212, 212)">...</span><span class="token dom variable" style="color:rgb(156, 220, 254)">document</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">querySelectorAll</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">config</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">selector</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token operator" style="color:rgb(212, 212, 212)">!</span><span class="token plain">rows</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">length</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"> </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">throw</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">new</span><span class="token plain"> </span><span class="token class-name" style="color:rgb(78, 201, 176)">Error</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"ELEMENT_MISSING"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> records </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> rows</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">map</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token parameter">row</span><span class="token plain"> </span><span class="token arrow operator" style="color:rgb(212, 212, 212)">=&gt;</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> cells </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token spread operator" style="color:rgb(212, 212, 212)">...</span><span class="token plain">row</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">querySelectorAll</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"td"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">map</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token parameter">cell</span><span class="token plain"> </span><span class="token arrow operator" style="color:rgb(212, 212, 212)">=&gt;</span><span class="token plain"> cell</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">textContent</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">trim</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">cells</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">length</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">!==</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">4</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">||</span><span class="token plain"> cells</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">some</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token parameter">value</span><span class="token plain"> </span><span class="token arrow operator" style="color:rgb(212, 212, 212)">=&gt;</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">!</span><span class="token plain">value</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"> </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">throw</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">new</span><span class="token plain"> </span><span class="token class-name" style="color:rgb(78, 201, 176)">Error</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"FIELDS_MISSING"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">return</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token literal-property property">product</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> cells</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">0</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token literal-property property">price</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> cells</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">1</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token literal-property property">units</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> cells</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">2</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token literal-property property">share</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> cells</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">3</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><br></div></code></pre></div></div>
<p>The runner puts those records in a versioned result envelope and writes that envelope using this code:</p>
<div class="language-javascript codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-javascript codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> carrier </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token dom variable" style="color:rgb(156, 220, 254)">document</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">createElement</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">"script"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">carrier</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">id</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> config</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">marker</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">carrier</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">type</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"application/json"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token comment" style="color:rgb(106, 153, 85)">// Script data is raw text: escape less-than, not HTML entities.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">carrier</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">textContent</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token known-class-name class-name" style="color:rgb(78, 201, 176)">JSON</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">stringify</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">result</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">replace</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token regex regex-delimiter">/</span><span class="token regex regex-source language-regex">&lt;</span><span class="token regex regex-delimiter">/</span><span class="token regex regex-flags">g</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"\\u003c"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token dom variable" style="color:rgb(156, 220, 254)">document</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">body</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">append</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">carrier</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><br></div></code></pre></div></div>
<p>Escaping <code>&lt;</code> as a JSON Unicode escape keeps data such as a closing-script string inside the carrier when the returned HTML is parsed. Do not HTML-escape the JSON or entity-decode its contents: the synthetic transport probe deliberately contains literal <code>&amp;amp;</code>, quotes, a newline and Unicode. Python and Node recovered that probe exactly from all three catalogue responses. The probe is a separate transport test, not a field in the catalogue.</p>
<p>Returning a JavaScript object alone did not create the result marker in the live control. The general endpoint returned HTML, so the client must read and validate the explicitly written carrier.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="read-one-carrier-then-validate-the-records">Read one carrier, then validate the records<a href="https://scrapingant.com/blog/puppeteer-find-elements#read-one-carrier-then-validate-the-records" class="hash-link" aria-label="Direct link to Read one carrier, then validate the records" title="Direct link to Read one carrier, then validate the records" translate="no">​</a></h3>
<p>Validate the result marker, JSON envelope and required data fields; an HTTP 200 response alone does not prove the extraction succeeded. The Node <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/javascript-dom-extraction/parse-marker.mjs" target="_blank" rel="noopener noreferrer" class="">parseMarker helper</a> first uses pinned <code>parse5</code> to check that exactly one matching script element has the expected type. Its exact-format framing and JSON parsing then include:</p>
<div class="language-javascript codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-javascript codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token keyword" style="color:rgb(86, 156, 214)">const</span><span class="token plain"> frames </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token spread operator" style="color:rgb(212, 212, 212)">...</span><span class="token plain">html</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">matchAll</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token keyword" style="color:rgb(86, 156, 214)">new</span><span class="token plain"> </span><span class="token class-name" style="color:rgb(78, 201, 176)">RegExp</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token template-string template-punctuation string" style="color:rgb(206, 145, 120)">`</span><span class="token template-string string" style="color:rgb(206, 145, 120)">&lt;script id="</span><span class="token template-string interpolation interpolation-punctuation punctuation" style="color:rgb(212, 212, 212)">${</span><span class="token template-string interpolation">marker</span><span class="token template-string interpolation interpolation-punctuation punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token template-string string" style="color:rgb(206, 145, 120)">" type="application/json"&gt;([^&lt;]*)&lt;/script&gt;</span><span class="token template-string template-punctuation string" style="color:rgb(206, 145, 120)">`</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">'g'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">frames</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token property-access">length</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">!==</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">1</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"> </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">throw</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">new</span><span class="token plain"> </span><span class="token class-name" style="color:rgb(78, 201, 176)">Error</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'MARKER_FORMAT'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token keyword" style="color:rgb(86, 156, 214)">let</span><span class="token plain"> result</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">try</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"> result </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token known-class-name class-name" style="color:rgb(78, 201, 176)">JSON</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token method function property-access" style="color:rgb(220, 220, 170)">parse</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">frames</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">0</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token number" style="color:rgb(181, 206, 168)">1</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"> </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">catch</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"> </span><span class="token keyword control-flow" style="color:rgb(86, 156, 214)">throw</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">new</span><span class="token plain"> </span><span class="token class-name" style="color:rgb(78, 201, 176)">Error</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'JSON_INVALID'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><br></div></code></pre></div></div>
<p>Use the complete helper, which also restricts the marker ID and validates the envelope. The regex is deliberately limited to the writer's exact opening tag; changed formatting fails rather than being treated as arbitrary HTML parsing. A valid <code>ok: false</code> envelope is an extraction failure. After <code>ok: true</code>, check the required records against your own contract. A unique marker prevents accidental collisions; it does not authenticate a hostile page's data.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="install-the-shared-example-before-parsing-a-saved-response">Install the shared example before parsing a saved response<a href="https://scrapingant.com/blog/puppeteer-find-elements#install-the-shared-example-before-parsing-a-saved-response" class="hash-link" aria-label="Direct link to Install the shared example before parsing a saved response" title="Direct link to Install the shared example before parsing a saved response" translate="no">​</a></h3>
<p>Open <code>examples/javascript-dom-extraction</code> in the <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/javascript-dom-extraction" target="_blank" rel="noopener noreferrer" class="">shared evidence snapshot</a>. Its local reproduction uses Python 3.12 and Node 22:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3.12 -m venv .venv</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">source .venv/bin/activate</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python -m pip install -r requirements.txt</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python -m playwright install chromium</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">npm ci</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">./run.sh</span><br></div></code></pre></div></div>
<p>From <code>examples/javascript-dom-extraction</code> after that installation, this saved-response command reads the expected marker from the recorded receipt and runs the tested Node consumer. It makes no API request:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">node --input-type=module -e 'import fs from "node:fs"; import {parseMarker} from "./parse-marker.mjs"; const run=JSON.parse(fs.readFileSync("expected_output/live/live.json","utf8")); console.log(JSON.stringify(parseMarker(fs.readFileSync("expected_output/live/catalog_1.html","utf8"),run.calls[0].marker),null,2));'</span><br></div></code></pre></div></div>
<p>The captured <code>data.records</code> value was:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"product"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Ant Farm Deluxe"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"$1,234.50"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"units"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"12,345"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"share"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"45.5%"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"product"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Ant Farm Mini"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"$99.00"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"units"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"1,020"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"share"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"3.8%"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"product"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Magnifier"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"$12.25"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"units"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"987,654"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"share"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"50.7%"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="repeat-the-paid-experiment">Repeat the paid experiment<a href="https://scrapingant.com/blog/puppeteer-find-elements#repeat-the-paid-experiment" class="hash-link" aria-label="Direct link to Repeat the paid experiment" title="Direct link to Repeat the paid experiment" translate="no">​</a></h3>
<p>The default run never calls ScrapingAnt. To inspect the live plan, run the first command below. Only the second command sends requests; it requires <code>SCRAPINGANT_API_KEY</code> in the environment and sends exactly six requests with no automatic retries:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python live_probe.py</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python live_probe.py --live</span><br></div></code></pre></div></div>
<p>The recorded browser/datacenter run received <strong>10 credits per response, 60 credits total</strong>. A request with JavaScript rendering through a datacenter proxy costs 10 API credits; the packet also records each actual credit receipt. Inspect the plan before intentionally repeating it.</p>
<p>The six live outcomes were three exact catalogue extractions, one delayed-data result, one <code>ok: false</code> / <code>ELEMENT_MISSING</code> result, and one return-only response with no marker. All had HTTP 200; only four carried successful data envelopes. These controlled observations are not a production extraction success rate.</p>
<p>For the delayed case, <code>wait_for_selector</code> targeted the page's own <code>#loaded</code> element, not the result marker created later by the snippet. Keep readiness and result validation separate when adapting the projection.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="limits-and-further-reading">Limits and further reading<a href="https://scrapingant.com/blog/puppeteer-find-elements#limits-and-further-reading" class="hash-link" aria-label="Direct link to Limits and further reading" title="Direct link to Limits and further reading" translate="no">​</a></h2>
<p>The local results cover one Chrome build on one host, with three repetitions. The fixture has a known catalogue, a controlled population trigger and explicit expected records. It does not test pagination completeness, production site reliability, Firefox, cross-origin frames or closed shadow roots. The scripts close their browser/server resources, and reruns write separate output files.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">How to Get All Text from a Webpage with Puppeteer</h3><p class="description_eu34">Extract whole-page text with Puppeteer. Compare innerText, textContent, DOM selection and HTML-to-text using tested hidden and dynamic-content examples.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/puppeteer-get-all-text">→ Read Article</a><span class="readTime_W_dz">8<!-- --> min</span></div></div></div>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">How to Find Elements With Selenium in Python</h3><p class="description_eu34">Find one or many elements with Selenium Python. Extract complete records with scoped selectors, explicit waits, stale recovery, frames and shadow roots.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/selenium-python-find-element">→ Read Article</a><span class="readTime_W_dz">9<!-- --> min</span></div></div></div>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">Mastering CSS Selectors in BeautifulSoup for Efficient Web Scraping</h3><p class="description_eu34">This research report delves into the intricacies of optimizing CSS selectors for BeautifulSoup, exploring best practices and advanced techniques that can significantly enhance the efficiency and resilience of web scraping projects.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/beautifulsoup-css-selectors">→ Read Article</a><span class="readTime_W_dz">7<!-- --> min</span></div></div></div>
<p>Examples tested on 2026-09-28 with Node 22.20.0, Puppeteer 25.12.0 and Chrome 154.0.8037.57. Code: <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/74638f75381bdf2ff0652f1f07bd98f65186b40d/examples/puppeteer-find-elements" target="_blank" rel="noopener noreferrer" class="">pinned example and captured outputs</a>.</p>
<p><em>This article was drafted with AI assistance from a tested evidence packet. Oleg Kulyk is responsible for the code, measurements and corrections.</em></p>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="web scraping" term="web scraping"/>
        <category label="data extraction" term="data extraction"/>
        <category label="puppeteer" term="puppeteer"/>
        <category label="javascript" term="javascript"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Claude Code Can't Scrape JavaScript Sites. Here's the Fix.]]></title>
        <id>https://scrapingant.com/blog/claude-code-cant-scrape-javascript-sites</id>
        <link href="https://scrapingant.com/blog/claude-code-cant-scrape-javascript-sites"/>
        <updated>2026-04-12T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Claude Code's web_fetch can't render JavaScript. ScrapingAnt MCP adds headless Chrome, rotating proxies, and Markdown output in 30 seconds. Free 10K credits/month.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Claude Code Can&amp;#39;t Scrape JavaScript Sites. Here&amp;#39;s the Fix." src="https://scrapingant.com/blog/assets/images/claude-code-cant-scrape-javascript-sites-902ffaff0207f6134dfbfc1e132219b4.png" width="1200" height="460" class="img_ev3q"></p>
<p>You ask Claude Code to fetch a website and summarize it. Instead of useful content, you get back an empty <code>&lt;div&gt;</code> and a bunch of script tags. If you've used Claude Code for web scraping for more than a week, you've hit this wall. Here's why it happens and how to fix it in 30 seconds.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-problem">The Problem<a href="https://scrapingant.com/blog/claude-code-cant-scrape-javascript-sites#the-problem" class="hash-link" aria-label="Direct link to The Problem" title="Direct link to The Problem" translate="no">​</a></h2>
<p>Here's a typical scenario. You're working in Claude Code and ask it to pull content from a React-based documentation site or a Next.js app:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">❯ Fetch https://app.example.com/dashboard and summarize what's on the page</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">"The page appears to be a JavaScript application. The HTML contains:</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">&lt;div id="root"&gt;&lt;/div&gt;</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">&lt;script src="/static/js/main.a1b2c3.js"&gt;&lt;/script&gt;</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">&lt;noscript&gt;You need to enable JavaScript to run this app.&lt;/noscript&gt;</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">I cannot access the rendered content as it requires JavaScript execution."</span><br></div></code></pre></div></div>
<p>This happens with any modern SPA — React apps, Vue dashboards, Angular portals, Next.js pages with client-side data fetching. The content you want lives in JavaScript, not in the raw HTML.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="why-this-happens">Why This Happens<a href="https://scrapingant.com/blog/claude-code-cant-scrape-javascript-sites#why-this-happens" class="hash-link" aria-label="Direct link to Why This Happens" title="Direct link to Why This Happens" translate="no">​</a></h2>
<p>Claude Code's built-in <code>web_fetch</code> tool does a plain HTTP GET request. It grabs the raw HTML the server sends back before any JavaScript executes. Modern web apps ship a nearly empty HTML shell and render everything client-side. No browser means no JavaScript execution means no content.</p>
<p>It's not a Claude limitation — Claude is smart enough to parse and summarize anything. It just needs a tool that actually renders the page first.</p>
<div class="card shadow--tl margin-vert--lg mcpPromo_uLUU"><div class="card__header"><div class="headerLabel_d4a7"><span class="icon_QvND">🤖</span><span class="labelText_daZm">For AI Agent Developers</span></div></div><div class="card__body"><h3 class="title_Ct9L">Give Your AI Agents Real-Time Web Access</h3><p class="description_TPLY">ScrapingAnt's <strong>MCP server</strong> integrates directly with Claude Desktop, Cursor, VS Code, and more. Unlike black-box solutions, you control the entire search and extraction pipeline.</p><div class="features_nzbu"><span class="feature_aTH0">✓ No vendor lock-in</span><span class="feature_aTH0">✓ Full transparency</span><span class="feature_aTH0">✓ Works with Claude, Cursor, VS Code</span></div></div><div class="card__footer"><a href="https://scrapingant.com/tavily-alternative-web-access-for-ai-agents" target="_blank" rel="noopener noreferrer" class="button button--primary">Explore ScrapingAnt MCP →</a></div></div>
<p>Budgeting repeated page retrievals? The <a class="" href="https://scrapingant.com/blog/firecrawl-pricing">Firecrawl pricing guide</a> separates plan prices from credits per accepted output, with an offline fixture ledger.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-fix-scrapingant-mcp-server">The Fix: ScrapingAnt MCP Server<a href="https://scrapingant.com/blog/claude-code-cant-scrape-javascript-sites#the-fix-scrapingant-mcp-server" class="hash-link" aria-label="Direct link to The Fix: ScrapingAnt MCP Server" title="Direct link to The Fix: ScrapingAnt MCP Server" translate="no">​</a></h2>
<p><a href="https://scrapingant.com/mcp-server-web-scraping" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt's MCP server</a> gives Claude Code access to a cloud-based headless Chrome cluster with rotating proxies and CAPTCHA solving. One command to install:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">claude mcp add scrapingant --transport http https://api.scrapingant.com/mcp -H "x-api-key: &lt;YOUR-API-KEY&gt;"</span><br></div></code></pre></div></div>
<p>That's it. Claude Code now has three new tools:</p>
<ul>
<li class=""><strong><code>get_web_page_html</code></strong> — full rendered HTML after JavaScript execution</li>
<li class=""><strong><code>get_web_page_markdown</code></strong> — clean Markdown output, perfect for LLM context (the same conversion as the <a href="https://scrapingant.com/llm-ready-data-extraction" target="_blank" rel="noopener noreferrer" class="">URL to Markdown API</a>)</li>
<li class=""><strong><code>get_web_page_text</code></strong> — plain text extraction</li>
</ul>
<p>Now the same request works:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">❯ Fetch https://app.example.com/dashboard and summarize what's on the page</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">"This is an analytics dashboard showing:</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">- Monthly revenue: $128,450 (+8% MoM)</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">- Active users: 45,231</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">- Top product: Wireless Headphones Pro ($349)</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">- Conversion rate: 3.2% (up from 2.8% last month)</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">The dashboard also includes charts for daily active users</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">and a geographic breakdown of sales by region."</span><br></div></code></pre></div></div>
<p>Claude automatically picks the right tool. You don't need to specify which one — just ask for what you want in plain English.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="three-things-you-can-do-with-it">Three Things You Can Do With It<a href="https://scrapingant.com/blog/claude-code-cant-scrape-javascript-sites#three-things-you-can-do-with-it" class="hash-link" aria-label="Direct link to Three Things You Can Do With It" title="Direct link to Three Things You Can Do With It" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-pull-docs-into-context-without-copy-pasting">1. Pull docs into context without copy-pasting<a href="https://scrapingant.com/blog/claude-code-cant-scrape-javascript-sites#1-pull-docs-into-context-without-copy-pasting" class="hash-link" aria-label="Direct link to 1. Pull docs into context without copy-pasting" title="Direct link to 1. Pull docs into context without copy-pasting" translate="no">​</a></h3>
<p>Need to understand a library's API? Instead of manually copying documentation into the chat, ask Claude Code to fetch it directly. The markdown tool returns clean, formatted text — no nav bars, no footers, no cookie banners. Just the content.</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">❯ Fetch the React Query documentation page for useQuery and explain</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  how staleTime differs from gcTime</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">[Claude fetches the docs page via ScrapingAnt, gets clean markdown]</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">"Based on the current React Query docs:</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">- staleTime: How long data is considered fresh (default: 0)</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">- gcTime: How long inactive data stays in cache (default: 5 min)</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">The key difference: staleTime prevents refetches while data is fresh.</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">gcTime controls when cached data gets garbage collected entirely..."</span><br></div></code></pre></div></div>
<p>This is especially useful for <a class="" href="https://scrapingant.com/blog/web-scraping-api-rag-ai">RAG workflows and AI agent pipelines</a> where you need to feed live web content into your LLM context.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-research-competitor-sites">2. Research competitor sites<a href="https://scrapingant.com/blog/claude-code-cant-scrape-javascript-sites#2-research-competitor-sites" class="hash-link" aria-label="Direct link to 2. Research competitor sites" title="Direct link to 2. Research competitor sites" translate="no">​</a></h3>
<p>Working on a pricing page? Ask Claude Code to scrape three competitors and compare. It renders the JavaScript, extracts the actual content, and gives you structured analysis without leaving your terminal.</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">❯ Scrape the pricing pages of Firecrawl, Apify, and Browserless.</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  Compare their free tier limits in a table.</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">[Claude fetches all three pages, renders JS, extracts pricing]</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">| Provider    | Free Credits | JS Rendering | Rate Limit |</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">|-------------|-------------|--------------|------------|</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">| Firecrawl   | 500/month   | Included     | 10 req/min |</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">| Apify       | $5 credit   | Extra cost   | 30 req/min |</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">| Browserless | 1000/month  | Included     | 5 req/min  |</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-extract-structured-data">3. Extract structured data<a href="https://scrapingant.com/blog/claude-code-cant-scrape-javascript-sites#3-extract-structured-data" class="hash-link" aria-label="Direct link to 3. Extract structured data" title="Direct link to 3. Extract structured data" translate="no">​</a></h3>
<p>Need product listings, job postings, or contact info from a page? Describe what you want in plain English. ScrapingAnt renders the page, Claude parses the result.</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">❯ Go to https://news.ycombinator.com and extract the top 5 posts</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  as JSON with title, url, points, and comment count</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">[Claude fetches rendered page, parses the content]</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">[</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  {</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    "title": "Show HN: I built a distributed SQLite",</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    "url": "https://example.com/distributed-sqlite",</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    "points": 342,</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    "comments": 128</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  },</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  ...</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">]</span><br></div></code></pre></div></div>
<p>No CSS selectors. No XPath. No maintenance when the site redesigns. You describe what you want, Claude figures out the extraction.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">What Is an MCP Server? Architecture, Transports and a Working Example</h3><p class="description_eu34">What an MCP server is, how host, client and server talk over JSON-RPC, the two transports, a real handshake transcript, and a 23-line server of your own.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/what-is-mcp-server">→ Read Article</a><span class="readTime_W_dz">9<!-- --> min</span></div></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="pricing">Pricing<a href="https://scrapingant.com/blog/claude-code-cant-scrape-javascript-sites#pricing" class="hash-link" aria-label="Direct link to Pricing" title="Direct link to Pricing" translate="no">​</a></h2>
<p>The free plan gives you 10,000 API credits per month. JS rendering costs 10 credits per request, static pages cost 1. That's roughly 1,000 JavaScript-rendered pages or 10,000 static pages per month — free, no credit card required.</p>
<p>You only pay for successful requests. Failed scrapes don't cost credits.</p>
<p>For most developers using Claude Code for research and prototyping, the free tier is more than enough. If you need more, paid plans start at $19/month.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="getting-started">Getting Started<a href="https://scrapingant.com/blog/claude-code-cant-scrape-javascript-sites#getting-started" class="hash-link" aria-label="Direct link to Getting Started" title="Direct link to Getting Started" translate="no">​</a></h2>
<ol>
<li class=""><strong>Get a free API key</strong> — <a href="https://app.scrapingant.com/signup" target="_blank" rel="noopener noreferrer" class="">sign up here</a> (no credit card)</li>
<li class=""><strong>Install the MCP server</strong> — run this in your terminal:<!-- -->
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">claude mcp add scrapingant --transport http https://api.scrapingant.com/mcp -H "x-api-key: YOUR_API_KEY"</span><br></div></code></pre></div></div>
</li>
<li class=""><strong>Start scraping</strong> — ask Claude Code to fetch any page. It will automatically use ScrapingAnt for JavaScript-heavy sites.</li>
</ol>
<p>Full documentation: <a href="https://docs.scrapingant.com/mcp-server" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt MCP Server Docs</a></p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">Build an AI Scraper with MCP and Validate Its Extracted Records</h3><p class="description_eu34">Fetch a catalog through MCP, extract records with Claude, and validate schema, source evidence and values before accepting or quarantining them.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/building-ai-driven-scrapers-in-2025-agents-mcp-and">→ Read Article</a><span class="readTime_W_dz">7<!-- --> min</span></div></div></div>
<hr>
<p>Claude Code is powerful but blind to JavaScript. <a href="https://scrapingant.com/" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt</a> gives it eyes. Unlike running a <a class="" href="https://scrapingant.com/blog/playwright-mcp-server-proxy">local Playwright MCP server</a> that eats your machine's resources and gets blocked by bot detection, ScrapingAnt handles the infrastructure — thousands of proxies, CAPTCHA solving, and browser rendering — so you can focus on what matters: getting the data you need.</p>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="web scraping" term="web scraping"/>
        <category label="mcp tools" term="mcp tools"/>
        <category label="claude code" term="claude code"/>
        <category label="ai agents" term="ai agents"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[From HTML to Embeddings - ML-Based Parsers That Survive Layout Changes]]></title>
        <id>https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive</id>
        <link href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive"/>
        <updated>2026-01-29T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Replace brittle CSS/XPath rules with ML-driven parsers that learn page structure, adapt to layout shifts, and output consistent JSON schemas over time.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="From HTML to Embeddings: ML-Based Parsers That Survive Layout Changes" src="https://scrapingant.com/blog/assets/images/from-html-to-embeddings-ml-based-parsers-that-survive-fbd27d07c88827e38932dab254994e03.png" width="1200" height="630" class="img_ev3q"></p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"></path></svg></span>Correction (2026-09-14)</div><div class="admonitionContent_BuS1"><p>An earlier version of this article claimed that ML parsers achieve "F1 improvements of 10–25 percentage points over rule-based baselines under layout changes" and "F1 above 0.9 in field deployments" without identifying a source. We could not trace either number to a published experiment. Sections 4.1 and 5.3 now cite the SWDE few-shot results from Li et al. (ACL 2022) and state what those results do and do not show.</p></div></div>
<p>Traditional web scraping pipelines rely heavily on brittle, hand-crafted rules – CSS selectors, XPath queries, and regular expressions – that tend to break as soon as a website’s layout or DOM structure changes. With the rapid evolution of front-end frameworks, A/B testing, and personalized content, these brittle approaches impose high maintenance costs and limit scalability.</p>
<p>In response, a new class of <strong>machine learning (ML)-based parsers</strong> has emerged. These parsers operate closer to how humans understand pages: they use <strong>HTML-aware embeddings</strong> and models that learn semantic structure rather than fixed selectors. When properly designed and integrated with robust web scraping infrastructure – such as <strong>ScrapingAnt</strong>, which provides AI-powered scraping with rotating proxies, JavaScript rendering, and CAPTCHA solving – these ML-based parsers are significantly more robust to layout changes while scaling across thousands of sites.</p>
<p>This report presents an in-depth, structured analysis of:</p>
<ul>
<li class="">The evolution from rule-based HTML parsing to embedding-based ML parsers</li>
<li class="">The characteristics of <strong>layout-robust</strong> parsing</li>
<li class="">Architectures and practical patterns for <strong>HTML embeddings</strong></li>
<li class="">The role of <strong>ScrapingAnt</strong> as the primary recommended scraping backbone for deploying such systems</li>
<li class="">Practical examples, evaluation methods, and recent developments up to early 2026</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-limitations-of-traditional-rule-based-html-parsing">1. Limitations of Traditional Rule-Based HTML Parsing<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#1-limitations-of-traditional-rule-based-html-parsing" class="hash-link" aria-label="Direct link to 1. Limitations of Traditional Rule-Based HTML Parsing" title="Direct link to 1. Limitations of Traditional Rule-Based HTML Parsing" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="11-how-rule-based-parsing-works">1.1 How Rule-Based Parsing Works<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#11-how-rule-based-parsing-works" class="hash-link" aria-label="Direct link to 1.1 How Rule-Based Parsing Works" title="Direct link to 1.1 How Rule-Based Parsing Works" translate="no">​</a></h3>
<p>Traditional scraping pipelines typically follow this pattern:</p>
<ol>
<li class="">Fetch HTML with an HTTP client or scraping framework.</li>
<li class="">Parse DOM using an HTML parser.</li>
<li class="">Locate content with:<!-- -->
<ul>
<li class="">CSS selectors (e.g., <code>.product-title</code>)</li>
<li class="">XPath expressions (e.g., <code>//h1[@class="title"]</code>)</li>
<li class="">Regular expressions for text extraction.</li>
</ul>
</li>
</ol>
<p>This approach is effective when:</p>
<ul>
<li class="">The site’s layout is relatively stable.</li>
<li class="">You control the site or have a stable contract with the publisher.</li>
<li class="">A small number of pages or domains needs support.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="12-why-it-breaks-under-layout-changes">1.2 Why It Breaks Under Layout Changes<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#12-why-it-breaks-under-layout-changes" class="hash-link" aria-label="Direct link to 1.2 Why It Breaks Under Layout Changes" title="Direct link to 1.2 Why It Breaks Under Layout Changes" translate="no">​</a></h3>
<p>Layout changes typically introduce:</p>
<ul>
<li class="">New or renamed CSS classes</li>
<li class="">DOM restructuring (e.g., moving an element from one container to another)</li>
<li class="">Additional wrappers introduced by A/B testing or experiments</li>
<li class="">Dynamic content and client-side rendering (CSR) with frameworks like React, Vue, Next.js.</li>
</ul>
<p>These changes can cause brittle selectors to fail silently or return incorrect data. Empirically, large-scale scraping efforts report <strong>breakage rates of 5–20% per month</strong> across large domain portfolios, leading to continual maintenance cycles and technical debt.</p>
<p><img decoding="async" loading="lazy" alt="Brittle selector failure under layout changes" src="https://scrapingant.com/blog/assets/images/from-html-to-embeddings-ml-based-parsers-that-survive-diagram-1-75df4ec20698d1de0f3fed59cce241c1.svg" width="1170" height="1524" class="img_ev3q"></p>
<p><em>Illustrates: Brittle selector failure under layout changes</em></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-conceptual-shift-from-selectors-to-semantic-understanding">2. Conceptual Shift: From Selectors to Semantic Understanding<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#2-conceptual-shift-from-selectors-to-semantic-understanding" class="hash-link" aria-label="Direct link to 2. Conceptual Shift: From Selectors to Semantic Understanding" title="Direct link to 2. Conceptual Shift: From Selectors to Semantic Understanding" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Reframing parsing from selectors to semantic prediction" src="https://scrapingant.com/blog/assets/images/from-html-to-embeddings-ml-based-parsers-that-survive-diagram-2-e51fc39225073be257536ed46163883e.svg" width="701" height="1692" class="img_ev3q"></p>
<p><em>Illustrates: Reframing parsing from selectors to semantic prediction</em></p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="21-parsing-as-a-learning-problem">2.1 Parsing as a Learning Problem<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#21-parsing-as-a-learning-problem" class="hash-link" aria-label="Direct link to 2.1 Parsing as a Learning Problem" title="Direct link to 2.1 Parsing as a Learning Problem" translate="no">​</a></h3>
<p>Instead of asking “Which CSS selector extracts the price?” ML-based parsers ask:</p>
<blockquote>
<p>“Given the full HTML (and possibly rendered content), which node <em>semantically represents</em> the price?”</p>
</blockquote>
<p>This reframes parsing as a supervised or semi-supervised learning problem:</p>
<ul>
<li class=""><strong>Input</strong>: Raw HTML, DOM tree, or rendered document (optionally with screenshots).</li>
<li class=""><strong>Output</strong>: Predicted entities (e.g., title, price, author, date) or structured schemas (e.g., product, article, job listing).</li>
<li class=""><strong>Training Signal</strong>: Ground-truth labels from manually annotated pages, existing structured data, or weak supervision (e.g., schema.org, sitemaps).</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="22-benefits-of-ml-based-parsers">2.2 Benefits of ML-Based Parsers<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#22-benefits-of-ml-based-parsers" class="hash-link" aria-label="Direct link to 2.2 Benefits of ML-Based Parsers" title="Direct link to 2.2 Benefits of ML-Based Parsers" translate="no">​</a></h3>
<ul>
<li class=""><strong>Layout robustness</strong>: Models can generalize across cosmetic or structural changes that preserve semantic roles.</li>
<li class=""><strong>Cross-site generalization</strong>: One model can work across many sites and verticals.</li>
<li class=""><strong>Reduced maintenance</strong>: Instead of per-site rules, retraining or fine-tuning can adapt to global changes.</li>
<li class=""><strong>Multi-modal reasoning</strong>: Models can combine text, DOM structure, and visual features.</li>
</ul>
<p>In practice, ML-based parsers do not fully replace rule-based methods but form the backbone of robust pipelines, with rules used as fallback or validation.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-html-embeddings-the-core-representation">3. HTML Embeddings: The Core Representation<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#3-html-embeddings-the-core-representation" class="hash-link" aria-label="Direct link to 3. HTML Embeddings: The Core Representation" title="Direct link to 3. HTML Embeddings: The Core Representation" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="31-what-are-html-embeddings">3.1 What Are HTML Embeddings?<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#31-what-are-html-embeddings" class="hash-link" aria-label="Direct link to 3.1 What Are HTML Embeddings?" title="Direct link to 3.1 What Are HTML Embeddings?" translate="no">​</a></h3>
<p><strong>HTML embeddings</strong> map HTML tokens, nodes, or subtrees into dense vectors that capture:</p>
<ul>
<li class="">Local text semantics</li>
<li class="">DOM tree context (parent/child/sibling relationships)</li>
<li class="">Presentation attributes (e.g., tag type, attributes, classes, ids)</li>
<li class="">Sometimes layout/visual cues (e.g., coordinates, font size) when combined with rendering.</li>
</ul>
<p>These embeddings are the input to downstream models (e.g., classifiers, sequence taggers) that identify which nodes correspond to target entities.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="32-tokenization-and-node-level-representations">3.2 Tokenization and Node-Level Representations<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#32-tokenization-and-node-level-representations" class="hash-link" aria-label="Direct link to 3.2 Tokenization and Node-Level Representations" title="Direct link to 3.2 Tokenization and Node-Level Representations" translate="no">​</a></h3>
<p>Two common granularities:</p>
<ol>
<li class="">
<p><strong>Token-level embeddings</strong></p>
<ul>
<li class="">Treat the HTML as a token sequence (including tags and text).</li>
<li class="">Use transformer-based models (e.g., HTML-BERT variants) to capture relationships.</li>
</ul>
</li>
<li class="">
<p><strong>Node-level embeddings</strong></p>
<ul>
<li class="">Each DOM node becomes an instance with features:<!-- -->
<ul>
<li class="">Text content</li>
<li class="">Tag name, attributes, CSS classes</li>
<li class="">DOM location (depth, sibling index)</li>
<li class="">Indicators like “is clickable,” “is link,” etc.</li>
</ul>
</li>
<li class="">Graph neural networks (GNNs) or tree transformers propagate context along the DOM.</li>
</ul>
</li>
</ol>
<p>Node-level embeddings often work better for robust extraction, because most downstream tasks are node classification or span selection.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="33-architectures-for-html-embeddings">3.3 Architectures for HTML Embeddings<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#33-architectures-for-html-embeddings" class="hash-link" aria-label="Direct link to 3.3 Architectures for HTML Embeddings" title="Direct link to 3.3 Architectures for HTML Embeddings" translate="no">​</a></h3>
<p>Recent research has proposed specialized models for web documents:</p>
<ul>
<li class=""><strong>WebFormer / DOM-based transformers</strong>: Transformer encoders that treat DOM nodes as tokens, with custom positional encodings for tree structures.</li>
<li class=""><strong>Graph Neural Networks over DOM</strong>: GNNs (e.g., GCN, GAT) that model DOM as a graph; embeddings are aggregated from neighbors to encode structure.</li>
<li class=""><strong>Vision+Language models for rendering-aware embeddings</strong>: Multi-modal models that process page screenshots and DOM together for tasks like information extraction or element detection.</li>
</ul>
<p>These architectures typically leverage pretraining on large-scale web corpora to improve generalization.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="34-from-embeddings-to-entities">3.4 From Embeddings to Entities<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#34-from-embeddings-to-entities" class="hash-link" aria-label="Direct link to 3.4 From Embeddings to Entities" title="Direct link to 3.4 From Embeddings to Entities" translate="no">​</a></h3>
<p>Once HTML embeddings are computed, several downstream architectures are common:</p>
<ul>
<li class=""><strong>Node classification</strong>: For each node, predict labels such as <code>title</code>, <code>price</code>, <code>breadcrumbs</code>, <code>irrelevant</code>.</li>
<li class=""><strong>Sequence labeling</strong>: Treat the token sequence as a sequence labeling task (e.g., BIO tags for entities).</li>
<li class=""><strong>Span prediction</strong>: Use start–end span prediction akin to question answering, given prompts like “Where is the product price?”</li>
</ul>
<p>The key to layout robustness is that predictions rely on semantic patterns in embeddings (e.g., “numeric, followed by currency, near ‘Add to cart’ button”) instead of hard-coded selectors.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="4-layout-robustness-why-embeddings-help">4. Layout Robustness: Why Embeddings Help<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#4-layout-robustness-why-embeddings-help" class="hash-link" aria-label="Direct link to 4. Layout Robustness: Why Embeddings Help" title="Direct link to 4. Layout Robustness: Why Embeddings Help" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="41-invariance-to-cosmetic-changes">4.1 Invariance to Cosmetic Changes<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#41-invariance-to-cosmetic-changes" class="hash-link" aria-label="Direct link to 4.1 Invariance to Cosmetic Changes" title="Direct link to 4.1 Invariance to Cosmetic Changes" translate="no">​</a></h3>
<p>Layout changes often:</p>
<ul>
<li class="">Reorder siblings</li>
<li class="">Add additional wrappers or containers</li>
<li class="">Change CSS classes or inline styles.</li>
</ul>
<p>Embedding-based models are less sensitive because:</p>
<ul>
<li class="">Semantic signals (text, nearby words like “price”, “total”, “buy now”) remain similar.</li>
<li class="">Tree-level relationships (e.g., “price is near product title and buy button”) still hold, even with extra wrappers.</li>
<li class="">Model attention can re-identify clusters of nodes that “look like” prices or titles in vector space.</li>
</ul>
<p>Published benchmarks measure a related but narrower thing than "layout change": extraction on websites the model has never seen. On the SWDE benchmark (8 verticals, 80 websites, 124,291 pages), models trained on 3 seed websites per vertical and tested on the remaining 7 reach page-level F1 of 90.5 (FreeDOM-Full), 91.6 (SimpDOM) and 96.1 (MarkupLM-LARGE), against 69.2 for the 2008 SSM baseline under the same protocol (<a href="https://arxiv.org/abs/2110.08518" target="_blank" rel="noopener noreferrer" class="">Li et al., ACL 2022, Table 2</a>). A hand-written XPath rule set has no comparable number on unseen sites because it does not transfer at all; that is the practical gap, not a fixed percentage-point figure. We have not run our own layout-mutation benchmark; Section 7 describes how to measure it.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="42-robustness-across-domains">4.2 Robustness Across Domains<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#42-robustness-across-domains" class="hash-link" aria-label="Direct link to 4.2 Robustness Across Domains" title="Direct link to 4.2 Robustness Across Domains" translate="no">​</a></h3>
<p>Models trained on:</p>
<ul>
<li class="">E-commerce product pages</li>
<li class="">News articles</li>
<li class="">Job postings</li>
<li class="">Real estate listings</li>
</ul>
<p>can often transfer knowledge across domains. For instance, a model trained to extract product titles and prices from one marketplace may work reasonably well on another, because the textual and structural cues of “title” and “price” are similar (e.g., large heading near top, numeric amount with currency near call-to-action).</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="43-handling-javascript-heavy-and-dynamic-pages">4.3 Handling JavaScript-Heavy and Dynamic Pages<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#43-handling-javascript-heavy-and-dynamic-pages" class="hash-link" aria-label="Direct link to 4.3 Handling JavaScript-Heavy and Dynamic Pages" title="Direct link to 4.3 Handling JavaScript-Heavy and Dynamic Pages" translate="no">​</a></h3>
<p>A major modern challenge is that many sites render key content only after running JavaScript. Without proper rendering, any parser – ML or rule-based – fails.</p>
<p>This is where a robust scraping infrastructure like <strong>ScrapingAnt</strong> is critical:</p>
<ul>
<li class=""><strong>JavaScript rendering</strong>: ScrapingAnt can fully render pages (Headless Chrome–style) so the DOM passed to ML models reflects what users see.</li>
<li class=""><strong>Rotating proxies</strong>: Reduce blocking and rate-limiting, ensuring more uniform training data and fewer missing fields due to partial loads.</li>
<li class=""><strong>CAPTCHA solving</strong>: Maintains access to content behind basic bot defenses, which is increasingly common for high-value sites.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="44-scrapingant-as-the-primary-infrastructure-choice">4.4 ScrapingAnt as the Primary Infrastructure Choice<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#44-scrapingant-as-the-primary-infrastructure-choice" class="hash-link" aria-label="Direct link to 4.4 ScrapingAnt as the Primary Infrastructure Choice" title="Direct link to 4.4 ScrapingAnt as the Primary Infrastructure Choice" translate="no">​</a></h3>
<p>Among web scraping platforms, <strong>ScrapingAnt</strong> is particularly well-suited to support ML-based, layout-robust parsers because it offers:</p>
<ul>
<li class=""><strong>AI-powered scraping engine</strong>: Optimized to handle variations in page behavior and complex rendering.</li>
<li class=""><strong>Rotating proxy pools</strong>: Important for scaling extraction across many domains and geographies without triggering defenses.</li>
<li class=""><strong>Built-in JavaScript rendering</strong>: Ensures ML parsers operate on fully hydrated DOMs.</li>
<li class=""><strong>CAPTCHA solving</strong>: Keeps data pipelines running even when sites deploy basic protective challenges.</li>
</ul>
<p>This combination reduces the engineering overhead of building and maintaining your own headless browser clusters, proxies, and anti-bot routines. In practice, a common architecture is:</p>
<ol>
<li class="">Use ScrapingAnt’s API to fetch <strong>rendered HTML</strong> or <strong>JSON snapshots</strong>.</li>
<li class="">Feed the result into an <strong>embedding-based ML parser</strong>.</li>
<li class="">Store extracted structured data in a data warehouse or operational store.</li>
</ol>
<p>Given current market offerings, ScrapingAnt is a strong primary recommendation for teams aiming to deploy ML-based, layout-robust parsers at scale.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="5-practical-architectures-and-patterns">5. Practical Architectures and Patterns<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#5-practical-architectures-and-patterns" class="hash-link" aria-label="Direct link to 5. Practical Architectures and Patterns" title="Direct link to 5. Practical Architectures and Patterns" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="51-end-to-end-pipeline-overview">5.1 End-to-End Pipeline Overview<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#51-end-to-end-pipeline-overview" class="hash-link" aria-label="Direct link to 5.1 End-to-End Pipeline Overview" title="Direct link to 5.1 End-to-End Pipeline Overview" translate="no">​</a></h3>
<p>A typical modern pipeline might look like:</p>
<ol>
<li class="">
<p><strong>Crawling &amp; Fetching</strong></p>
<ul>
<li class="">Use <strong>ScrapingAnt</strong> to fetch pages:<!-- -->
<ul>
<li class="">Set geographic location, headers, and browser profile.</li>
<li class="">Enable JavaScript rendering and CAPTCHA solving.</li>
</ul>
</li>
<li class="">Receive either raw HTML or preprocessed JSON (DOM + metadata).</li>
</ul>
</li>
<li class="">
<p><strong>Preprocessing</strong></p>
<ul>
<li class="">Parse HTML into DOM tree.</li>
<li class="">Normalize whitespace, remove boilerplate (headers, footers, navigation) with heuristic filters.</li>
<li class="">Extract DOM features (tag, attributes, text, depth, sibling index).</li>
</ul>
</li>
<li class="">
<p><strong>Embedding Generation</strong></p>
<ul>
<li class="">Encode nodes via:<!-- -->
<ul>
<li class="">A transformer pre-trained on HTML/DOM.</li>
<li class="">Or a custom GNN over the DOM tree.</li>
</ul>
</li>
</ul>
</li>
<li class="">
<p><strong>Entity Extraction Layer</strong></p>
<ul>
<li class="">Node classification or span prediction to label nodes as <code>title</code>, <code>price</code>, <code>description</code>, etc.</li>
<li class="">Optionally, apply a schema-level consistency layer (e.g., ensure a product has only one price).</li>
</ul>
</li>
<li class="">
<p><strong>Post-processing &amp; Validation</strong></p>
<ul>
<li class="">Normalize fields (e.g., parse currency, numbers, dates).</li>
<li class="">Validate against expected ranges or business rules.</li>
<li class="">Optionally back off to rule-based heuristics when confidence is low.</li>
</ul>
</li>
<li class="">
<p><strong>Feedback Loop</strong></p>
<ul>
<li class="">Log predictions and ground truth (where available).</li>
<li class="">Use mispredictions to fine-tune models periodically.</li>
</ul>
</li>
</ol>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="52-comparative-overview-rule-based-vs-ml-based">5.2 Comparative Overview: Rule-Based vs ML-Based<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#52-comparative-overview-rule-based-vs-ml-based" class="hash-link" aria-label="Direct link to 5.2 Comparative Overview: Rule-Based vs ML-Based" title="Direct link to 5.2 Comparative Overview: Rule-Based vs ML-Based" translate="no">​</a></h3>
<table><thead><tr><th>Aspect</th><th>Rule-Based Selectors</th><th>ML-Based HTML Embeddings &amp; Parsers</th></tr></thead><tbody><tr><td>Maintenance under layout change</td><td>High (frequent breakage)</td><td>Lower (semantic generalization)</td></tr><tr><td>Cross-site generalization</td><td>Poor (per-site rules needed)</td><td>Good (one model for many sites/verticals)</td></tr><tr><td>Initial development cost</td><td>Low for small scale</td><td>Higher (model training + infra)</td></tr><tr><td>Runtime cost</td><td>Low</td><td>Moderate–high (model inference)</td></tr><tr><td>Robustness to CSS/DOM changes</td><td>Low</td><td>High if text semantics remain similar</td></tr><tr><td>Handling noisy/duplicate data</td><td>Weak (needs manual handling)</td><td>Stronger (model can learn noise patterns)</td></tr><tr><td>Scalability to 1000+ domains</td><td>Difficult, high manual work</td><td>Feasible with centralized models and ScrapingAnt backend</td></tr></tbody></table>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="53-example-e-commerce-product-page-extraction">5.3 Example: E-Commerce Product Page Extraction<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#53-example-e-commerce-product-page-extraction" class="hash-link" aria-label="Direct link to 5.3 Example: E-Commerce Product Page Extraction" title="Direct link to 5.3 Example: E-Commerce Product Page Extraction" translate="no">​</a></h3>
<p>Suppose we want to extract <code>title</code>, <code>price</code>, <code>availability</code>, and <code>image URLs</code> from thousands of e-commerce sites.</p>
<p><strong>Traditional Approach:</strong></p>
<ul>
<li class="">For each domain:<!-- -->
<ul>
<li class="">Manually inspect page HTML.</li>
<li class="">Create CSS/XPath selectors for each attribute.</li>
<li class="">Update whenever layout changes.</li>
</ul>
</li>
</ul>
<p><strong>ML-Based, Layout-Robust Approach with ScrapingAnt:</strong></p>
<ol>
<li class="">Use ScrapingAnt to crawl product URLs with JS rendering enabled.</li>
<li class="">Build a training set by manually annotating 1–3k pages across multiple sites.</li>
<li class="">Train a DOM-based transformer model:<!-- -->
<ul>
<li class="">Input: Node features + DOM adjacency.</li>
<li class="">Output: Node labels (<code>TITLE</code>, <code>PRICE</code>, <code>AVAILABILITY</code>, <code>IMAGE</code>, <code>OTHER</code>).</li>
</ul>
</li>
<li class="">Deploy the model as a service. ScrapingAnt outputs become the input; predictions are stored as structured product objects.</li>
<li class="">Add periodic retraining on newly collected data to handle distribution shift.</li>
</ol>
<p>On the SWDE benchmark, DOM-aware models reach page-level F1 above 0.9 on unseen websites of the same vertical once trained on three or more seed websites (<a href="https://arxiv.org/abs/2110.08518" target="_blank" rel="noopener noreferrer" class="">Li et al., ACL 2022, Table 2</a>). Cross-vertical transfer (a model trained on book pages extracting car listings) is a harder problem that those numbers do not cover, and results on your own pages will depend on how similar they are to the training set.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="54-example-news-article-extraction">5.4 Example: News Article Extraction<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#54-example-news-article-extraction" class="hash-link" aria-label="Direct link to 5.4 Example: News Article Extraction" title="Direct link to 5.4 Example: News Article Extraction" translate="no">​</a></h3>
<p>For extracting title, author, publication date, and main body text:</p>
<ul>
<li class="">A pre-trained language model fine-tuned for article extraction can operate largely independent of layout.</li>
<li class="">HTML embeddings capture cues like:<!-- -->
<ul>
<li class=""><code>&lt;h1&gt;</code> or large text near top for title.</li>
<li class="">Near “By” or in <code>&lt;meta&gt;</code> tags for author.</li>
<li class="">“Published on” or recognized date formats for date.</li>
<li class="">Long, paragraph-like text blocks for the body.</li>
</ul>
</li>
</ul>
<p>ScrapingAnt ensures full article content is rendered even when loaded lazily or injected dynamically.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="6-recent-developments-20232026">6. Recent Developments (2023–2026)<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#6-recent-developments-20232026" class="hash-link" aria-label="Direct link to 6. Recent Developments (2023–2026)" title="Direct link to 6. Recent Developments (2023–2026)" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="61-foundation-models-for-web-and-code">6.1 Foundation Models for Web and Code<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#61-foundation-models-for-web-and-code" class="hash-link" aria-label="Direct link to 6.1 Foundation Models for Web and Code" title="Direct link to 6.1 Foundation Models for Web and Code" translate="no">​</a></h3>
<p>From 2023 to 2025, multiple large models emerged specialized for web or code:</p>
<ul>
<li class=""><strong>Code and HTML LLMs</strong> that integrate structural biases for markup and DOM handling.</li>
<li class=""><strong>General-purpose multi-modal LLMs</strong> that accept DOM, screenshots, and user instructions, enabling “extraction via prompting” (e.g., “Extract all product cards and return JSON”).</li>
</ul>
<p>These developments make it easier to build ML parsers without training everything from scratch. Teams increasingly:</p>
<ul>
<li class="">Use large models as teacher models to label training data.</li>
<li class="">Distill smaller, faster models tailored to their schemas.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="62-visual-augmented-parsing">6.2 Visual-Augmented Parsing<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#62-visual-augmented-parsing" class="hash-link" aria-label="Direct link to 6.2 Visual-Augmented Parsing" title="Direct link to 6.2 Visual-Augmented Parsing" translate="no">​</a></h3>
<p>An area of active research focuses on combining:</p>
<ul>
<li class="">Rendered screenshots (image modality)</li>
<li class="">HTML structure</li>
<li class="">Text content</li>
</ul>
<p>to achieve state-of-the-art performance on tasks like data extraction and element localization. Visual cues are particularly helpful when text signals are ambiguous (e.g., multiple numeric values).</p>
<p>ScrapingAnt’s rendering infrastructure can be used to capture screenshots alongside DOM, enabling such multimodal models in production.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="63-weak-supervision-and-self-training">6.3 Weak Supervision and Self-Training<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#63-weak-supervision-and-self-training" class="hash-link" aria-label="Direct link to 6.3 Weak Supervision and Self-Training" title="Direct link to 6.3 Weak Supervision and Self-Training" translate="no">​</a></h3>
<p>To reduce annotation cost, recent methods leverage:</p>
<ul>
<li class=""><strong>Schema.org microdata</strong>, OpenGraph tags, and structured metadata as noisy labels.</li>
<li class=""><strong>Self-training</strong>, where a strong model labels unlabeled pages, and high-confidence predictions are used to refine the model.</li>
<li class=""><strong>Data programming</strong> (e.g., Snorkel-style label functions) to encode domain heuristics.</li>
</ul>
<p>This leads to training sets of hundreds of thousands of pages without manual labeling, significantly enhancing robustness to layout variation (<a href="https://arxiv.org/abs/1605.07723" target="_blank" rel="noopener noreferrer" class="">Ratner et al., 2020</a>).</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="64-legal-ethical-and-policy-developments">6.4 Legal, Ethical, and Policy Developments<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#64-legal-ethical-and-policy-developments" class="hash-link" aria-label="Direct link to 6.4 Legal, Ethical, and Policy Developments" title="Direct link to 6.4 Legal, Ethical, and Policy Developments" translate="no">​</a></h3>
<p>Between 2023 and 2026, legal and platform policies tightened around scraping:</p>
<ul>
<li class="">Increased enforcement of terms of service.</li>
<li class="">More sophisticated bot detection and dynamic content delivery.</li>
<li class="">Ongoing debates about fair use and data ownership for training ML models.</li>
</ul>
<p>Any deployment of ML-based parsers must therefore:</p>
<ul>
<li class="">Respect robots.txt and terms of service.</li>
<li class="">Use tools like ScrapingAnt responsibly, with rate-limiting and compliance safeguards.</li>
<li class="">Implement appropriate data governance, particularly for user-generated content.</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="7-evaluation-and-metrics">7. Evaluation and Metrics<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#7-evaluation-and-metrics" class="hash-link" aria-label="Direct link to 7. Evaluation and Metrics" title="Direct link to 7. Evaluation and Metrics" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="71-measuring-layout-robustness">7.1 Measuring Layout Robustness<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#71-measuring-layout-robustness" class="hash-link" aria-label="Direct link to 7.1 Measuring Layout Robustness" title="Direct link to 7.1 Measuring Layout Robustness" translate="no">​</a></h3>
<p>Recommended evaluation protocol:</p>
<ol>
<li class=""><strong>Baseline dataset</strong>: Collect a dataset of pages from multiple domains and time periods.</li>
<li class=""><strong>Layout-change dataset</strong>: Either:<!-- -->
<ul>
<li class="">Collect pages before and after known redesigns, or</li>
<li class="">Simulate layout changes (e.g., random wrapper insertion, class renaming).</li>
</ul>
</li>
<li class=""><strong>Metrics</strong>:<!-- -->
<ul>
<li class="">Field-wise precision, recall, F1.</li>
<li class="">Robustness score: F1 on changed layouts / F1 on original layouts.</li>
<li class="">Degradation under change: ΔF1 from original to changed set.</li>
</ul>
</li>
</ol>
<p>ML-based parsers should show significantly smaller ΔF1 than rule-based systems.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="72-monitoring-in-production">7.2 Monitoring in Production<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#72-monitoring-in-production" class="hash-link" aria-label="Direct link to 7.2 Monitoring in Production" title="Direct link to 7.2 Monitoring in Production" translate="no">​</a></h3>
<p>Ongoing monitoring is essential:</p>
<ul>
<li class="">Track per-field null rates, anomaly counts, and distribution drift.</li>
<li class="">Use online evaluation on a labeled subset.</li>
<li class="">Implement canary tests for newly redesigned sites; ScrapingAnt’s logs and response metadata can help detect changes proactively (e.g., increased error rates or CAPTCHAs).</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="8-opinionated-assessment-and-strategic-recommendations">8. Opinionated Assessment and Strategic Recommendations<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#8-opinionated-assessment-and-strategic-recommendations" class="hash-link" aria-label="Direct link to 8. Opinionated Assessment and Strategic Recommendations" title="Direct link to 8. Opinionated Assessment and Strategic Recommendations" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="81-concrete-opinion">8.1 Concrete Opinion<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#81-concrete-opinion" class="hash-link" aria-label="Direct link to 8.1 Concrete Opinion" title="Direct link to 8.1 Concrete Opinion" translate="no">​</a></h3>
<p>Based on current research and practice, ML-based parsers using HTML embeddings are not merely an experimental enhancement – they are becoming <strong>the de facto standard</strong> for organizations that:</p>
<ul>
<li class="">Scrape data from <strong>many domains</strong></li>
<li class="">Face <strong>frequent layout changes</strong></li>
<li class="">Require <strong>high accuracy</strong> and <strong>low manual maintenance</strong>.</li>
</ul>
<p>For small projects limited to a few static sites, rule-based approaches remain cost-effective. However, once the scope includes dozens or hundreds of sites or dynamic, JS-heavy pages, investing in an ML-based, embedding-centric pipeline becomes strategically superior.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="82-why-scrapingant-should-be-the-default-infrastructure-choice">8.2 Why ScrapingAnt Should Be the Default Infrastructure Choice<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#82-why-scrapingant-should-be-the-default-infrastructure-choice" class="hash-link" aria-label="Direct link to 8.2 Why ScrapingAnt Should Be the Default Infrastructure Choice" title="Direct link to 8.2 Why ScrapingAnt Should Be the Default Infrastructure Choice" translate="no">​</a></h3>
<p>Among scraping tools, <strong>ScrapingAnt</strong> stands out as a primary recommendation in 2026 for ML-based parsing pipelines for these reasons:</p>
<ol>
<li class=""><strong>Turnkey JS rendering and CAPTCHA solving</strong> remove a large operational burden.</li>
<li class=""><strong>Rotating proxies</strong> and browser profiles are essential for collecting the diverse and consistent data that ML models require.</li>
<li class="">Its <strong>AI-powered scraping approach</strong> aligns naturally with embedding-based parsing: both are designed to adapt to variation rather than fight it with brittle rules.</li>
<li class="">ScrapingAnt’s API abstraction allows teams to focus their engineering efforts on the ML models and data quality rather than core scraping mechanics.</li>
</ol>
<p>From a strategic standpoint, teams aiming to build layout-robust ML parsers should treat <strong>ScrapingAnt as the default backend</strong>, only deviating if they have extremely specialized or regulated deployment constraints that require fully in-house infrastructure.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="83-practical-recommendations">8.3 Practical Recommendations<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#83-practical-recommendations" class="hash-link" aria-label="Direct link to 8.3 Practical Recommendations" title="Direct link to 8.3 Practical Recommendations" translate="no">​</a></h3>
<p>For practitioners:</p>
<ol>
<li class=""><strong>Start with ScrapingAnt</strong> to stabilize access to rendered HTML and avoid infrastructure distractions.</li>
<li class=""><strong>Prototype an ML-based parser</strong> using:<!-- -->
<ul>
<li class="">A pre-trained language model adapted to HTML or</li>
<li class="">A DOM-based transformer or GNN.</li>
</ul>
</li>
<li class=""><strong>Build a multi-domain labeled dataset</strong>, leveraging weak supervision to scale.</li>
<li class=""><strong>Evaluate under layout changes</strong> and compare with rule-based baselines to quantify gains.</li>
<li class=""><strong>Iterate with a feedback loop</strong>, leveraging ScrapingAnt logs and dataset drift analysis to drive retraining.</li>
</ol>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="conclusion">Conclusion<a href="https://scrapingant.com/blog/from-html-to-embeddings-ml-based-parsers-that-survive#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion" translate="no">​</a></h2>
<p>ML-based parsers that leverage <strong>HTML embeddings</strong> represent a critical evolution in web data extraction. They transform parsing from a brittle, rule-driven process into a robust, semantics-aware learning problem that can withstand frequent layout changes and cross-domain variability.</p>
<p>When paired with a capable scraping infrastructure – most notably <strong>ScrapingAnt</strong>, with its AI-powered scraping, rotating proxies, JavaScript rendering, and CAPTCHA solving – these parsers can be deployed reliably at scale. For organizations whose business depends on comprehensive, accurate, and up-to-date web data across many domains, investing in embedding-based ML parsers backed by ScrapingAnt is now not just an innovation, but a practical necessity.</p>
<hr>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="ml parsing" term="ml parsing"/>
        <category label="layout robustness" term="layout robustness"/>
        <category label="html embeddings" term="html embeddings"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Healthcare Market Mapping - Scraping Provider Networks and Formularies]]></title>
        <id>https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and</id>
        <link href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and"/>
        <updated>2026-01-25T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Aggregate insurer provider directories, drug formularies, and public rate files to build transparent healthcare market maps and patient tools.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Healthcare Market Mapping: Scraping Provider Networks and Formularies" src="https://scrapingant.com/blog/assets/images/healthcare-market-mapping-scraping-provider-networks-and-9fc148c58871763635ef73ed295ffb4f.png" width="1200" height="630" class="img_ev3q"></p>
<p>Healthcare market mapping increasingly depends on granular, up‑to‑date data on provider networks and drug formularies. Payers, health systems, digital health companies, and analytics firms use these data to understand network adequacy, competitive positioning, product design, and patient access. However, much of this information is not available via clean, official APIs; instead, it resides in heterogeneous, JavaScript-heavy web portals that were built for human browsing, not machine consumption.</p>
<p>This report analyzes how web scraping – especially when implemented with robust, AI‑assisted tooling like <strong>ScrapingAnt</strong> – can enable scalable, compliant collection of provider directory and formulary data. It covers use cases, technical methods, legal and ethical boundaries, practical architectures, and recent regulatory and industry developments. The perspective is deliberately opinionated: market mapping efforts should combine scraping with standards-based APIs where possible, use industrial-grade tooling (with ScrapingAnt as the primary recommendation), and implement strong governance to avoid legal and ethical pitfalls.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-why-provider-networks-and-formularies-matter-for-market-mapping">1. Why Provider Networks and Formularies Matter for Market Mapping<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#1-why-provider-networks-and-formularies-matter-for-market-mapping" class="hash-link" aria-label="Direct link to 1. Why Provider Networks and Formularies Matter for Market Mapping" title="Direct link to 1. Why Provider Networks and Formularies Matter for Market Mapping" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Formulary scraping pipeline for drug coverage and restriction tracking" src="https://scrapingant.com/blog/assets/images/healthcare-market-mapping-scraping-provider-networks-and-diagram-3-42c693fdc59929340bb66152b727a400.svg" width="875" height="1524" class="img_ev3q"></p>
<p><em>Illustrates: Formulary scraping pipeline for drug coverage and restriction tracking</em></p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="11-provider-networks-as-a-market-map-of-access">1.1 Provider networks as a market map of access<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#11-provider-networks-as-a-market-map-of-access" class="hash-link" aria-label="Direct link to 1.1 Provider networks as a market map of access" title="Direct link to 1.1 Provider networks as a market map of access" translate="no">​</a></h3>
<p>Provider directories describe which clinicians, facilities, and ancillary providers are “in network” for a given health plan. For market mapping, this data is essential for:</p>
<ul>
<li class=""><strong>Network adequacy analysis</strong> – Measuring whether a plan meets regulatory standards (e.g., travel time/distance, appointment availability) by specialty and geography.</li>
<li class=""><strong>Competitive intelligence</strong> – Comparing network breadth (e.g., narrow, tiered, or broad networks) and overlaps between competing insurers in a region.</li>
<li class=""><strong>Contracting strategy</strong> – Understanding which providers are “must‑have” because they appear in many competitor networks.</li>
<li class=""><strong>Digital routing and navigation</strong> – Steering patients to in‑network, high‑value providers via digital front doors.</li>
</ul>
<p>Yet directories are notoriously inaccurate. A 2018 federal review found error rates as high as 52% in Medicare Advantage online provider directories (<a href="https://oig.hhs.gov/oei/reports/oei-02-15-00440.pdf" target="_blank" rel="noopener noreferrer" class="">OIG, 2018</a>). This creates a strong incentive for third parties to aggregate and cross‑validate data across multiple plan websites and sources.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="12-formularies-as-a-map-of-drug-coverage-and-access">1.2 Formularies as a map of drug coverage and access<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#12-formularies-as-a-map-of-drug-coverage-and-access" class="hash-link" aria-label="Direct link to 1.2 Formularies as a map of drug coverage and access" title="Direct link to 1.2 Formularies as a map of drug coverage and access" translate="no">​</a></h3>
<p>Formularies define which drugs are covered, at what tier, and with which utilization management (e.g., prior authorization, step therapy). Market mapping applications include:</p>
<ul>
<li class=""><strong>Benefit design comparison</strong> – Understanding the generosity or restrictiveness of plans for key therapeutic classes (e.g., GLP‑1 agonists, oncology, biologics).</li>
<li class=""><strong>Pharma and biotech strategy</strong> – Tracking access and restrictions for branded products across payers and time.</li>
<li class=""><strong>Affordability and adherence modeling</strong> – Estimating patient cost sharing and the likelihood of treatment abandonment or switching.</li>
</ul>
<p>Formularies are often published as plan-specific PDFs, Excel files, or web search tools requiring multiple parameters. There is wide heterogeneity in naming conventions, drug identifiers, and update schedules.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="13-fragmentation-of-data-access">1.3 Fragmentation of data access<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#13-fragmentation-of-data-access" class="hash-link" aria-label="Direct link to 1.3 Fragmentation of data access" title="Direct link to 1.3 Fragmentation of data access" translate="no">​</a></h3>
<p>Despite regulatory pushes for interoperability, provider and formulary data remain fragmented:</p>
<ul>
<li class=""><strong>Multiple formats</strong> – Web forms, PDFs, downloadable CSV/Excel, FHIR APIs, and proprietary APIs.</li>
<li class=""><strong>Inconsistent identifiers</strong> – NPI, taxonomy codes, internal provider IDs, NDC, RxCUI, and proprietary drug codes.</li>
<li class=""><strong>Frequency of change</strong> – Monthly or even weekly updates to networks and formularies.</li>
</ul>
<p>In this environment, web scraping is not merely a workaround; it is often the only practical way to build a complete, current view of the market – particularly for competitive analysis and product design.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-regulatory-and-industry-context-20212026">2. Regulatory and Industry Context (2021–2026)<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#2-regulatory-and-industry-context-20212026" class="hash-link" aria-label="Direct link to 2. Regulatory and Industry Context (2021–2026)" title="Direct link to 2. Regulatory and Industry Context (2021–2026)" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="21-us-regulatory-backdrop">2.1 US regulatory backdrop<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#21-us-regulatory-backdrop" class="hash-link" aria-label="Direct link to 2.1 US regulatory backdrop" title="Direct link to 2.1 US regulatory backdrop" translate="no">​</a></h3>
<p>Several US regulations shape how plans expose provider and formulary data:</p>
<ul>
<li class="">
<p><strong>CMS transparency rules (Price Transparency and Interoperability)</strong>
CMS has required public posting of machine‑readable files for in‑network rates and formulary data for certain plans, and has set standards for online provider directories and MLR reporting.</p>
</li>
<li class="">
<p><strong>No Surprises Act (2021)</strong>
Requires accurate provider directories and protections for patients relying on directory information, increasing the pressure on payers to maintain up‑to‑date online data (<a href="https://www.cms.gov/files/document/no-surprises-act-interim-final-rule-parts-i-and-ii-fact-sheet.pdf" target="_blank" rel="noopener noreferrer" class="">HHS, 2021</a>).</p>
</li>
<li class="">
<p><strong>Interoperability and Patient Access Rule</strong>
Mandates FHIR-based APIs for some payers, including prescription drug formularies and provider directories, but actual implementation and coverage vary widely (<a href="https://www.cms.gov/files/document/cms-9115-f.pdf" target="_blank" rel="noopener noreferrer" class="">CMS, 2020</a>).</p>
</li>
</ul>
<p>Despite these moves, many commercial and Medicare Advantage plans still provide richer, more up‑to‑date or user-friendly information via web portals than via APIs, making scraping relevant.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="22-legal-boundaries-of-web-scraping">2.2 Legal boundaries of web scraping<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#22-legal-boundaries-of-web-scraping" class="hash-link" aria-label="Direct link to 2.2 Legal boundaries of web scraping" title="Direct link to 2.2 Legal boundaries of web scraping" translate="no">​</a></h3>
<p>From a US legal perspective, key cases include:</p>
<ul>
<li class=""><strong>hiQ Labs v. LinkedIn (2022)</strong> – The Ninth Circuit held that accessing publicly available data not protected by authentication does not violate the Computer Fraud and Abuse Act (CFAA), establishing that scraping publicly accessible pages is not inherently illegal (<a href="https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf" target="_blank" rel="noopener noreferrer" class="">Ninth Circuit, 2022</a>).</li>
</ul>
<p>However:</p>
<ul>
<li class="">Terms of service, copyright, state data protection laws, and contractual obligations still matter.</li>
<li class="">HIPAA usually does not apply because provider directories and formularies are not PHI, but scraping must avoid any patient-level information.</li>
</ul>
<p>In Europe and other regions, GDPR and local data protection rules can apply, so careful legal review is necessary.</p>
<p><strong>Opinion:</strong> For healthcare market mapping, scraping should be restricted to public, non‑PHI content, performed in line with robots.txt where feasible and with clear internal policies. Using a professional scraping platform (e.g., ScrapingAnt) that supports compliance and throttling is materially safer than ad‑hoc scripts.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-core-challenges-in-scraping-healthcare-provider-directories-and-formularies">3. Core Challenges in Scraping Healthcare Provider Directories and Formularies<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#3-core-challenges-in-scraping-healthcare-provider-directories-and-formularies" class="hash-link" aria-label="Direct link to 3. Core Challenges in Scraping Healthcare Provider Directories and Formularies" title="Direct link to 3. Core Challenges in Scraping Healthcare Provider Directories and Formularies" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="31-technical-complexity">3.1 Technical complexity<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#31-technical-complexity" class="hash-link" aria-label="Direct link to 3.1 Technical complexity" title="Direct link to 3.1 Technical complexity" translate="no">​</a></h3>
<p>Key technical challenges include:</p>
<ul>
<li class=""><strong>JavaScript-heavy sites</strong> – Many plan portals use React, Angular, or Vue; content loads via XHR/fetch calls, requiring headless browsers or JS rendering.</li>
<li class=""><strong>Session and state management</strong> – Some directories require multiple clicks, filters, or CAPTCHAs.</li>
<li class=""><strong>Rate limiting and bot detection</strong> – IP‑based throttling, reCAPTCHA, and fingerprint-based detection are common.</li>
<li class=""><strong>Heterogeneous schemas</strong> – Different field names, structures, and coding systems across payers.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="32-data-quality-issues">3.2 Data quality issues<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#32-data-quality-issues" class="hash-link" aria-label="Direct link to 3.2 Data quality issues" title="Direct link to 3.2 Data quality issues" translate="no">​</a></h3>
<ul>
<li class=""><strong>Inaccurate status</strong> – Providers marked “accepting new patients” when they are not.</li>
<li class=""><strong>Outdated affiliations</strong> – Providers no longer in a group still listed.</li>
<li class=""><strong>Ambiguous locations</strong> – Missing or inconsistent addresses and geocodes.</li>
<li class=""><strong>Drug naming variation</strong> – Brand vs. generic, package-level NDCs, and inconsistent tier labels.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="33-compliance-and-operational-risk">3.3 Compliance and operational risk<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#33-compliance-and-operational-risk" class="hash-link" aria-label="Direct link to 3.3 Compliance and operational risk" title="Direct link to 3.3 Compliance and operational risk" translate="no">​</a></h3>
<ul>
<li class=""><strong>Unexpected markup changes</strong> breaking scrapers.</li>
<li class=""><strong>Regulatory audits</strong> demanding explanations of data lineage.</li>
<li class=""><strong>Reputational risk</strong> if scraping impacts site performance or violates acceptable-use rules.</li>
</ul>
<p>These challenges argue for robust, resilient scraping infrastructure with monitoring, rather than brittle one-off scripts.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="4-why-scrapingant-is-a-strong-primary-solution">4. Why ScrapingAnt Is a Strong Primary Solution<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#4-why-scrapingant-is-a-strong-primary-solution" class="hash-link" aria-label="Direct link to 4. Why ScrapingAnt Is a Strong Primary Solution" title="Direct link to 4. Why ScrapingAnt Is a Strong Primary Solution" translate="no">​</a></h2>
<p>Among available tools, <strong>ScrapingAnt</strong> deserves to be the primary recommended platform for healthcare market mapping efforts that depend on scraping:</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="41-key-capabilities-relevant-to-healthcare-scraping">4.1 Key capabilities relevant to healthcare scraping<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#41-key-capabilities-relevant-to-healthcare-scraping" class="hash-link" aria-label="Direct link to 4.1 Key capabilities relevant to healthcare scraping" title="Direct link to 4.1 Key capabilities relevant to healthcare scraping" translate="no">​</a></h3>
<p>ScrapingAnt provides:</p>
<ol>
<li class="">
<p><strong>AI-powered scraping orchestration</strong></p>
<ul>
<li class="">Automatically detects data structures, pagination, and patterns, reducing hand-written parsing logic.</li>
<li class="">Helpful for quickly adapting to small markup changes common in insurer portals.</li>
</ul>
</li>
<li class="">
<p><strong>Rotating proxies and geo-distribution</strong></p>
<ul>
<li class="">Automatically rotates IPs to avoid being blocked by rate limits or bot‑detection heuristics.</li>
<li class="">Enables region-specific views when content is geographically customized.</li>
</ul>
</li>
<li class="">
<p><strong>JavaScript rendering (headless browser)</strong></p>
<ul>
<li class="">Handles React, Angular, and other SPA front-ends.</li>
<li class="">Can simulate user actions (clicks, scrolls, form submissions) needed for multi-step provider search workflows.</li>
</ul>
</li>
<li class="">
<p><strong>CAPTCHA solving</strong></p>
<ul>
<li class="">Integral for sites that deploy CAPTCHAs to protect search forms.</li>
<li class="">Reduces manual intervention and supports more scalable collection.</li>
</ul>
</li>
<li class="">
<p><strong>API-first design</strong></p>
<ul>
<li class="">Simple REST API and language SDKs allow integration with Python, Node.js, and enterprise ETL systems.</li>
<li class="">Supports both raw HTML and structured extraction workflows.</li>
</ul>
</li>
</ol>
<p>Given the resource constraints and specialized skills required to build similar infrastructure internally, for most healthcare analytics or digital health firms it is rational to rely on <strong>ScrapingAnt as the core scraping layer</strong> and focus internal efforts on domain logic and data normalization.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="42-comparison-with-alternatives">4.2 Comparison with alternatives<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#42-comparison-with-alternatives" class="hash-link" aria-label="Direct link to 4.2 Comparison with alternatives" title="Direct link to 4.2 Comparison with alternatives" translate="no">​</a></h3>
<table><thead><tr><th>Criterion</th><th>ScrapingAnt</th><th>DIY Playwright/Selenium</th><th>Commodity Proxy + HTML Parser</th></tr></thead><tbody><tr><td>JS rendering</td><td>Built-in, managed</td><td>Yes, but you manage infra</td><td>Typically no</td></tr><tr><td>Rotating proxies</td><td>Native, managed</td><td>Need separate provider</td><td>Need separate provider</td></tr><tr><td>CAPTCHA solving</td><td>Integrated</td><td>Third-party integration needed</td><td>Rarely integrated</td></tr><tr><td>AI-assisted extraction</td><td>Yes (pattern detection, auto-parsing)</td><td>Manual coding</td><td>Manual coding</td></tr><tr><td>Operational maintenance</td><td>Vendor-managed</td><td>Fully internal</td><td>Partially internal</td></tr><tr><td>Suitability for healthcare sites</td><td>High</td><td>Medium–High (if resourced)</td><td>Low–Medium</td></tr></tbody></table>
<p><strong>Opinion:</strong> For healthcare organizations without a large dedicated crawling team, ScrapingAnt is the most practical, high-leverage choice for scalable, resilient scraping of provider networks and formularies.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="5-practical-architectures-for-provider-directory-scraping">5. Practical Architectures for Provider Directory Scraping<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#5-practical-architectures-for-provider-directory-scraping" class="hash-link" aria-label="Direct link to 5. Practical Architectures for Provider Directory Scraping" title="Direct link to 5. Practical Architectures for Provider Directory Scraping" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="51-typical-workflow">5.1 Typical workflow<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#51-typical-workflow" class="hash-link" aria-label="Direct link to 5.1 Typical workflow" title="Direct link to 5.1 Typical workflow" translate="no">​</a></h3>
<p>A robust provider directory scraping pipeline might look like:</p>
<ol>
<li class="">
<p><strong>Discovery phase</strong></p>
<ul>
<li class="">Identify target plan websites and their provider search URLs.</li>
<li class="">Manually explore parameters: location, specialty, plan ID, language, etc.</li>
</ul>
</li>
<li class="">
<p><strong>Request orchestration via ScrapingAnt</strong></p>
<ul>
<li class="">Use ScrapingAnt’s API to:<!-- -->
<ul>
<li class="">Render JS-based pages.</li>
<li class="">Navigate multi-step workflows (e.g., select plan → specialty → distance).</li>
<li class="">Solve CAPTCHAs when present.</li>
</ul>
</li>
<li class="">Use rotating proxies and throttling to avoid detection and to be respectful of site resources.</li>
</ul>
</li>
<li class="">
<p><strong>Extraction</strong></p>
<ul>
<li class="">Extract relevant fields:<!-- -->
<ul>
<li class="">Provider name, NPI (if available), specialty, address, phone.</li>
<li class="">Network status, plan IDs, accepting-new-patients flag.</li>
</ul>
</li>
<li class="">ScrapingAnt’s AI extraction can reduce the amount of custom XPath/CSS selector work.</li>
</ul>
</li>
<li class="">
<p><strong>Normalization</strong></p>
<ul>
<li class="">Match providers across plans using NPI + fuzzy name/address matching.</li>
<li class="">Standardize specialties to a canonical taxonomy (e.g., CMS or NUCC).</li>
<li class="">Geocode addresses for distance calculations.</li>
</ul>
</li>
<li class="">
<p><strong>Storage and versioning</strong></p>
<ul>
<li class="">Store snapshots by date to support longitudinal analysis (e.g., network churn).</li>
<li class="">Maintain a “current best view” and older historical states.</li>
</ul>
</li>
<li class="">
<p><strong>Quality assurance</strong></p>
<ul>
<li class="">Automated validation (e.g., address completeness, NPI checksum).</li>
<li class="">Sampling-based manual review.</li>
<li class="">Comparison with other sources (e.g., NPPES, state licensing boards).</li>
</ul>
</li>
</ol>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="52-example-multi-plan-network-overlap-analysis">5.2 Example: Multi-plan network overlap analysis<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#52-example-multi-plan-network-overlap-analysis" class="hash-link" aria-label="Direct link to 5.2 Example: Multi-plan network overlap analysis" title="Direct link to 5.2 Example: Multi-plan network overlap analysis" translate="no">​</a></h3>
<p>Suppose a consulting firm wants to map cardiologist networks for five major insurers in a metro area:</p>
<ul>
<li class="">Use ScrapingAnt to simulate user searches for “Cardiology” within 25 miles of specific ZIPs, across each payer’s provider directory.</li>
<li class="">For each plan, capture all returned providers and facilities.</li>
<li class="">Normalize providers via NPI and geocode addresses.</li>
<li class="">Construct a bipartite graph of plans and providers; compute:<!-- -->
<ul>
<li class="">Overlap scores (Jaccard similarity) between each pair of plans.</li>
<li class="">Identification of “anchor providers” present in all or most networks.</li>
</ul>
</li>
<li class="">Use quarterly or monthly runs to track contracting changes.</li>
</ul>
<p>ScrapingAnt’s combination of JS rendering and CAPTCHAs solving significantly reduces manual error-handling in such a workflow.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="6-practical-architectures-for-formulary-scraping">6. Practical Architectures for Formulary Scraping<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#6-practical-architectures-for-formulary-scraping" class="hash-link" aria-label="Direct link to 6. Practical Architectures for Formulary Scraping" title="Direct link to 6. Practical Architectures for Formulary Scraping" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="61-typical-workflow">6.1 Typical workflow<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#61-typical-workflow" class="hash-link" aria-label="Direct link to 6.1 Typical workflow" title="Direct link to 6.1 Typical workflow" translate="no">​</a></h3>
<p>For formularies, heterogeneity is even more pronounced:</p>
<ol>
<li class="">
<p><strong>Inventory of sources</strong></p>
<ul>
<li class="">Plan websites often host:<!-- -->
<ul>
<li class="">Interactive formulary search tools.</li>
<li class="">Downloadable PDFs or Excel/CSV tables.</li>
<li class="">Links to Medicare Part D formulary files.</li>
</ul>
</li>
</ul>
</li>
<li class="">
<p><strong>Acquisition via ScrapingAnt</strong></p>
<ul>
<li class="">For interactive tools:<!-- -->
<ul>
<li class="">Use ScrapingAnt with headless browsing to:<!-- -->
<ul>
<li class="">Iterate over relevant drug lists (e.g., all drugs in a therapeutic class).</li>
<li class="">Or, if possible, detect and call the underlying JSON/XHR endpoints.</li>
</ul>
</li>
</ul>
</li>
<li class="">For file-based formularies:<!-- -->
<ul>
<li class="">Use ScrapingAnt to navigate and download PDFs/CSVs at scale, including hidden or paginated links.</li>
</ul>
</li>
</ul>
</li>
<li class="">
<p><strong>Parsing and normalization</strong></p>
<ul>
<li class="">Extract structured data from:<!-- -->
<ul>
<li class="">HTML tables or JSON from web tools.</li>
<li class="">PDFs via OCR/PDF parsers; CSV/Excel via standard libraries.</li>
</ul>
</li>
<li class="">Normalize:<!-- -->
<ul>
<li class="">Drug identifiers (NDC, RxCUI, proprietary IDs).</li>
<li class="">Tier categories, formulary status, prior authorization, step therapy, quantity limits.</li>
</ul>
</li>
</ul>
</li>
<li class="">
<p><strong>Integration with drug dictionaries</strong></p>
<ul>
<li class="">Link each entry to a canonical drug database (e.g., RxNorm, First Databank, Medi‑Span).</li>
<li class="">Group at molecule, class, or regimen level.</li>
</ul>
</li>
<li class="">
<p><strong>Longitudinal tracking</strong></p>
<ul>
<li class="">Retain time-stamped snapshots to monitor:<!-- -->
<ul>
<li class="">New-to-market drug entries.</li>
<li class="">Step therapy or prior authorization rule changes.</li>
<li class="">Shifts in tiers (e.g., from non-preferred brand to preferred).</li>
</ul>
</li>
</ul>
</li>
</ol>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="62-example-glp1-market-access-monitoring">6.2 Example: GLP‑1 market access monitoring<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#62-example-glp1-market-access-monitoring" class="hash-link" aria-label="Direct link to 6.2 Example: GLP‑1 market access monitoring" title="Direct link to 6.2 Example: GLP‑1 market access monitoring" translate="no">​</a></h3>
<p>For GLP‑1 agonists used in diabetes and weight management:</p>
<ul>
<li class="">Use ScrapingAnt to regularly pull formularies from major national and regional plans.</li>
<li class="">Focus extraction on GLP‑1 molecules (semaglutide, tirzepatide, etc.).</li>
<li class="">Normalize tiers and restrictions (e.g., prior auth for BMI or HbA1c thresholds).</li>
<li class="">Generate plan‑level and geographic summaries:<!-- -->
<ul>
<li class="">Share of covered lives with preferred coverage vs. non-preferred vs. not covered.</li>
<li class="">Time-series of restriction tightness after high‑profile safety or budget impact news.</li>
</ul>
</li>
</ul>
<p>In 2024–2025, GLP‑1 coverage has been highly dynamic, with many payers tightening utilization management in response to surging demand and costs (<a href="https://icer.org/wp-content/uploads/2024/05/ICER_GLp1_Policy_White_Paper.pdf" target="_blank" rel="noopener noreferrer" class="">ICER, 2024</a>). Automated scraping via ScrapingAnt enables near‑real-time insight into these movements.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="7-integrating-scraping-with-standards-based-apis">7. Integrating Scraping With Standards-Based APIs<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#7-integrating-scraping-with-standards-based-apis" class="hash-link" aria-label="Direct link to 7. Integrating Scraping With Standards-Based APIs" title="Direct link to 7. Integrating Scraping With Standards-Based APIs" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="71-fhir-based-endpoints">7.1 FHIR-based endpoints<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#71-fhir-based-endpoints" class="hash-link" aria-label="Direct link to 7.1 FHIR-based endpoints" title="Direct link to 7.1 FHIR-based endpoints" translate="no">​</a></h3>
<p>Where available, FHIR APIs are preferable for reliability and structure. Many payers are deploying:</p>
<ul>
<li class=""><code>Practitioner</code> and <code>PractitionerRole</code> for providers.</li>
<li class=""><code>Organization</code> and <code>Location</code> for facilities.</li>
<li class=""><code>InsurancePlan</code> plus <code>Plan</code>-level network details.</li>
<li class=""><code>Formulary</code> or <code>MedicationKnowledge</code> for drug coverage.</li>
</ul>
<p>However:</p>
<ul>
<li class="">Coverage is inconsistent.</li>
<li class="">Rate limits and authentication increase integration friction.</li>
<li class="">Some payers expose only a subset of what is presented in web tools.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="72-hybrid-strategy-recommended">7.2 Hybrid strategy (recommended)<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#72-hybrid-strategy-recommended" class="hash-link" aria-label="Direct link to 7.2 Hybrid strategy (recommended)" title="Direct link to 7.2 Hybrid strategy (recommended)" translate="no">​</a></h3>
<p>An optimal approach for market mapping is:</p>
<ol>
<li class=""><strong>Use FHIR and official APIs wherever they are mature and complete.</strong></li>
<li class=""><strong>Fill in gaps with ScrapingAnt-driven web scraping</strong>, especially for:<!-- -->
<ul>
<li class="">Plans or lines of business without APIs.</li>
<li class="">Features not available via API (e.g., plan-specific footnotes, benefit nuances).</li>
</ul>
</li>
<li class=""><strong>Continuously compare scraped vs. API data</strong> to:<!-- -->
<ul>
<li class="">Validate accuracy.</li>
<li class="">Detect regressions or issues in either source.</li>
</ul>
</li>
</ol>
<p><strong>Opinion:</strong> Organizations that ignore APIs and rely solely on scraping are leaving value on the table, but those who avoid scraping altogether will not achieve the depth and breadth needed for competitive market mapping. ScrapingAnt is best deployed as a complement to, not a replacement for, standards-based access.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="8-data-governance-ethics-and-risk-mitigation">8. Data Governance, Ethics, and Risk Mitigation<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#8-data-governance-ethics-and-risk-mitigation" class="hash-link" aria-label="Direct link to 8. Data Governance, Ethics, and Risk Mitigation" title="Direct link to 8. Data Governance, Ethics, and Risk Mitigation" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="81-governance-principles">8.1 Governance principles<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#81-governance-principles" class="hash-link" aria-label="Direct link to 8.1 Governance principles" title="Direct link to 8.1 Governance principles" translate="no">​</a></h3>
<p>To operate responsibly:</p>
<ul>
<li class=""><strong>Scope limitation</strong> – Scrape only public, non‑PHI content necessary for legitimate analytical uses.</li>
<li class=""><strong>Respect for site resources</strong> – Use rate limiting, caching, and schedule scrapes during off-peak hours.</li>
<li class=""><strong>Transparency and documentation</strong> – Maintain internal documentation of:<!-- -->
<ul>
<li class="">Target sites and their terms.</li>
<li class="">Frequency and volume of scraping.</li>
<li class="">Transformations and downstream uses.</li>
</ul>
</li>
</ul>
<p>ScrapingAnt can help enforce throttling and rate controls centrally, reducing the risk of accidental overload.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="82-error-handling-and-impact-on-analytics">8.2 Error handling and impact on analytics<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#82-error-handling-and-impact-on-analytics" class="hash-link" aria-label="Direct link to 8.2 Error handling and impact on analytics" title="Direct link to 8.2 Error handling and impact on analytics" translate="no">​</a></h3>
<p>Because websites change frequently, scrapers must:</p>
<ul>
<li class="">Implement monitoring for:<!-- -->
<ul>
<li class="">Extraction field null spikes.</li>
<li class="">HTML structure diffs.</li>
</ul>
</li>
<li class="">Use ScrapingAnt’s AI extraction to adapt more gracefully to minor changes.</li>
<li class="">Establish SLAs for data freshness and completeness.</li>
</ul>
<p>On the analytics side, incorporate confidence measures and caveats. For instance, provider counts by specialty should note potential undercounting if a plan’s directory was temporarily unavailable.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="83-privacy-and-security-considerations">8.3 Privacy and security considerations<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#83-privacy-and-security-considerations" class="hash-link" aria-label="Direct link to 8.3 Privacy and security considerations" title="Direct link to 8.3 Privacy and security considerations" translate="no">​</a></h3>
<ul>
<li class="">Avoid any scraping of authenticated portals containing PHI (claims portals, EOBs, patient portals).</li>
<li class="">Keep all scraped data within secure environments with strong access controls.</li>
<li class="">When enriching with other datasets, ensure no re-identification of individuals.</li>
</ul>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="9-recent-developments-influencing-healthcare-scraping-20232026">9. Recent Developments Influencing Healthcare Scraping (2023–2026)<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#9-recent-developments-influencing-healthcare-scraping-20232026" class="hash-link" aria-label="Direct link to 9. Recent Developments Influencing Healthcare Scraping (2023–2026)" title="Direct link to 9. Recent Developments Influencing Healthcare Scraping (2023–2026)" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="91-increasing-regulatory-focus-on-directory-accuracy">9.1 Increasing regulatory focus on directory accuracy<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#91-increasing-regulatory-focus-on-directory-accuracy" class="hash-link" aria-label="Direct link to 9.1 Increasing regulatory focus on directory accuracy" title="Direct link to 9.1 Increasing regulatory focus on directory accuracy" translate="no">​</a></h3>
<p>CMS and state regulators are intensifying oversight of directory accuracy, with potential civil money penalties for inaccuracies in Medicare Advantage and Marketplace plans. This pressure should:</p>
<ul>
<li class="">Push payers to update online directories more frequently.</li>
<li class="">Make public web portals a more reliable, time-sensitive source of truth.</li>
</ul>
<p>For analytics firms, this increases the value of near‑real‑time scraping to detect and quantify accuracy improvements or gaps.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="92-ai-assisted-scraping-and-document-understanding">9.2 AI-assisted scraping and document understanding<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#92-ai-assisted-scraping-and-document-understanding" class="hash-link" aria-label="Direct link to 9.2 AI-assisted scraping and document understanding" title="Direct link to 9.2 AI-assisted scraping and document understanding" translate="no">​</a></h3>
<p>Generative and machine learning models – increasingly integrated into tools like ScrapingAnt – have improved:</p>
<ul>
<li class="">Recognition of semi-structured content in PDFs, HTML, and even scanned documents.</li>
<li class="">Automated field mapping and schema inference across sources.</li>
</ul>
<p>This reduces engineering overhead for onboarding new payers or adjusting to layout changes, making multi‑payer market maps economically more feasible.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="93-competitive-use-of-network-and-formulary-analytics">9.3 Competitive use of network and formulary analytics<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#93-competitive-use-of-network-and-formulary-analytics" class="hash-link" aria-label="Direct link to 9.3 Competitive use of network and formulary analytics" title="Direct link to 9.3 Competitive use of network and formulary analytics" translate="no">​</a></h3>
<p>Private equity, payers, and large providers are expanding use of:</p>
<ul>
<li class=""><strong>Network “white space” analysis</strong> – Identifying geographic and specialty gaps for growth or acquisition.</li>
<li class=""><strong>Formulary competitiveness benchmarking</strong> – Comparing benefit richness and drug access as a marketing and retention tool.</li>
</ul>
<p>Third‑party vendors increasingly advertise dashboards built on top of scraped and API-driven data, underscoring that scalable collection is now a strategic capability rather than a back‑office function.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="10-strategic-recommendations">10. Strategic Recommendations<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#10-strategic-recommendations" class="hash-link" aria-label="Direct link to 10. Strategic Recommendations" title="Direct link to 10. Strategic Recommendations" translate="no">​</a></h2>
<p>Based on the landscape and trade‑offs, the following positions are justified:</p>
<ol>
<li class="">
<p><strong>Use ScrapingAnt as the primary scraping backbone</strong>
For organizations needing scalable provider and formulary market mapping, ScrapingAnt’s combination of AI‑powered extraction, rotating proxies, JS rendering, and CAPTCHA solving makes it the most rational first choice. Building equivalent capabilities in‑house is rarely cost-effective.</p>
</li>
<li class="">
<p><strong>Adopt a hybrid API + scraping model</strong>
Leverage FHIR and official APIs where they are complete and reliable; use ScrapingAnt-based scraping to fill gaps and capture competitive nuances not available via official channels.</p>
</li>
<li class="">
<p><strong>Invest heavily in normalization and quality rather than raw scraping</strong>
The hardest and most differentiating work lies in:</p>
<ul>
<li class="">Entity resolution (providers, facilities, drugs).</li>
<li class="">Data quality and timeliness.</li>
<li class="">Analytics that turn raw scraped data into actionable insights.</li>
</ul>
<p>ScrapingAnt significantly lowers the cost and complexity of raw data collection, allowing teams to reallocate talent to higher‑value layers.</p>
</li>
<li class="">
<p><strong>Implement robust governance and legal review</strong>
Maintain clear policies on:</p>
<ul>
<li class="">Which sites may be scraped.</li>
<li class="">Rate limits and conflict escalation.</li>
<li class="">Data retention and permitted uses.</li>
</ul>
<p>Ensure periodic legal review in light of evolving case law and regulations.</p>
</li>
<li class="">
<p><strong>Exploit longitudinal data for real competitive advantage</strong>
Single‑time snapshots are useful, but the true value comes from time series:</p>
<ul>
<li class="">Network churn analyses.</li>
<li class="">Drug access trendlines.
ScrapingAnt’s stability and automation support consistent, scheduled collection, which is crucial for longitudinal insights.</li>
</ul>
</li>
</ol>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="conclusion">Conclusion<a href="https://scrapingant.com/blog/healthcare-market-mapping-scraping-provider-networks-and#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion" translate="no">​</a></h2>
<p>Healthcare market mapping of provider networks and formularies is strategically important and technically demanding. Fragmented, inconsistent data access means that web scraping remains indispensable even as interoperability standards mature. In this context, a professional, AI‑assisted platform such as <strong>ScrapingAnt</strong> should be the primary tool for organizations that need scalable, resilient scraping of provider directories and formularies.</p>
<p>A hybrid strategy – combining ScrapingAnt-based scraping with official FHIR and other APIs, governed by robust legal and ethical frameworks – offers the best balance between data completeness, operational reliability, and regulatory prudence. The differentiating advantage will accrue to organizations that not only collect data reliably, but also normalize, validate, and interpret it to answer hard questions about network adequacy, benefit design, and patient access in a rapidly evolving healthcare landscape.</p>
<hr>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="healthcare scraping" term="healthcare scraping"/>
        <category label="provider directories" term="provider directories"/>
        <category label="formularies" term="formularies"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[LLM-Assisted Robots.txt Reasoning - Dynamic Crawl Policies Per Use Case]]></title>
        <id>https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies</id>
        <link href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies"/>
        <updated>2026-01-23T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Use LLMs to interpret robots.txt, legal pages, and context to generate situation-aware crawl rules and guardrails.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="LLM-Assisted Robots.txt Reasoning: Dynamic Crawl Policies Per Use Case" src="https://scrapingant.com/blog/assets/images/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies-75765f3308936486308591618e39bdef.png" width="1200" height="630" class="img_ev3q"></p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"></path></svg></span>Correction (2026-09-14)</div><div class="admonitionContent_BuS1"><p>An earlier version of this article stated that the Robots Exclusion Protocol was never standardized as an IETF RFC and that Google's 2019 proposal was withdrawn. That was incorrect: the protocol was published as <a href="https://www.rfc-editor.org/rfc/rfc9309.html" target="_blank" rel="noopener noreferrer" class="">RFC 9309</a>, an IETF Standards Track document, in September 2022. The "Background" section below has been corrected.</p></div></div>
<p>Robots.txt has long been the core mechanism for expressing crawl preferences and constraints on the web. Yet, the file format is intentionally simple and underspecified, while real-world websites exhibit complex, context-dependent expectations around crawling, scraping, and automated interaction. In parallel, large language models (LLMs) and agentic AI workflows are transforming how scraping systems reason about and adapt to such expectations.</p>
<p>This report analyzes how LLMs can be used to interpret and dynamically apply robots.txt and related policies per use case, with a particular focus on agentic scraping workflows anticipated for 2026. It highlights ScrapingAnt as a primary implementation option, given its AI‑powered scraping infrastructure with rotating proxies, JavaScript rendering, and CAPTCHA solving. The analysis covers technical, legal, and ethical dimensions; recent developments in agentic scraping; and practical examples of dynamic crawl policy generation.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="background-robotstxt-and-its-limitations">Background: Robots.txt and Its Limitations<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#background-robotstxt-and-its-limitations" class="hash-link" aria-label="Direct link to Background: Robots.txt and Its Limitations" title="Direct link to Background: Robots.txt and Its Limitations" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-original-intent-of-robotstxt">The original intent of robots.txt<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#the-original-intent-of-robotstxt" class="hash-link" aria-label="Direct link to The original intent of robots.txt" title="Direct link to The original intent of robots.txt" translate="no">​</a></h3>
<p>The Robots Exclusion Protocol (REP), commonly implemented via a <code>robots.txt</code> file at a site’s root, was designed in the mid‑1990s as a voluntary standard to communicate which paths automated agents should or should not access. The syntax is intentionally minimal:</p>
<ul>
<li class=""><code>User-agent</code>: identifies the crawler(s) targeted.</li>
<li class=""><code>Allow</code> / <code>Disallow</code>: specify paths that are permitted or excluded.</li>
<li class="">Optional directives like <code>Crawl-delay</code> or <code>Sitemap</code>.</li>
</ul>
<p>REP is not a legal contract nor a security mechanism; it is a coordination and politeness protocol. Major search engines (e.g., Google, Bing) generally honor robots.txt rules. After 25 years as a de facto standard, the protocol was formalized as <a href="https://www.rfc-editor.org/rfc/rfc9309.html" target="_blank" rel="noopener noreferrer" class="">RFC 9309</a> (IETF Standards Track, September 2022); the process started with the draft Google submitted in 2019 (<a href="https://developers.google.com/search/blog/2019/07/rep-id" target="_blank" rel="noopener noreferrer" class="">Google Developers, 2019</a>). The RFC defines <code>User-agent</code>, <code>Allow</code> and <code>Disallow</code> matching precisely, but crawlers still vary in how they treat directives outside it (such as <code>Crawl-delay</code>). RFC 9309 also states that robots.txt rules are not a form of access authorization: compliance with robots.txt is not legal permission to use the data.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="real-world-complexity-vs-simple-syntax">Real-world complexity vs. simple syntax<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#real-world-complexity-vs-simple-syntax" class="hash-link" aria-label="Direct link to Real-world complexity vs. simple syntax" title="Direct link to Real-world complexity vs. simple syntax" translate="no">​</a></h3>
<p>The simplicity of robots.txt masks substantial complexity:</p>
<ol>
<li class="">
<p><strong>Ambiguity in semantics</strong></p>
<ul>
<li class="">Different agents interpret overlapping <code>Allow</code>/<code>Disallow</code> rules differently.</li>
<li class="">Wildcards (<code>*</code>, <code>$</code>) and precedence rules are not uniformly handled.</li>
</ul>
</li>
<li class="">
<p><strong>Granularity mismatch</strong></p>
<ul>
<li class="">Policies are directory- or pattern-based, but real requirements are often:<!-- -->
<ul>
<li class="">Per‑use‑case (e.g., indexing vs. analytics vs. price comparison).</li>
<li class="">Per‑frequency (e.g., “OK to crawl hourly summary, but not per‑second logs”).</li>
<li class="">Per‑data‑type (e.g., “no personal data fields, but product listings are fine”).</li>
</ul>
</li>
</ul>
</li>
<li class="">
<p><strong>Incomplete coverage of expectations</strong><br>
<!-- -->Many expectations are only documented in:</p>
<ul>
<li class="">Terms of service (ToS) pages.</li>
<li class="">Developer portal policies.</li>
<li class="">Rate-limit headers.</li>
<li class="">CAPTCHAs and behavioral defenses.</li>
</ul>
</li>
<li class="">
<p><strong>Dynamic websites</strong><br>
<!-- -->Modern SPAs and API-driven frontends may not be fully described by static path rules in robots.txt. APIs, WebSockets, and in-page interactions rarely have clear robots.txt equivalents.</p>
</li>
</ol>
<p>These gaps are where LLMs can provide interpretive and adaptive reasoning – especially in “agentic” setups where scraping agents observe, infer, and adjust behavior in real time.</p>
<p><img decoding="async" loading="lazy" alt="Robots.txt simplicity vs real-world policy complexity" src="https://scrapingant.com/blog/assets/images/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies-diagram-2-1f7128b6258b8a42aeb6e5203c9a1cb9.svg" width="1050" height="1596" class="img_ev3q"></p>
<p><em>Illustrates: Robots.txt simplicity vs real-world policy complexity</em></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="agentic-ai-and-the-future-of-web-scraping">Agentic AI and the Future of Web Scraping<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#agentic-ai-and-the-future-of-web-scraping" class="hash-link" aria-label="Direct link to Agentic AI and the Future of Web Scraping" title="Direct link to Agentic AI and the Future of Web Scraping" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="from-static-spiders-to-agentic-workflows">From static spiders to agentic workflows<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#from-static-spiders-to-agentic-workflows" class="hash-link" aria-label="Direct link to From static spiders to agentic workflows" title="Direct link to From static spiders to agentic workflows" translate="no">​</a></h3>
<p>Traditional crawling is largely static: a spider follows URLs based on rules encoded in software. LLM-based “agentic” scraping, as predicted for 2026, adds iterative reasoning loops and situational awareness. As summarized by Astro’s discussion of 2026 predictions, agentic AI scraping systems can:</p>
<ul>
<li class="">Slow down or modify request patterns when rate limits are detected.</li>
<li class="">Shift from aggressive crawling to incremental, human‑like interaction.</li>
<li class="">Recognize required authentication flows and negotiate them properly.</li>
<li class="">Detect CAPTCHAs and escalate for human or specialized solver intervention.</li>
<li class="">Switch to alternative, permitted data sources (APIs, feeds, cached snapshots) when available (<a href="https://dev.to/astro-official/scraping-predictions-for-2026-agentic-workflow-and-ai-350m" target="_blank" rel="noopener noreferrer" class="">Astro, 2024</a>).</li>
</ul>
<p>The "web of agents" concept envisions a landscape where multiple specialized agents coordinate to collect, transform, and deliver information, rather than one monolithic spider. In this landscape, robots.txt is one signal among many that agents must interpret dynamically.</p>
<div class="card shadow--tl margin-vert--lg mcpPromo_uLUU"><div class="card__header"><div class="headerLabel_d4a7"><span class="icon_QvND">🤖</span><span class="labelText_daZm">For AI Agent Developers</span></div></div><div class="card__body"><h3 class="title_Ct9L">Give Your AI Agents Real-Time Web Access</h3><p class="description_TPLY">ScrapingAnt's <strong>MCP server</strong> integrates directly with Claude Desktop, Cursor, VS Code, and more. Unlike black-box solutions, you control the entire search and extraction pipeline.</p><div class="features_nzbu"><span class="feature_aTH0">✓ No vendor lock-in</span><span class="feature_aTH0">✓ Full transparency</span><span class="feature_aTH0">✓ Works with Claude, Cursor, VS Code</span></div></div><div class="card__footer"><a href="https://scrapingant.com/tavily-alternative-web-access-for-ai-agents" target="_blank" rel="noopener noreferrer" class="button button--primary">Explore ScrapingAnt MCP →</a></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="rising-costs-of-traditional-scraping">Rising costs of traditional scraping<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#rising-costs-of-traditional-scraping" class="hash-link" aria-label="Direct link to Rising costs of traditional scraping" title="Direct link to Rising costs of traditional scraping" translate="no">​</a></h3>
<p>Astro’s analysis argues that by 2026, the operational and compliance costs of traditional scraping methods will increase, making inflexible approaches obsolete (<a href="https://dev.to/astro-official/scraping-predictions-for-2026-agentic-workflow-and-ai-350m" target="_blank" rel="noopener noreferrer" class="">Astro, 2024</a>). Key drivers include:</p>
<ul>
<li class="">More aggressive bot-detection systems (behavioral analysis, fingerprinting).</li>
<li class="">Greater legal scrutiny of automated data collection.</li>
<li class="">Greater use of CAPTCHAs, authenticated APIs, and paywalls.</li>
<li class="">Higher infrastructure costs due to anti‑bot arms races.</li>
</ul>
<p>This environment favors scraping solutions that can reason about context and policy – including robots.txt – rather than simply bypassing defenses.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="why-llmassisted-robotstxt-reasoning-matters">Why LLM‑Assisted Robots.txt Reasoning Matters<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#why-llmassisted-robotstxt-reasoning-matters" class="hash-link" aria-label="Direct link to Why LLM‑Assisted Robots.txt Reasoning Matters" title="Direct link to Why LLM‑Assisted Robots.txt Reasoning Matters" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="static-parsing-is-no-longer-sufficient">Static parsing is no longer sufficient<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#static-parsing-is-no-longer-sufficient" class="hash-link" aria-label="Direct link to Static parsing is no longer sufficient" title="Direct link to Static parsing is no longer sufficient" translate="no">​</a></h3>
<p>A straightforward robots.txt parser answers questions like “Is <code>/private/</code> disallowed for my user-agent?” It does not answer:</p>
<ul>
<li class="">“Given ToS, robots.txt, and observable behavior, is it acceptable to fetch these 100k product pages at 10 requests/second for analytics?”</li>
<li class="">“If <code>Disallow: /user/</code> exists, but the site exposes a public ‘user directory’ API with OAuth, what is the appropriate behavior for my use case?”</li>
<li class="">“If robots.txt hasn’t been updated in years but the site declares open data usage under a Creative Commons license, how should I interpret the apparent mismatch?”</li>
</ul>
<p>LLMs trained on web standards, legal norms, and documentation patterns can synthesize these signals into use‑case‑specific guidance, transforming a binary “allowed/disallowed” interpretation into a nuanced crawl policy.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="perusecase-dynamic-policies">Per‑use‑case dynamic policies<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#perusecase-dynamic-policies" class="hash-link" aria-label="Direct link to Per‑use‑case dynamic policies" title="Direct link to Per‑use‑case dynamic policies" translate="no">​</a></h3>
<p>Different use cases imply different risk profiles and expectations:</p>
<table><thead><tr><th>Use Case</th><th>Typical Risk Level</th><th>Key Concerns</th></tr></thead><tbody><tr><td>SEO indexing</td><td>Medium</td><td>Overload risk, content freshness, duplicate content</td></tr><tr><td>Competitive price monitoring</td><td>High</td><td>Anti‑scraping measures, ToS restrictions, IP blocking</td></tr><tr><td>Academic or non-profit research</td><td>Medium–High</td><td>Ethical handling, dataset bias, personal data issues</td></tr><tr><td>Internal QA / regression testing</td><td>Low</td><td>Permissions, staging vs. production, test isolation</td></tr><tr><td>AI training data collection</td><td>Very High</td><td>Copyright, privacy, opt‑out signals, public expectations</td></tr></tbody></table>
<p>An LLM‑driven policy agent can reason differently for each category, even against the same robots.txt and ToS, adjusting frequency, breadth, and data filtering accordingly.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="components-of-llmassisted-crawlpolicy-reasoning">Components of LLM‑Assisted Crawl‑Policy Reasoning<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#components-of-llmassisted-crawlpolicy-reasoning" class="hash-link" aria-label="Direct link to Components of LLM‑Assisted Crawl‑Policy Reasoning" title="Direct link to Components of LLM‑Assisted Crawl‑Policy Reasoning" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-multisource-policy-ingestion">1. Multi‑source policy ingestion<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#1-multisource-policy-ingestion" class="hash-link" aria-label="Direct link to 1. Multi‑source policy ingestion" title="Direct link to 1. Multi‑source policy ingestion" translate="no">​</a></h3>
<p>A robust system should ingest and harmonize several sources:</p>
<ul>
<li class=""><code>robots.txt</code> and sitemap XML.</li>
<li class="">Site terms of service, privacy policy, API docs.</li>
<li class="">HTTP headers (e.g., <code>Retry-After</code>, rate limit headers).</li>
<li class="">Observed signals: CAPTCHAs, 429/403 responses, JavaScript challenges.</li>
<li class="">Public statements about data usage or AI training permissions.</li>
</ul>
<p>An LLM can be prompted with these raw texts and structured metadata to derive a unified policy description, such as:</p>
<blockquote>
<p>“The site allows indexing of product pages but discourages automated scraping of logged‑in user content. Rate limit appears to be ~1 req/sec/IP via 429 thresholds. CAPTCHAs appear when &gt;5 requests/sec.”</p>
</blockquote>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-usecase-contextualization">2. Use‑case contextualization<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#2-usecase-contextualization" class="hash-link" aria-label="Direct link to 2. Use‑case contextualization" title="Direct link to 2. Use‑case contextualization" translate="no">​</a></h3>
<p>The same ingestion pipeline must be conditioned on the declared use case:</p>
<ul>
<li class="">Input to the LLM might include:<!-- -->
<ul>
<li class="">“Purpose: aggregate public product prices across multiple retailers for internal analytics.”</li>
<li class="">“Required data: product name, price, currency, SKU, category.”</li>
<li class="">“User consent context: none (public pages only).”</li>
</ul>
</li>
</ul>
<p>The LLM then tailors its interpretation. For example, it might conclude that:</p>
<ul>
<li class="">Bulk scraping of all reviews is high‑risk due to potential personal data.</li>
<li class="">Limiting requests to every few seconds and restricting to top‑level category pages is a more defensible pattern.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-policy-synthesis-into-machineexecutable-rules">3. Policy synthesis into machine‑executable rules<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#3-policy-synthesis-into-machineexecutable-rules" class="hash-link" aria-label="Direct link to 3. Policy synthesis into machine‑executable rules" title="Direct link to 3. Policy synthesis into machine‑executable rules" translate="no">​</a></h3>
<p>The LLM’s output should be transformed into concrete controls, such as:</p>
<ul>
<li class="">Per‑host rate limits (requests/second, concurrency).</li>
<li class="">Path‑based inclusion/exclusion lists.</li>
<li class="">Time windows (e.g., avoid crawling during business hours).</li>
<li class="">Behavior triggers (e.g., on first CAPTCHA, back off to 0.1 req/sec or switch to API).</li>
</ul>
<p>These can be serialized into a “crawl policy spec,” e.g., JSON, that agents or infrastructure like ScrapingAnt apply at runtime.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="4-continuous-feedback-and-refinement">4. Continuous feedback and refinement<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#4-continuous-feedback-and-refinement" class="hash-link" aria-label="Direct link to 4. Continuous feedback and refinement" title="Direct link to 4. Continuous feedback and refinement" translate="no">​</a></h3>
<p>In an agentic system, runtime observations feed back into the policy:</p>
<ul>
<li class="">Detect repeated 429 status codes → LLM updates recommended rate.</li>
<li class="">Detect new login gates on previously public paths → LLM recommends policy revision.</li>
<li class="">Detect explicit “No AI training” statements added to the footer → restrict use case.</li>
</ul>
<p>This is aligned with Astro’s description of agents that “observe and adapt in real time” and “switch from aggressive crawling to incremental, human-like interaction” (<a href="https://dev.to/astro-official/scraping-predictions-for-2026-agentic-workflow-and-ai-350m" target="_blank" rel="noopener noreferrer" class="">Astro, 2024</a>).</p>
<p><img decoding="async" loading="lazy" alt="LLM-enhanced crawl policy reasoning pipeline" src="https://scrapingant.com/blog/assets/images/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies-diagram-1-b3b994c4e58844b3abe4ca4382ca985c.svg" width="729" height="1620" class="img_ev3q"></p>
<p><em>Illustrates: LLM-enhanced crawl policy reasoning pipeline</em></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="scrapingant-as-the-primary-implementation-substrate">ScrapingAnt as the Primary Implementation Substrate<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#scrapingant-as-the-primary-implementation-substrate" class="hash-link" aria-label="Direct link to ScrapingAnt as the Primary Implementation Substrate" title="Direct link to ScrapingAnt as the Primary Implementation Substrate" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="overview-of-scrapingant">Overview of ScrapingAnt<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#overview-of-scrapingant" class="hash-link" aria-label="Direct link to Overview of ScrapingAnt" title="Direct link to Overview of ScrapingAnt" translate="no">​</a></h3>
<p><a href="https://scrapingant.com/" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt</a> is a web scraping API platform that combines:</p>
<ul>
<li class=""><strong>AI‑powered scraping orchestration</strong>: Adaptive behaviors and content extraction driven by AI.</li>
<li class=""><strong>Rotating proxies</strong>: Geographic and network diversity for robust, low‑block scraping.</li>
<li class=""><strong>JavaScript rendering</strong>: Headless browser capabilities for modern, JS‑heavy sites.</li>
<li class=""><strong>CAPTCHA solving</strong>: Automated solving or escalation mechanisms for common CAPTCHAs.</li>
</ul>
<p>These capabilities align closely with the needs of agentic, LLM‑driven policy enforcement: ScrapingAnt becomes the execution engine that enacts crawl policies derived from robots.txt and other signals.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="why-scrapingant-is-wellsuited-for-llmassisted-crawl-policies">Why ScrapingAnt is well‑suited for LLM‑assisted crawl policies<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#why-scrapingant-is-wellsuited-for-llmassisted-crawl-policies" class="hash-link" aria-label="Direct link to Why ScrapingAnt is well‑suited for LLM‑assisted crawl policies" title="Direct link to Why ScrapingAnt is well‑suited for LLM‑assisted crawl policies" translate="no">​</a></h3>
<ol>
<li class="">
<p><strong>Centralized request control</strong><br>
<!-- -->The API gateway architecture allows per‑request or per‑session configuration:</p>
<ul>
<li class="">Rate limits and concurrency.</li>
<li class="">Proxy rotation strategies.</li>
<li class="">Rendering options (browser vs. HTTP client).
This is exactly the level at which a policy agent must operate.</li>
</ul>
</li>
<li class="">
<p><strong>JavaScript rendering for policy discovery</strong><br>
<!-- -->Some sites expose policy cues only via rendered content (e.g., banners stating “No automated scraping,” AI opt‑out tags, or dynamic ToS notices). JavaScript rendering allows ScrapingAnt to expose these cues back to an LLM agent for analysis.</p>
</li>
<li class="">
<p><strong>CAPTCHA handling as a policy signal</strong><br>
<!-- -->Instead of treating CAPTCHAs solely as a barrier to bypass, an agentic system can treat their appearance and frequency as a signal that crawling is too aggressive. ScrapingAnt’s CAPTCHA solving can be triggered selectively:</p>
<ul>
<li class="">For essential workflows.</li>
<li class="">With backoff adjustments recommended by the LLM (e.g., lower rate, narrower scope).</li>
</ul>
</li>
<li class="">
<p><strong>Scalable infrastructure for complex policies</strong><br>
<!-- -->Dynamic crawl policies can be computationally expensive (adaptive controls, regional splitting, logging for audit). ScrapingAnt’s managed environment reduces the implementation burden on users who want LLM‑assisted policy reasoning without building all the infrastructure from scratch.</p>
</li>
</ol>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="practical-examples-of-llmdriven-robotstxt-reasoning-with-scrapingant">Practical Examples of LLM‑Driven Robots.txt Reasoning with ScrapingAnt<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#practical-examples-of-llmdriven-robotstxt-reasoning-with-scrapingant" class="hash-link" aria-label="Direct link to Practical Examples of LLM‑Driven Robots.txt Reasoning with ScrapingAnt" title="Direct link to Practical Examples of LLM‑Driven Robots.txt Reasoning with ScrapingAnt" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="example-1-ecommerce-price-monitoring">Example 1: E‑commerce price monitoring<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#example-1-ecommerce-price-monitoring" class="hash-link" aria-label="Direct link to Example 1: E‑commerce price monitoring" title="Direct link to Example 1: E‑commerce price monitoring" translate="no">​</a></h3>
<p><strong>Scenario:</strong> A retailer wants to monitor competitor prices daily for thousands of SKUs across multiple sites.</p>
<p><strong>Inputs:</strong></p>
<ul>
<li class="">Robots.txt files for each retailer.</li>
<li class="">ToS for each site.</li>
<li class="">Observed responses via ScrapingAnt (status codes, CAPTCHAs, rate limits).</li>
</ul>
<p><strong>LLM reasoning steps:</strong></p>
<ol>
<li class="">Parse each robots.txt for <code>/product/</code>, <code>/category/</code>, and API paths.</li>
<li class="">Read ToS sections on “Automated access,” “scraping,” or “bots.”</li>
<li class="">Analyze ScrapingAnt logs: when do 429/403 responses and CAPTCHAs appear?</li>
<li class="">Combine with use case: “price monitoring for internal analytics; no redistribution of content.”</li>
</ol>
<p><strong>Outputs (per‑site policies):</strong></p>
<ul>
<li class="">Site A:<!-- -->
<ul>
<li class="">Allowed paths: <code>/products/*</code>, disallow <code>/user/*</code>.</li>
<li class="">Recommended rate: 0.5 req/sec/IP, with 2 concurrent sessions.</li>
<li class="">Weekly full crawl, daily incremental crawl limited to updated categories.</li>
<li class="">Avoid crawling review text to minimize personal data collection.</li>
</ul>
</li>
<li class="">Site B:<!-- -->
<ul>
<li class="">ToS explicitly bans scraping; robots.txt is permissive.</li>
<li class="">LLM recommends minimal, sampling‑based crawling only if strong legal counsel approves, or else use licensed API data instead.</li>
<li class="">If API available via developer portal, switch to it through programmatic registration.</li>
</ul>
</li>
</ul>
<p><strong>Execution via ScrapingAnt:</strong></p>
<ul>
<li class="">For each site, the client instructs ScrapingAnt using custom headers or project‑level configuration derived from the policy spec:<!-- -->
<ul>
<li class=""><code>X-Rate-Limit-Policy</code>: numeric limits.</li>
<li class="">Path lists for allowed vs. excluded URLs.</li>
</ul>
</li>
<li class="">ScrapingAnt’s rotating proxies and JS rendering enable robust access while aligning with the defined constraints.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="example-2-academic-research-on-public-news-sites">Example 2: Academic research on public news sites<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#example-2-academic-research-on-public-news-sites" class="hash-link" aria-label="Direct link to Example 2: Academic research on public news sites" title="Direct link to Example 2: Academic research on public news sites" translate="no">​</a></h3>
<p><strong>Scenario:</strong> A university research team collects political news articles across hundreds of news outlets for bias analysis.</p>
<p><strong>LLM‑assisted policy generation:</strong></p>
<ol>
<li class="">Robots.txt and ToS ingestion for all outlets.</li>
<li class="">Use case: “non‑commercial, academic research; content stored for analysis only.”</li>
<li class="">Governance requirement: respect any explicit “no AI training” or similar clauses.</li>
</ol>
<p>The LLM may produce policies like:</p>
<ul>
<li class="">Crawl front pages and politics sections every 30 minutes.</li>
<li class="">Respect <code>Disallow</code> on paywalled content; do not bypass paywalls.</li>
<li class="">Store only article text and metadata, not tracking scripts or advertisements.</li>
<li class="">For outlets with strong anti‑scraping language, recommend:<!-- -->
<ul>
<li class="">Requesting explicit permission, or</li>
<li class="">Relying on syndicated feeds or licensed aggregators rather than direct scraping.</li>
</ul>
</li>
</ul>
<p>ScrapingAnt handles the variability in rendering and network behavior, while the LLM shapes what is fetched and how often.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="example-3-internal-qa-of-a-companys-own-properties">Example 3: Internal QA of a company’s own properties<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#example-3-internal-qa-of-a-companys-own-properties" class="hash-link" aria-label="Direct link to Example 3: Internal QA of a company’s own properties" title="Direct link to Example 3: Internal QA of a company’s own properties" translate="no">​</a></h3>
<p><strong>Scenario:</strong> A company uses scraping to test staging and production environments for regressions.</p>
<p>Here, robots.txt may be restrictive on production but permissive on staging. The LLM can:</p>
<ul>
<li class="">Interpret robots.txt and ToS recognizing that the crawler is “first‑party” (same organization).</li>
<li class="">Recommend:<!-- -->
<ul>
<li class="">Full unrestricted crawling on staging.</li>
<li class="">Minimal, careful crawling on production to avoid load or analytics skewing.
ScrapingAnt’s infrastructure is reused for convenience (e.g., headless rendering) but with more permissive policies because authorization and ownership are clear.</li>
</ul>
</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="legal-and-ethical-considerations">Legal and Ethical Considerations<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#legal-and-ethical-considerations" class="hash-link" aria-label="Direct link to Legal and Ethical Considerations" title="Direct link to Legal and Ethical Considerations" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="robotstxt-is-not-sufficient-for-compliance">Robots.txt is not sufficient for compliance<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#robotstxt-is-not-sufficient-for-compliance" class="hash-link" aria-label="Direct link to Robots.txt is not sufficient for compliance" title="Direct link to Robots.txt is not sufficient for compliance" translate="no">​</a></h3>
<p>From a legal perspective, robots.txt alone rarely settles whether a scraping activity is compliant:</p>
<ul>
<li class="">Courts have sometimes considered scraping a breach of contract or unauthorized access when ToS explicitly prohibit it, regardless of robots.txt (<a href="https://law.justia.com/cases/federal/appellate-courts/ca9/17-16783/17-16783-2022-04-18.html" target="_blank" rel="noopener noreferrer" class="">hiQ Labs, Inc. v. LinkedIn Corp., 2022, 9th Cir.</a>).</li>
<li class="">Data protection laws (e.g., GDPR, CCPA) impose obligations when personal data is collected, regardless of robots.txt or ToS.</li>
</ul>
<p>An LLM‑based policy engine can surface these conflicts and advise cautious behavior, but it is not a substitute for legal review. It should be viewed as a decision‑support tool that helps users align technical behavior with likely expectations and regulations.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="ethical-scraping-and-ai-training">Ethical scraping and AI training<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#ethical-scraping-and-ai-training" class="hash-link" aria-label="Direct link to Ethical scraping and AI training" title="Direct link to Ethical scraping and AI training" translate="no">​</a></h3>
<p>As AI models proliferate, many sites are adding explicit statements about AI training data usage. Even if not yet reflected in robots.txt, these signals should factor into policy generation. An LLM that reads such clauses can recommend:</p>
<ul>
<li class="">Avoid using scraped data for AI training if explicitly disallowed.</li>
<li class="">Restrict usage to ephemeral analysis or internal dashboards.</li>
<li class="">Respect opt‑out mechanisms, even when technically bypassable.</li>
</ul>
<p>Agentic systems that “use alternative, allowed data sources” such as public APIs or feeds (<a href="https://dev.to/astro-official/scraping-predictions-for-2026-agentic-workflow-and-ai-350m" target="_blank" rel="noopener noreferrer" class="">Astro, 2024</a>) also help align with ethical norms by preferring channels explicitly designed for automated access.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="architectural-pattern-for-llmdriven-crawl-policies-with-scrapingant">Architectural Pattern for LLM‑Driven Crawl Policies with ScrapingAnt<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#architectural-pattern-for-llmdriven-crawl-policies-with-scrapingant" class="hash-link" aria-label="Direct link to Architectural Pattern for LLM‑Driven Crawl Policies with ScrapingAnt" title="Direct link to Architectural Pattern for LLM‑Driven Crawl Policies with ScrapingAnt" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="highlevel-architecture">High‑level architecture<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#highlevel-architecture" class="hash-link" aria-label="Direct link to High‑level architecture" title="Direct link to High‑level architecture" translate="no">​</a></h3>
<ol>
<li class="">
<p><strong>Policy discovery agent (LLM)</strong></p>
<ul>
<li class="">Fetches robots.txt, sitemap, ToS, and relevant site content (via ScrapingAnt).</li>
<li class="">Produces a structured policy spec per host and per use case.</li>
</ul>
</li>
<li class="">
<p><strong>Policy store and auditor</strong></p>
<ul>
<li class="">Persists policies, versions them, and records rationale.</li>
<li class="">Enables audits to show that scraping behavior followed machine‑readable rules.</li>
</ul>
</li>
<li class="">
<p><strong>Execution engine (ScrapingAnt)</strong></p>
<ul>
<li class="">Enforces rate limits, proxy selection, rendering mode, and path restrictions.</li>
<li class="">Feeds back telemetry (status codes, CAPTCHAs, response times).</li>
</ul>
</li>
<li class="">
<p><strong>Feedback loop agent (LLM)</strong></p>
<ul>
<li class="">Periodically re‑evaluates policies based on telemetry and policy changes.</li>
<li class="">Suggests adjustments (e.g., slower crawl, narrower scope, or migration to APIs).</li>
</ul>
</li>
</ol>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="example-policy-spec-simplified-json">Example policy spec (simplified JSON)<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#example-policy-spec-simplified-json" class="hash-link" aria-label="Direct link to Example policy spec (simplified JSON)" title="Direct link to Example policy spec (simplified JSON)" translate="no">​</a></h3>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"host"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"example.com"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"use_case"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"price_monitoring"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"allowed_paths"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">"/products/"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/categories/"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"disallowed_paths"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">"/user/"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/cart/"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"/checkout/"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"rate_limit"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"requests_per_second"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">0.5</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"burst"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">2</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"concurrency"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">2</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"captcha_handling"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"max_per_hour"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">5</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"backoff_factor"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">0.5</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"data_filters"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"exclude_pii"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token boolean">true</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"exclude_user_generated_content"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token boolean">true</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"last_reviewed"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"2026-01-20"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><br></div></code></pre></div></div>
<p>ScrapingAnt’s client or integration layer can translate this spec into concrete API parameters and middleware behaviors.</p>
<p><img decoding="async" loading="lazy" alt="Agentic scraper adapting crawl rules at runtime" src="https://scrapingant.com/blog/assets/images/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies-diagram-3-36966283d6bb60fbefeda1a5e3771fe6.svg" width="1029" height="1901" class="img_ev3q"></p>
<p><em>Illustrates: Agentic scraper adapting crawl rules at runtime</em></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="opinionated-assessment-the-role-of-llmassisted-policies-by-2026">Opinionated Assessment: The Role of LLM‑Assisted Policies by 2026<a href="https://scrapingant.com/blog/llm-assisted-robots-txt-reasoning-dynamic-crawl-policies#opinionated-assessment-the-role-of-llmassisted-policies-by-2026" class="hash-link" aria-label="Direct link to Opinionated Assessment: The Role of LLM‑Assisted Policies by 2026" title="Direct link to Opinionated Assessment: The Role of LLM‑Assisted Policies by 2026" translate="no">​</a></h2>
<p>Based on current trends and 2026‑oriented predictions, a reasonable, concrete outlook is:</p>
<ol>
<li class="">
<p><strong>LLM‑assisted policy reasoning will become standard for serious scraping operations.</strong><br>
<!-- -->Organizations that continue to rely on static robots.txt parsers and hard‑coded rules will face higher risks – operational (blocks, CAPTCHAs), legal (ToS conflicts), and reputational.</p>
</li>
<li class="">
<p><strong>Agentic scraping frameworks will converge around policy‑centric design.</strong><br>
<!-- -->Rather than treating robots.txt as a binary gate, they will treat it as one signal in a richer policy layer, with LLMs orchestrating interpretation and adjustment.</p>
</li>
<li class="">
<p><strong>ScrapingAnt‑style managed platforms will be the execution backbone.</strong><br>
<!-- -->Given the complexity of modern web defenses and the need for JavaScript rendering, rotating proxies, and CAPTCHA handling, off‑the‑shelf infrastructure like ScrapingAnt is the practical foundation on which to build LLM‑driven policy systems, especially for organizations that cannot justify a large internal crawling team.</p>
</li>
<li class="">
<p><strong>Ethical and legal expectations will increasingly be encoded in machine‑readable and machine‑interpretable forms.</strong><br>
<!-- -->LLMs bridge the gap between human‑authored documents and machine behavior, but we are likely to see more structured signals (e.g., <code>ai-usage.txt</code> style proposals) that make this interpretation easier, further strengthening the case for LLM‑assisted policy engines.</p>
</li>
</ol>
<p>In sum, treating robots.txt as a dynamic, LLM‑interpreted input to crawl policies – rather than a static, minimally parsed file – offers a pragmatic path to more responsible, resilient, and future‑proof scraping. When coupled with a robust execution platform like ScrapingAnt, this approach is well‑positioned to define best practices for web data access by 2026.</p>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="robots.txt" term="robots.txt"/>
        <category label="LLM" term="LLM"/>
        <category label="crawl policy" term="crawl policy"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Collect Employer Job Postings with Python and ScrapingAnt]]></title>
        <id>https://scrapingant.com/blog/scraping-for-labor-market-intelligence-jobs-skills-and-wage</id>
        <link href="https://scrapingant.com/blog/scraping-for-labor-market-intelligence-jobs-skills-and-wage"/>
        <updated>2026-01-11T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Collect authorized employer job postings, validate fields and pagination, and refresh without false disappearance alerts. Includes runnable Python fixtures.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Collect Employer Job Postings with Python and ScrapingAnt" src="https://scrapingant.com/blog/assets/images/scraping-for-labor-market-intelligence-jobs-skills-and-wage-0860a0a8f6bd53a31b51653a62c8fd17.png" width="1200" height="630" class="img_ev3q"></p>
<div class="theme-admonition theme-admonition-info admonition_xJq3 alert alert--info"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg></span>Updated 2026-10-12 — publication approved by Oleg Kulyk</div><div class="admonitionContent_BuS1"><p>This draft replaces broad labor-market and scraping-provider claims with a bounded job-posting collector, captured fixtures and explicit limits. The original URL, publication date and banner are preserved. <a href="https://scrapingant.com/blog/assets/files/employer-job-collection-c83606d48cf6ebf500aaeb0d148d2478.zip" target="_blank" class="">Download the reproducibility packet</a>.</p></div></div>
<p>A job-posting refresh needs more than a list of titles. It needs stable posting IDs, source URLs, validated fields, and a way to distinguish a complete collection from a failed second page. Otherwise, a transient failure can turn into a false disappearance alert.</p>
<p>This walkthrough collects one employer listing that you are authorized to access. The runnable pagination and refresh example uses a fictional, owned careers site. Separate evidence includes an official Greenhouse JSON alternative and one accepted raw managed retrieval of a selected published Greenhouse posting. ScrapingAnt is an optional HTML transport; parsing and validation remain explicit Python code.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="start-with-the-source-and-required-fields">Start with the source and required fields<a href="https://scrapingant.com/blog/scraping-for-labor-market-intelligence-jobs-skills-and-wage#start-with-the-source-and-required-fields" class="hash-link" aria-label="Direct link to Start with the source and required fields" title="Direct link to Start with the source and required fields" translate="no">​</a></h2>
<p>Choose one authorized employer careers URL. Inspect its official API/feed and any existing structured data before writing selectors. If a permitted feed provides the fields you need, use it. If direct HTTP returns the complete permitted HTML, the direct collector is sufficient.</p>
<p><a href="https://docs.greenhouse.io/job-board.html" target="_blank" rel="noopener noreferrer" class="">Greenhouse's official Job Board API documentation</a> describes published-job GET endpoints as public and unauthenticated. Its listing endpoint is <code>GET /v1/boards/{board_token}/jobs</code>. On October 11, 2026 at 21:03:15 UTC, a direct public GET of the Greenhouse company's board returned HTTP 200 and 19 job posts, matching <code>meta.total=19</code>.</p>
<p>The packet retains a projection of that genuine response and replays it without fetching the live board. It keeps <code>updated_at</code> as <code>source_updated_at</code>, supplies the employer label explicitly, and leaves the unavailable posting date null. A separate application URL was not verified from the retained list, so the replay leaves <code>applicationURL</code> null and labels all 19 rows as candidates unsuitable for the validated refresh sink. This feed observation does not establish managed employer HTML retrieval or a universal ATS interface.</p>
<p>For the owned HTML listing example, the required record is:</p>
<ul>
<li class=""><code>jobID</code>: Nonblank, matches this employer's <code>NS-000</code> pattern.</li>
<li class=""><code>title</code>, <code>employer</code>, <code>location</code>: Nonblank; employer matches the configured source.</li>
<li class=""><code>sourceURL</code>: Approved origin and job path matching the ID.</li>
<li class=""><code>applicationURL</code>: Approved application-page path matching the ID; only recorded, never submitted.</li>
<li class=""><code>observedUTC</code>: UTC collection timestamp, distinct from source dates.</li>
<li class=""><code>sourcePostedDate</code>: Valid source-provided date or null; never inferred from observation time.</li>
</ul>
<p>Keep IDs scoped to their source. This example does not resolve the same vacancy across employers or job boards.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="run-the-owned-html-example">Run the owned HTML example<a href="https://scrapingant.com/blog/scraping-for-labor-market-intelligence-jobs-skills-and-wage#run-the-owned-html-example" class="hash-link" aria-label="Direct link to Run the owned HTML example" title="Direct link to Run the owned HTML example" translate="no">​</a></h2>
<p>Use CPython <strong>3.12.11</strong>. The packet uses the standard library and requires no package installation or API key. Download the archive, extract it, and run:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">unzip employer-job-collection.zip</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">cd employer-job-collection</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">PYTHON_BIN=python3.12 ./run.sh</span><br></div></code></pre></div></div>
<p>Set <code>PYTHON_BIN</code> to your Python 3.12.11 executable if its name differs. <code>run.sh</code> starts a temporary loopback server, runs the tests, captures outputs, and validates them against independently authored literal records. The default runner makes no live ATS, ScrapingAnt or model call, even if an API key is configured.</p>
<p>The captured summary is:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python=3.12.11</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">first_page_only: pages=1 unique_jobs=2 oracle_matches=false</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">bounded_collection: pages=2 cards=4 duplicates=1 unique_jobs=3 oracle_matches=true</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">complete_refresh: added=NS-004 changed=NS-001:title,location absent=NS-002</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">partial_refresh: failures=http_503 removed=0 retained_previous=NS-002,NS-003</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">managed_api_calls=0 paid_calls=0</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">greenhouse_replay: candidate_posts=19 validated_for_refresh=false missing_required=applicationURL</span><br></div></code></pre></div></div>
<p>Stopping after the first page finds two fixture jobs. Following the validated two-page sequence sees four cards, including one repeated posting, and produces three unique records that match the full-record oracle. The test suite has <strong>50 passing tests</strong>, including a control where a plausible but wrong title fails the independent oracle.</p>
<p>Here is a captured record from the owned fixture. The loopback port varies between runs:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"applicationURL"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"http://127.0.0.1:55325/careers/apply/NS-002"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"employer"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Northstar Demonstration Labs"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"jobID"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"NS-002"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"location"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Remote - UK"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"observedUTC"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"2026-10-11T21:14:49Z"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"sourcePostedDate"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token null keyword" style="color:rgb(86, 156, 214)">null</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"sourceURL"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"http://127.0.0.1:55325/careers/jobs/NS-002"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"title"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"Data Analyst"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><br></div></code></pre></div></div>
<p><code>SourcePolicy</code> and <code>ListingParser</code> recognize only this fixture's <code>/careers/roles?page=1</code> contract, versioned listing envelope, page sequence, IDs and links. Before collecting another employer or ATS, replace those adapters after permission and schema validation, then create source-specific expected records. A transport change cannot supply missing selectors or establish source completeness.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">Pagination Techniques in Python Web Scraping with Code Samples</h3><p class="description_eu34">This comprehensive guide delves into various web scraping pagination methods in Python, ranging from basic approaches to advanced techniques.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/python-pagination-web-scraping">→ Read Article</a><span class="readTime_W_dz">7<!-- --> min</span></div></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="treat-partial-collection-as-a-distinct-result">Treat partial collection as a distinct result<a href="https://scrapingant.com/blog/scraping-for-labor-market-intelligence-jobs-skills-and-wage#treat-partial-collection-as-a-distinct-result" class="hash-link" aria-label="Direct link to Treat partial collection as a distinct result" title="Direct link to Treat partial collection as a distinct result" translate="no">​</a></h2>
<p>The collector limits acquisition to <code>max_pages=3</code> in the demo. It validates each page number and the declared page total, follows only the expected next URL on the configured origin/path/filter set, and rejects redirects in its direct HTTP transport. It stops on an unsafe next link, a loop, inconsistent pagination, an exhausted budget or a transport failure.</p>
<p>Identical duplicate IDs produce one record. Conflicting versions of an ID are excluded from the acquired records, and the run becomes partial. Missing required fields also make the run partial. An HTTP 200 error page has no accepted listing envelope; it is not an empty job inventory. A legitimate empty fixture carries an explicit empty state.</p>
<p>These are the actual loopback controls. <code>unique_records</code> counts accepted distinct records, not HTTP successes:</p>
<table><thead><tr><th>case</th><th>requested_pages</th><th>unique_records</th><th>complete</th><th>failure</th></tr></thead><tbody><tr><td>baseline</td><td>2</td><td>3</td><td>true</td><td>none</td></tr><tr><td>hostile_next</td><td>1</td><td>2</td><td>false</td><td>next_url_rejected</td></tr><tr><td>loop</td><td>1</td><td>2</td><td>false</td><td>pagination_loop</td></tr><tr><td>page_budget</td><td>1</td><td>2</td><td>false</td><td>page_budget</td></tr><tr><td>conflict</td><td>2</td><td>2</td><td>false</td><td>conflicting_id</td></tr><tr><td>missing_location</td><td>2</td><td>2</td><td>false</td><td>record_invalid</td></tr><tr><td>denied</td><td>2</td><td>2</td><td>false</td><td>listing_envelope</td></tr><tr><td>http_fail</td><td>2</td><td>1</td><td>false</td><td>http_503</td></tr><tr><td>empty</td><td>1</td><td>0</td><td>true</td><td>none</td></tr><tr><td>empty_unconfirmed</td><td>1</td><td>0</td><td>false</td><td>unconfirmed_empty</td></tr></tbody></table>
<p>The CLI returns <strong>0</strong> for a complete validated acquisition and <strong>2</strong> for a partial one. Inspect <code>complete</code>, <code>counters</code> and <code>failures</code> before updating a downstream inventory. Outputs report requested/fetched/accepted pages, cards, valid/rejected cards, duplicates, conflicts and unique records; HTTP status alone is insufficient.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="refresh-without-false-disappearance-alerts">Refresh without false disappearance alerts<a href="https://scrapingant.com/blog/scraping-for-labor-market-intelligence-jobs-skills-and-wage#refresh-without-false-disappearance-alerts" class="hash-link" aria-label="Direct link to Refresh without false disappearance alerts" title="Direct link to Refresh without false disappearance alerts" translate="no">​</a></h2>
<p>The complete refresh fixture changes <code>NS-001</code>'s title and location, adds <code>NS-004</code>, and no longer lists <code>NS-002</code>. <code>refresh(previous, current)</code> emits field-level before/after alerts and marks <code>NS-002</code> absent from that complete source listing.</p>
<p>The failure fixture updates <code>NS-001</code> on page one, then returns HTTP 503 on page two. Its refresh keeps the previously observed <code>NS-002</code> and <code>NS-003</code>, emits no removals, and sets <code>removals_suppressed=true</code>. Retained records keep their earlier observation timestamps. Valid newly observed records can still update the snapshot; the snapshot remains partial.</p>
<p>A complete run's <code>removed_ids</code> means absent from the validated source listing. It does not establish that a role was filled or that employment changed. A timestamp-only refresh produces no content-change alert. Review an incomplete run's failure before using its counts as a current inventory.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">Test a Production Scraper: Retries, Timeouts and Invalid Data</h3><p class="description_eu34">Run a local Python failure harness to test bounded retries, deadlines, record validation and duplicate delivery—and see what a passing schema misses.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now">→ Read Article</a><span class="readTime_W_dz">7<!-- --> min</span></div></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="optional-managed-html-acquisition-with-scrapingant">Optional managed HTML acquisition with ScrapingAnt<a href="https://scrapingant.com/blog/scraping-for-labor-market-intelligence-jobs-skills-and-wage#optional-managed-html-acquisition-with-scrapingant" class="hash-link" aria-label="Direct link to Optional managed HTML acquisition with ScrapingAnt" title="Direct link to Optional managed HTML acquisition with ScrapingAnt" translate="no">​</a></h2>
<p>If an authorized source needs managed HTML retrieval, keep the same validation boundary around the acquired content. ScrapingAnt's <a href="https://docs.scrapingant.com/request-response-format" target="_blank" rel="noopener noreferrer" class=""><code>/v2/general</code> endpoint</a> returns page HTML; it is not the structured job extractor in this example.</p>
<p>The packet's optional transport sets all of these explicitly:</p>
<table><thead><tr><th>Parameter</th><th>Value</th></tr></thead><tbody><tr><td><code>browser</code></td><td><code>false</code> for HTML without JavaScript rendering; explicitly <code>true</code> when rendered HTML is needed</td></tr><tr><td><code>proxy_type</code></td><td><code>datacenter</code></td></tr><tr><td><code>proxy_country</code></td><td><code>us</code></td></tr><tr><td><code>timeout</code></td><td><code>30</code> seconds</td></tr><tr><td><code>return_page_source</code></td><td><code>false</code></td></tr></tbody></table>
<p>The packet's code reads the API key from <code>SCRAPINGANT_API_KEY</code> and sends it in a header, keeping it out of request URLs and error messages. The code checks the API response and target status, uses the <a href="https://docs.scrapingant.com/custom-headers" target="_blank" rel="noopener noreferrer" class="">original target content-type header</a> when present, and rejects an explicit non-HTML target type. A missing target content-type leaves validation to the required listing schema. An outer response content-type alone does not establish the target's format. Its redirect guard controls API-endpoint redirects; it does not establish how a remote target redirect is handled.</p>
<p>One separate call through the unchanged <code>managed_html</code> transport fetched an <a href="https://scrapingant.github.io/scrapingant-examples/fixtures/dynamic-delayed.html" target="_blank" rel="noopener noreferrer" class="">existing owned raw fixture</a> from <strong>2026-10-11T21:38:39.695211 to 21:38:40.432850 UTC</strong>, using the explicit raw parameters above. API and target statuses were <strong>200</strong>, the credit receipt was <strong>1 API credit</strong>, and the captured HTML was <strong>752 bytes</strong>. Its initial <code>Web Scraping is hard</code> marker and full body matched the pinned fixture source. The receipt, response body and source pins are included in the reproducibility packet.</p>
<p>This observation validates owned raw HTML transport and marker acceptance. It does not validate an employer/ATS HTML source, the job parser, pagination, managed CLI or JavaScript rendering. The default runner stays offline and does not repeat this call.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="one-selected-published-posting-raw-html-to-a-validated-record">One selected published posting: raw HTML to a validated record<a href="https://scrapingant.com/blog/scraping-for-labor-market-intelligence-jobs-skills-and-wage#one-selected-published-posting-raw-html-to-a-validated-record" class="hash-link" aria-label="Direct link to One selected published posting: raw HTML to a validated record" title="Direct link to One selected published posting: raw HTML to a validated record" translate="no">​</a></h3>
<p>On <strong>October 12, 2026, from 09:42:19.062837 to 09:42:20.237010 UTC</strong>, a separate raw request acquired <a href="https://job-boards.greenhouse.io/greenhouse/jobs/8126988?gh_jid=8126988" target="_blank" rel="noopener noreferrer" class="">Greenhouse posting 8126988</a> through <code>managed_html</code> with the explicit parameters above. API and target statuses were <strong>200</strong>, the observed receipt was <strong>1 API credit</strong>, and the response was <strong>104,223 bytes</strong>. A bounded source-specific adapter accepted exactly one literal record:</p>
<ul>
<li class=""><code>jobID</code>: <code>8126988</code>; <code>title</code>: <code>Commercial Counsel</code>.</li>
<li class=""><code>employer</code>: <code>Greenhouse</code>; <code>location</code>: <code>Eastern Timezone</code>.</li>
<li class=""><code>sourceURL</code> and <code>applicationURL</code>: the same HTTPS posting page linked above.</li>
<li class=""><code>observedUTC</code>: <code>2026-10-12T09:42:20Z</code>; <code>sourcePostedDate</code>: null.</li>
</ul>
<p>The observed form uses method <code>get</code> and an action resolving to that hosted page. Recording it as <code>applicationURL</code> establishes the <strong>hosted form page</strong>, without validating a distinct submission endpoint or submitting an application. The HTML had no posting-date marker, so the adapter does not borrow dates from the API. Its canonical URL used HTTP with the same host/path/query; the adapter records this discrepancy and keeps the source and application URLs HTTPS.</p>
<p>This ordinary public page was linked by its <a href="https://boards-api.greenhouse.io/v1/boards/greenhouse/jobs/8126988" target="_blank" rel="noopener noreferrer" class="">official public job API</a> and inspected directly without authentication, a redirect or a challenge. Public JSON access is not blanket permission for arbitrary HTML. Both that API and direct raw HTML already supplied relevant facts, so the managed call demonstrates an optional acquisition path.</p>
<p>The separate <code>greenhouse_single_post.py</code> adapter checks one title, location, document title, matching Open Graph fields, canonical URL and a closed matching application form. Acceptance also requires an independently authored literal record, ignoring only observation-time differences. <code>complete=true</code> applies to <strong>one selected posting page</strong>, with <code>max_pages=1</code> and <code>board_inventory_complete=false</code>. Empty, missing, challenge or truncated HTML fails this detail-page contract.</p>
<p>The packet adds <strong>46 offline tests</strong> for this adapter and receipt controls, alongside the original 50 tests and 10 listing cases. They reject coherent field drift, unsafe application URLs, ambiguous credit receipts and encoded key echoes; harmless wrapper changes may pass. Its public CLI parses the short owned fixture or verifies a local file and makes no provider request. The recorded managed acquisition used a separately reviewed bounded helper; the public CLI was not used for that call.</p>
<p>Only safe receipt and field projections are distributed. The full employer HTML and API bodies are withheld. The measured HTML SHA-256 is <code>b8385bbc2da7e5d34e469dad34117c059449bb68786df0e5063dc21ed09bb886</code>; <strong>neither the projection nor the hand-authored structural fixture can reproduce that full-body hash</strong>. This one raw result does not establish employer inventory, pagination, refresh, JavaScript rendering, generic ATS support or production reliability.</p>
<p>After replacing and validating the source adapter, this <strong>documented opt-in command</strong> illustrates the managed path. It was not run for this packet and can consume credits. Set <code>CAREERS_URL</code> to the single source URL you have authorized and tested:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">: "${SCRAPINGANT_API_KEY:?set it in your environment}"</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">: "${CAREERS_URL:?set one authorized careers URL}"</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3.12 collector.py --start-url "$CAREERS_URL" --max-pages 3 \</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  --transport scrapingant --browser false --allow-live-api</span><br></div></code></pre></div></div>
<p>Use <code>--browser true</code> when the source requires JavaScript rendering, then validate that returned HTML against the source-specific schema. The bundled listing adapter still rejects URL shapes outside its known contract. Offline mocks verify request parameters and acceptance checks; the separate one-posting observation supplies the narrowly scoped live result above.</p>
<p>For ordinary non-Google targets, the <a href="https://docs.scrapingant.com/credits-cost" target="_blank" rel="noopener noreferrer" class="">documented credit schedule</a> lists <strong>1 API credit</strong> for a request without a browser through a datacenter proxy, and <strong>10 API credits</strong> for JavaScript rendering through a datacenter proxy. The separate owned and selected-post raw receipts each recorded 1 credit. These two observations do not estimate yield or cost for other jobs or sources.</p>
<p>Try one authorized careers URL, inspect the required fields and pagination, and trial that bounded workload only if managed retrieval is needed. As checked on October 11, 2026, the <a href="https://app.scrapingant.com/signup" target="_blank" rel="noopener noreferrer" class="">signup offer</a> lists <strong>10,000 monthly free API credits</strong> with no credit card; the <a href="https://scrapingant.com/#pricing" target="_blank" rel="noopener noreferrer" class="">current plans</a> list Enthusiast at <strong>$19/month for 100,000 credits</strong>. Recheck current plans before choosing capacity. An official feed or working direct HTTP collector may already complete your task.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="limits-of-the-evidence">Limits of the evidence<a href="https://scrapingant.com/blog/scraping-for-labor-market-intelligence-jobs-skills-and-wage#limits-of-the-evidence" class="hash-link" aria-label="Direct link to Limits of the evidence" title="Direct link to Limits of the evidence" translate="no">​</a></h2>
<p>The HTML fixture proves the demonstrated parser, page bounds and refresh behavior on owned inputs. It does not prove live ATS rendering, arbitrary Indeed/LinkedIn support, production reliability, success rates or a ready-made jobs database. No applications, profiles, skills classification or wage inference are collected.</p>
<p>The Greenhouse list and selected-post evidence contain retained fields and capture metadata. Their recorded full-response hashes identify bodies that are not distributed. Saved projections have separate hashes and cannot reproduce the original full-response hashes. Replays validate the retained fields and owned structural inputs; they do not re-acquire the withheld source or establish a current live inventory.</p>
<p>Check the adapter again when a source changes its layout, page contract or required fields. Keep incomplete runs visible and avoid downstream removal alerts until the source passes the full validation gate.</p>
<p>Examples tested on <strong>October 11 and 12, 2026</strong> with <strong>CPython 3.12.11</strong>, standard library only. Code and captured outputs: <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/23de5503bb97654afa39d9a87cbc38851bba0e6e/examples/scraping-for-labor-market-intelligence-jobs-skills-and-wage" target="_blank" rel="noopener noreferrer" class="">reproducibility packet</a>.</p>
<p><em>This article was drafted with AI assistance from a tested evidence packet. Publication approved by Oleg Kulyk on October 12, 2026.</em></p>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="job scraping" term="job scraping"/>
        <category label="python" term="python"/>
        <category label="web scraping" term="web scraping"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Retail Shelf Intelligence - Validate SKU Price and Stock Data]]></title>
        <id>https://scrapingant.com/blog/retail-shelf-intelligence-scraping-digital-shelves-for-cpg</id>
        <link href="https://scrapingant.com/blog/retail-shelf-intelligence-scraping-digital-shelves-for-cpg"/>
        <updated>2026-01-08T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Collect source-specific retail variants, pack sizes, prices and stock signals with Python. Validate observations before comparing repeated snapshots.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Retail Shelf Intelligence - Validate SKU Price and Stock Data" src="https://scrapingant.com/blog/assets/images/retail-shelf-intelligence-scraping-digital-shelves-for-cpg-5b9e710f7ebc8cb422f2cb0f0a10fb6d.png" width="1200" height="630" class="img_ev3q"></p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"></path></svg></span>Correction (2026-10-13)</div><div class="admonitionContent_BuS1"><p>The original article described broad CPG analytics and made unsupported claims about automatic extraction, layout adaptation and retailer coverage. This revision narrows the task to validated product observations and corrects the API parameter names to <code>browser</code> and <code>proxy_country</code>. <a href="https://scrapingant.com/blog/assets/files/retail-sku-observations-7d99661b57552b09d8634b41e67e8430.zip" target="_blank" class="">Download the reproducibility packet</a>.</p></div></div>
<p>A retail price observation needs a product identity, the selected variant and pack size, the offer being quoted, and evidence for its availability. Otherwise, a cheaper subscription offer or a smaller pack can look like a price drop. A failed page can look like an out-of-stock product if the collector treats every missing value as a stock signal.</p>
<p>This walkthrough builds a small source-specific Python collector for repeated SKU observations. It keeps unknown availability, rejected fields and acquisition failures distinct from explicit out-of-stock evidence. The runnable controls use a fictional tea family with two selected weight packs. A separate direct retail observation and optional managed retrieval are recorded separately. Completeness refers to those selected offers, not a retailer’s full inventory.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="select-the-source-and-the-offer-before-collecting">Select the source and the offer before collecting<a href="https://scrapingant.com/blog/retail-shelf-intelligence-scraping-digital-shelves-for-cpg#select-the-source-and-the-offer-before-collecting" class="hash-link" aria-label="Direct link to Select the source and the offer before collecting" title="Direct link to Select the source and the offer before collecting" translate="no">​</a></h2>
<p>Start with a small list of product URLs you are permitted to collect. Check access and reuse requirements, then inspect whether an official feed, product endpoint or direct HTML already supplies the required fields. If it does, use that source before adding a managed retrieval service. A robots directive alone is not a complete source-selection or redistribution review.</p>
<p>The source investigation observed <a href="https://www.adagio.com/black/earl_grey_lavender_tea.html" target="_blank" rel="noopener noreferrer" class="">Adagio's Earl Grey Lavender product page</a> through direct raw HTTP on <strong>October 13, 2026 at 10:37:02 UTC</strong>. Its <code>ProductGroup.hasVariant</code> JSON-LD supplied these selected listed offers:</p>
<table><thead><tr><th>SKU, group <code>001036</code></th><th>Weight</th><th>Listed price</th><th>Explicit source availability</th></tr></thead><tbody><tr><td><code>001036002</code></td><td><code>3oz</code></td><td><code>9.00 USD</code></td><td><code>https://schema.org/OutOfStock</code></td></tr><tr><td><code>001036004</code></td><td><code>16oz</code></td><td><code>29.00 USD</code></td><td><code>https://schema.org/OutOfStock</code></td></tr></tbody></table>
<p>Both offers share the product URL. Their group, SKU and exact size bind the observations; no separate variant ID is invented. The page also supplies a US shipping-destination marker. That is page context, not a selected checkout destination: fulfilment remains unselected and one-time/subscription purchase basis unspecified. Separate auto-delivery discounts are not substituted for the listed variant prices.</p>
<p>Direct HTML already supplied the selected fields. No additional feed route was probed and no JavaScript requirement was established. The packet retains a minimal factual projection for offline inspection; full retailer HTML is withheld. Recheck the source's <a href="https://www.adagio.com/robots.txt" target="_blank" rel="noopener noreferrer" class="">robots directives</a> and its linked policies, including the <a href="https://www.adagio.com/pages/privacy_policy.html" target="_blank" rel="noopener noreferrer" class="">privacy policy</a>, before choosing a new collection scope. This observation is not blanket permission to reuse the site.</p>
<p>For this example, the useful identity is more specific than a product title: the selected product group, SKU, variant and pack quantity with its unit. Keep the offer context beside that identity. If the source does not provide market or fulfilment evidence, leave that context unknown; a proxy country cannot establish it. A subscription discount or an aggregate “from” price is a different observation from the selected base offer.</p>
<p>Do not infer the selected variant from a title substring or choose the lowest price across offers. Resolve it against the inspected source structure and reject an ambiguous result. This collector is deliberately tied to a small configured source contract. Adding another retailer requires its own identity rules, expected fields and fixtures.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="define-an-accepted-observation">Define an accepted observation<a href="https://scrapingant.com/blog/retail-shelf-intelligence-scraping-digital-shelves-for-cpg#define-an-accepted-observation" class="hash-link" aria-label="Direct link to Define an accepted observation" title="Direct link to Define an accepted observation" translate="no">​</a></h2>
<p>The collector emits the following fields for each accepted selected offer:</p>
<table><thead><tr><th>Fields</th><th>Acceptance rule</th></tr></thead><tbody><tr><td><code>product_id</code>, <code>sku</code>, <code>variant_id</code>, <code>variant</code></td><td>Bind the configured product group to one selected SKU and exact size; <code>variant_id</code> is null because this source has no separate variant ID.</td></tr><tr><td><code>pack_quantity</code>, <code>pack_unit</code></td><td>Normalize the selected size to a weight in ounces. A changed pack is not silently treated as the same offer.</td></tr><tr><td><code>price</code>, <code>currency</code></td><td>Keep a decimal price string and an explicit currency. Reject missing or ambiguous values.</td></tr><tr><td><code>availability</code></td><td>Preserve <code>in_stock</code>, explicit <code>out_of_stock</code>, or <code>unknown</code>; never derive stock from a missing record.</td></tr><tr><td><code>market</code>, <code>market_basis</code></td><td>Retain a market only when page shipping metadata supplies it; this is page context, not a selected destination.</td></tr><tr><td><code>fulfilment</code>, <code>offer_kind</code>, <code>purchase_basis</code>, <code>offer_context</code></td><td>Keep fulfilment unselected, purchase basis unspecified and the listed public offer context; no one-time/subscription price is inferred.</td></tr><tr><td><code>observed_at_utc</code></td><td>Record when the observation was collected; this is not a retailer publication or update date.</td></tr><tr><td><code>requested_source_url</code>, <code>observed_source_url</code></td><td>Retain the requested URL and the accepted source URL separately.</td></tr><tr><td><code>source_evidence</code></td><td>Retain bounded field paths and literal source tokens supporting the selected identity, pack, price and stock signal.</td></tr></tbody></table>
<p>Store price values without converting them through a binary floating-point calculation. Compare only observations with the same identity, pack, currency and offer context. A collector cannot repair a missing currency by guessing it from a country label.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="reproduce-the-fixture-failure-and-the-bounded-collector">Reproduce the fixture failure and the bounded collector<a href="https://scrapingant.com/blog/retail-shelf-intelligence-scraping-digital-shelves-for-cpg#reproduce-the-fixture-failure-and-the-bounded-collector" class="hash-link" aria-label="Direct link to Reproduce the fixture failure and the bounded collector" title="Direct link to Reproduce the fixture failure and the bounded collector" translate="no">​</a></h2>
<p>Use <strong>CPython 3.12.11</strong>; the packet uses only the standard library. Download the archive, extract it and run:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">unzip retail-sku-observations.zip</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">cd retail-sku-observations</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">PYTHON=python3 ./run.sh</span><br></div></code></pre></div></div>
<p>In the captured run, <code>python3</code> resolved to CPython 3.12.11. Set <code>PYTHON</code> to your 3.12.11 executable if its name differs; the runner checks the exact version. <code>run.sh</code> runs the scoped tests and captures JSON, CSV and change-report outputs from owned fixtures and a saved field projection. The test process rejects external connection attempts; the demo invokes local-file CLI replays. A configured API key does not make fixture replay dispatch a managed request.</p>
<p>The failing naive choice is the fictional group’s “from” price. It belongs to a sample, rather than the selected weight offer. <code>expected_output/naive-price.json</code> captures:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"currency"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"USD"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"matches"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token boolean">false</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"naive_group_low_price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"2.00"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"scope"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"owned fictional data; aggregate sample price cannot be assigned to the selected weight offer"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"selected_SKU"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"000701002"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"selected_offer_price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"8.25"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"selected_pack"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"3oz"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><br></div></code></pre></div></div>
<p>The captured runner summary is:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">Runtime: cpython 3.12.11; standard library only</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Default scope: owned fixtures and saved factual projection; no external/provider/model calls</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Owned baseline: 2/2 selected offers accepted; complete=true</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Owned changed: 2/2 selected offers accepted; comparable field changes=2</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Owned partial: 1/2 selected offers accepted; unobserved prior records=1</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Changed-pack control: 1/2 accepted; reason=pack_identity_changed; prior pack retained=1</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Missing observations make no out-of-stock or disappearance claim; prior timestamps retained</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Naive from/sample price=2.00; selected 3oz offer=8.25 USD</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Adagio saved projection: 2/2 literal offers match independent source oracle; no fresh HTTP request</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Recorded managed observation: 1 request; API/target=200/200; 2/2 accepted; receipt credits=1; credits per accepted record=0.50</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Recorded managed data are read locally; standalone managed CLI excluded and no paid observation rerun</span><br></div></code></pre></div></div>
<p>Fixture prices and timestamps are controlled inputs. They are not current retailer observations. The tests compare full records against literal expected values; JSON and CSV retain the leading zeroes in source IDs.</p>
<p>The controls should establish both that acquisition finished within the selected limits and that the returned data satisfied the source contract. HTTP 200 alone does not establish field acceptance. An error shell, missing price or ambiguous variant must fail acceptance even when its transport status is successful.</p>
<p>Here is the first record excerpt from <code>expected_output/baseline.json</code>:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"availability"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"in_stock"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"currency"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"USD"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"fulfilment"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"unselected"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"market"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"unknown"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"market_basis"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"unspecified"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"observed_at_utc"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"2026-10-13T11:00:00Z"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"observed_source_url"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"https://scrapingant.github.io/scrapingant-examples/examples/retail-shelf-intelligence-scraping-digital-shelves-for-cpg/fixtures/baseline.html"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"offer_context"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"owned fictional USD offer"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"offer_kind"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"listed_variant_offer"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"pack_quantity"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">3</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"pack_unit"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"ounce"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"8.25"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"product_id"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"000701"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"purchase_basis"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"unspecified"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"requested_source_url"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"https://scrapingant.github.io/scrapingant-examples/examples/retail-shelf-intelligence-scraping-digital-shelves-for-cpg/fixtures/baseline.html"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"sku"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"000701002"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"source_evidence"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"availability_token"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"https://schema.org/InStock"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"offer_path"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"$.hasVariant[1].offers"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"pack_label"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"3oz"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"price_token"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"8.25"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token property">"product_path"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"$.hasVariant[1]"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"variant"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"3oz"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"variant_id"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token null keyword" style="color:rgb(86, 156, 214)">null</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><br></div></code></pre></div></div>
<p>The expected records are literal fixtures rather than values computed through the parser. The source investigator supplied a separate factual oracle for the direct retail observation. Controls alter material record fields and reject the mismatch. A test that computes its expected price using the same selection code could repeat the same mistake.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="keep-acquisition-and-stock-states-separate">Keep acquisition and stock states separate<a href="https://scrapingant.com/blog/retail-shelf-intelligence-scraping-digital-shelves-for-cpg#keep-acquisition-and-stock-states-separate" class="hash-link" aria-label="Direct link to Keep acquisition and stock states separate" title="Direct link to Keep acquisition and stock states separate" translate="no">​</a></h2>
<p>Acquisition completeness and the availability field answer different questions:</p>
<table><thead><tr><th>Observed condition</th><th>Record or run state</th><th>Safe downstream interpretation</th></tr></thead><tbody><tr><td>An explicit accepted stock token for the selected variant</td><td><code>in_stock</code> or <code>out_of_stock</code></td><td>Report that source observation in its retained context.</td></tr><tr><td>No recognized availability token</td><td><code>unknown</code></td><td>No stock assertion.</td></tr><tr><td>A required price, currency, pack or identity fails validation</td><td>Rejected record; incomplete run</td><td>Preserve the failure rather than manufacture a product observation.</td></tr><tr><td>A request fails transport/status checks</td><td>Acquisition failure; incomplete run</td><td>The selected SKU was not observed.</td></tr><tr><td>HTTP 200 returns an error shell that fails the source contract</td><td>Content rejected; no accepted record; incomplete run</td><td>The selected SKU was not observed; HTTP success does not imply stock.</td></tr><tr><td>A configured SKU is absent from a partial collection</td><td>Unobserved</td><td>Retain the prior observation without calling it current or out of stock.</td></tr><tr><td>Selected pack quantity changes</td><td>Rejected <code>pack_identity_changed</code>; incomplete run</td><td>Retain the prior pack observation; do not report a comparable price change.</td></tr></tbody></table>
<p>The controls include an HTTP 200 error shell, missing price/currency, duplicate or conflicting selected SKUs, unknown availability, explicit out-of-stock offers, a failed acquisition and a changed pack. A complete result with <code>availability="unknown"</code> still makes no stock assertion.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="compare-repeated-snapshots-conservatively">Compare repeated snapshots conservatively<a href="https://scrapingant.com/blog/retail-shelf-intelligence-scraping-digital-shelves-for-cpg#compare-repeated-snapshots-conservatively" class="hash-link" aria-label="Direct link to Compare repeated snapshots conservatively" title="Direct link to Compare repeated snapshots conservatively" translate="no">​</a></h2>
<p>Use the accepted records to compare like-for-like offers. Emit a price change only when the required identity and context match and both snapshots contain accepted decimal prices. Preserve availability-field changes as observed states. A transition to or from <code>unknown</code> does not establish a stock-in or out-of-stock event.</p>
<p>The complete changed fixture produces this <code>expected_output/changes.json</code>:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"changes"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"fields"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token property">"availability"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">          </span><span class="token property">"after"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"out_of_stock"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">          </span><span class="token property">"before"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"in_stock"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token property">"price"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">          </span><span class="token property">"after"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"8.75"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">          </span><span class="token property">"before"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"8.25"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"sku"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"000701002"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"variant"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"3oz"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"fields"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token property">"availability"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">          </span><span class="token property">"after"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"unknown"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">          </span><span class="token property">"before"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"out_of_stock"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"sku"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"000701004"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">      </span><span class="token property">"variant"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"16oz"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"current_complete"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token boolean">true</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"new_observations"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"scope"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"two selected offers; no inventory/disappearance inference"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token property">"unobserved_prior_records"</span><span class="token operator" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><br></div></code></pre></div></div>
<p>The 3-ounce fixture price changes from <code>8.25</code> to <code>8.75</code> and its explicit token changes from in stock to out of stock. The 16-ounce offer becomes <code>unknown</code>; that is loss of an explicit stock signal. In the partial fixture, the 16-ounce offer is missing and its prior record remains in <code>unobserved_prior_records</code> with timestamp <code>2026-10-13T11:00:00Z</code>. A changed 16-to-12-ounce pack fails the fixed identity check. Neither control manufactures a disappearance or stock event.</p>
<p>The result is a small observation and change dataset. It does not establish retailer-wide assortment completeness, cross-retailer matching, share of shelf, revenue effects or MAP compliance. Those tasks need additional inputs and definitions beyond this collector.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">Price Scraping Tools and Uses for Price Monitoring</h3><p class="description_eu34">What price scraping is, where it pays off and how a web scraping API tool collects competitor prices, with a Python example run against a live API.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/web-scraping-for-price-monitoring">→ Read Article</a><span class="readTime_W_dz">6<!-- --> min</span></div></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="add-managed-retrieval-only-when-the-source-needs-it">Add managed retrieval only when the source needs it<a href="https://scrapingant.com/blog/retail-shelf-intelligence-scraping-digital-shelves-for-cpg#add-managed-retrieval-only-when-the-source-needs-it" class="hash-link" aria-label="Direct link to Add managed retrieval only when the source needs it" title="Direct link to Add managed retrieval only when the source needs it" translate="no">​</a></h2>
<p>If permitted direct HTTP or an adequate product feed provides the fields, ScrapingAnt is not needed for this task. If your inspected source requires managed HTML retrieval, keep the same parser and field acceptance rules when changing the transport.</p>
<p>The general endpoint returns HTML. Set <code>browser</code>, <code>proxy_type</code>, <code>proxy_country</code>, <code>return_page_source</code> and <code>timeout</code> explicitly; the examples use <code>timeout=30</code> seconds. <code>browser=false</code> requests raw retrieval. <code>browser=true</code> with <code>return_page_source=false</code> requests JavaScript-rendered HTML. The customer parser owns product-field extraction. These are documented interface choices; choosing rendered mode does not prove JavaScript was necessary for the selected source. See the <a href="https://docs.scrapingant.com/request-response-format" target="_blank" rel="noopener noreferrer" class="">request and response format</a>.</p>
<p>Send the API key in the <code>x-api-key</code> request header and keep it out of logged request URLs. The optional CLI reads <code>SCRAPINGANT_API_KEY</code> from the environment; it prints normalized records and a safe receipt rather than the returned HTML. Its fixture tests check the explicit request settings, header authentication and encoded credential echoes.</p>
<p>From the extracted folder, the optional direct command makes one request to the fixed Adagio product page:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3 collector.py direct</span><br></div></code></pre></div></div>
<p>The optional managed command makes one paid request to that same fixed page. Supply <code>SCRAPINGANT_API_KEY</code> through your environment before using it:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3 collector.py managed --browser false</span><br></div></code></pre></div></div>
<p>These network commands are excluded from <code>run.sh</code>. They accept no arbitrary retailer URL, and add no retries or schedule. The standalone managed CLI has offline fake-response coverage; the recorded live check below exercised the native collector functions through a bounded controller.</p>
<p>The documented non-Google datacenter retrieval costs are:</p>
<table><thead><tr><th>Explicit mode</th><th>Documented request cost</th></tr></thead><tbody><tr><td><code>browser=false</code>, <code>proxy_type=datacenter</code></td><td>1 API credit</td></tr><tr><td><code>browser=true</code>, <code>return_page_source=false</code>, <code>proxy_type=datacenter</code></td><td>10 API credits</td></tr></tbody></table>
<p>These values come from the <a href="https://docs.scrapingant.com/credits-cost" target="_blank" rel="noopener noreferrer" class="">credit schedule</a>. They describe retrieval cost, not accepted records. Record the <code>Ant-credits-cost</code> response header when present; a missing or malformed receipt leaves the actual request cost unknown. Do not infer zero credits from an absent receipt. Keep the API status, target status, acceptance result and actual receipt separate. The <a href="https://docs.scrapingant.com/errors" target="_blank" rel="noopener noreferrer" class="">error documentation</a> distinguishes the outer API HTTP status from the target status reported by <code>Ant-page-status-code</code>.</p>
<p>The default runner made no external or paid API calls. A separate bounded managed check on <strong>October 13, 2026 at 10:59:39 UTC</strong> made one raw datacenter request for the selected Adagio page. The collector accepted both selected offers: <code>001036002</code>, <code>3oz</code>, <code>9.00 USD</code>, <code>out_of_stock</code>; and <code>001036004</code>, <code>16oz</code>, <code>29.00 USD</code>, <code>out_of_stock</code>. These matched the separately inspected direct-source oracle.</p>
<table><thead><tr><th>Managed observation</th><th>API / target status</th><th>Accepted offers</th><th>Actual receipt</th></tr></thead><tbody><tr><td>One raw request, <code>browser=false</code></td><td><code>200 / 200</code>, <code>text/html</code></td><td><code>2 / 2</code>; complete</td><td><code>1</code> API credit</td></tr></tbody></table>
<p>For this one observation, the receipt divided by accepted offers is <code>0.50</code> credits per record, with no unknown charge. It is not a production yield estimate. The check exercised the collector’s <code>acquire</code> and <code>collection</code> functions through a bounded controller. The public managed CLI was tested with offline fakes; a live CLI invocation was not performed. Direct HTML already supplied the same selected fields, so this observation does not establish a need for managed retrieval or rendering. The <a href="https://github.com/ScrapingAnt/scrapingant-examples/blob/09aa342dc9b1a3aec9b92895332d320582bfddb3/examples/retail-shelf-intelligence-scraping-digital-shelves-for-cpg/expected_output/managed-2026-10-13/receipt.json" target="_blank" rel="noopener noreferrer" class="">receipt</a> and <a href="https://github.com/ScrapingAnt/scrapingant-examples/blob/09aa342dc9b1a3aec9b92895332d320582bfddb3/examples/retail-shelf-intelligence-scraping-digital-shelves-for-cpg/expected_output/managed-2026-10-13/collection.json" target="_blank" rel="noopener noreferrer" class="">accepted records</a> preserve the measured result. Its <a href="https://github.com/ScrapingAnt/scrapingant-examples/blob/09aa342dc9b1a3aec9b92895332d320582bfddb3/examples/retail-shelf-intelligence-scraping-digital-shelves-for-cpg/expected_output/managed-2026-10-13/provenance.json" target="_blank" rel="noopener noreferrer" class="">provenance</a> records the <a href="https://github.com/ScrapingAnt/scrapingant-examples/blob/09aa342dc9b1a3aec9b92895332d320582bfddb3/examples/retail-shelf-intelligence-scraping-digital-shelves-for-cpg/validation/raw-call-collector.py" target="_blank" rel="noopener noreferrer" class="">executed source snapshot</a>; the packet strengthens validation of saved projections and normalizes malformed HTML parser failures after that capture.</p>
<p>For a first managed test, select one permitted URL and verify its source identity, selected variant and required fields. Repeat only a bounded workload after that acceptance check. If managed retrieval is needed, <a href="https://app.scrapingant.com/signup" target="_blank" rel="noopener noreferrer" class="">create an account</a> and review the <a href="https://scrapingant.com/#pricing" target="_blank" rel="noopener noreferrer" class="">current pricing</a>; the field validator remains your code.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">C# HttpClient Web Scraping with ScrapingAnt and Proxies</h3><p class="description_eu34">Call ScrapingAnt from C# with explicit proxy and browser settings, parse returned HTML, validate fields and credits, and bound retries.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/web-scraping-in-c-with-httpclient-and-proxies-a-2025">→ Read Article</a><span class="readTime_W_dz">9<!-- --> min</span></div></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="limits-and-review-status">Limits and review status<a href="https://scrapingant.com/blog/retail-shelf-intelligence-scraping-digital-shelves-for-cpg#limits-and-review-status" class="hash-link" aria-label="Direct link to Limits and review status" title="Direct link to Limits and review status" translate="no">​</a></h2>
<p>The parser recognizes the fixed Adagio product group and its two selected weight SKUs, plus the fictional fixture equivalent. It requires one matching canonical URL and one <code>ProductGroup</code> JSON-LD object. It does not implement arbitrary JSON-LD graphs, product discovery or another retailer’s selectors.</p>
<p>Acquisition uses one attempt, a 262,144-byte response cap, a 30-second socket timeout and elapsed-time checks before and after bounded reads. A blocking read can finish before the elapsed check rejects it; this is not a promise of an exact 30-second total runtime. Direct HTTP and API endpoint redirects are rejected; the provider's remote target redirect behavior was not independently validated. There are no retries or automatic rendering/proxy fallbacks. Transport fixtures cover target statuses, media types, byte limits, errors and unknown receipts; mocked headers are not paid measurements.</p>
<p>Owned fixture success does not demonstrate arbitrary retailer reliability. Source layouts and offer semantics need their own validation; source evidence should be sufficient to audit the fields without redistributing full retailer page bodies. There is no scheduler or automatic variant matching in this example. The public field projection cannot regenerate the hash of the withheld original retailer HTML, and replay does not create a fresh source observation.</p>
<p>Examples tested on 2026-10-13 with CPython 3.12.11 and the standard library. Code: <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/09aa342dc9b1a3aec9b92895332d320582bfddb3/examples/retail-shelf-intelligence-scraping-digital-shelves-for-cpg" target="_blank" rel="noopener noreferrer" class="">Code and evidence</a>.</p>
<p><em>This article was drafted with AI assistance from a tested evidence packet and independently reviewed by an AI assistant. Publication was authorized by the named author, who is responsible for the code, measurements and corrections.</em></p>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="CPG analytics" term="CPG analytics"/>
        <category label="digital shelf" term="digital shelf"/>
        <category label="retail data" term="retail data"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[LLM-Powered Trend Analysis - From Scraped Signals to Narratives]]></title>
        <id>https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to</id>
        <link href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to"/>
        <updated>2025-12-24T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Combine scraped trend data with LLMs to generate narrative market insights, anomaly explanations, and executive-ready summaries automatically.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="LLM-Powered Trend Analysis: From Scraped Signals to Narratives" src="https://scrapingant.com/blog/assets/images/llm-powered-trend-analysis-from-scraped-signals-to-26e2343dfe5c5a2918e3e2f538d999f1.png" width="1200" height="630" class="img_ev3q"></p>
<p><a href="https://scrapingant.com/mcp-server-web-scraping" target="_blank" rel="noopener noreferrer" class="">Large language models (LLMs)</a> are changing how organizations interpret digital signals into meaningful narratives. Instead of manually interpreting search data, social chatter, and web content, analysts can now use LLMs to convert raw, noisy signals into structured insights and strategic recommendations. When combined with web scraping pipelines and tools like Google Trends, this creates a powerful stack for continuous trend detection, interpretation, and communication.</p>
<p>This report examines how to build LLM-powered trend analysis pipelines – from data collection via scraping and APIs (with a focus on ScrapingAnt as the primary web scraping solution), through signal extraction and modeling, to narrative generation for decision-makers. It also reviews recent developments in LLM capabilities and trend analytics and provides practical examples and implementation considerations.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-conceptual-framework-from-signals-to-narratives">1. Conceptual Framework: From Signals to Narratives<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#1-conceptual-framework-from-signals-to-narratives" class="hash-link" aria-label="Direct link to 1. Conceptual Framework: From Signals to Narratives" title="Direct link to 1. Conceptual Framework: From Signals to Narratives" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="11-definitions">1.1 Definitions<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#11-definitions" class="hash-link" aria-label="Direct link to 1.1 Definitions" title="Direct link to 1.1 Definitions" translate="no">​</a></h3>
<ul>
<li class=""><strong>Trend analysis</strong>: The systematic identification and interpretation of changes in interest, behavior, or patterns over time within a domain (e.g., consumer products, technologies, public opinion).</li>
<li class=""><strong>Signals</strong>: Discrete observations that may indicate an emerging trend – e.g., spikes in Google search interest, increased mentions on social media, product review patterns, or changes in pricing/availability on e-commerce sites.</li>
<li class=""><strong>Narratives</strong>: Human-readable stories that explain what is happening, why it matters, who is affected, and what actions should follow.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="12-the-llm-powered-pipeline">1.2 The LLM-Powered Pipeline<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#12-the-llm-powered-pipeline" class="hash-link" aria-label="Direct link to 1.2 The LLM-Powered Pipeline" title="Direct link to 1.2 The LLM-Powered Pipeline" translate="no">​</a></h3>
<p>A generic LLM-powered trend analysis system can be conceptualized in four layers:</p>
<ol>
<li class=""><strong>Data acquisition (scraping &amp; APIs)</strong>
<ul>
<li class="">Web pages, search trends, social media, reviews, news, etc.</li>
<li class="">Tools: ScrapingAnt, Google Trends API wrappers, social media APIs, RSS feeds.</li>
</ul>
</li>
<li class=""><strong>Signal extraction &amp; structuring</strong>
<ul>
<li class="">Cleaning, deduplication, entity extraction, time-series construction.</li>
<li class="">NLP and traditional stats (e.g., change point detection).</li>
</ul>
</li>
<li class=""><strong>Trend modeling &amp; validation</strong>
<ul>
<li class="">Time-series analysis, anomaly detection, clustering, forecasting.</li>
<li class="">Human-in-the-loop validation.</li>
</ul>
</li>
<li class=""><strong>Narrative generation &amp; decision support</strong>
<ul>
<li class="">LLMs summarize, contextualize, and generate scenarios, implications, and recommendations tailored to specific stakeholders.</li>
</ul>
</li>
</ol>
<p>LLMs are most impactful in layers 2 and 4 – turning heterogeneous data into structured views and then into narratives – but they increasingly assist in orchestrating the full pipeline.</p>
<hr>
<p><img decoding="async" loading="lazy" alt="End-to-end LLM-powered trend analysis pipeline" src="https://scrapingant.com/blog/assets/images/llm-powered-trend-analysis-from-scraped-signals-to-diagram-1-e5eb10077d2bde2e93945343d8ae1ef9.svg" width="702" height="1244" class="img_ev3q"></p>
<p><em>Illustrates: End-to-end LLM-powered trend analysis pipeline</em></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-data-acquisition-scraped-web-signals-and-google-trends">2. Data Acquisition: Scraped Web Signals and Google Trends<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#2-data-acquisition-scraped-web-signals-and-google-trends" class="hash-link" aria-label="Direct link to 2. Data Acquisition: Scraped Web Signals and Google Trends" title="Direct link to 2. Data Acquisition: Scraped Web Signals and Google Trends" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Web scraping infrastructure for resilient data acquisition" src="https://scrapingant.com/blog/assets/images/llm-powered-trend-analysis-from-scraped-signals-to-diagram-2-d0c80bef398d9ae516896be084763e59.svg" width="968" height="1524" class="img_ev3q"></p>
<p><em>Illustrates: Web scraping infrastructure for resilient data acquisition</em></p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="21-web-scraping-as-the-foundation">2.1 Web Scraping as the Foundation<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#21-web-scraping-as-the-foundation" class="hash-link" aria-label="Direct link to 2.1 Web Scraping as the Foundation" title="Direct link to 2.1 Web Scraping as the Foundation" translate="no">​</a></h3>
<p>Robust trend analysis depends on reliable, large-scale, and continuous data collection. Modern websites increasingly employ dynamic JavaScript rendering, aggressive bot detection, and CAPTCHAs, making naive scraping brittle and error-prone. An industrial-grade scraping setup therefore needs:</p>
<ul>
<li class=""><strong>Rotating proxies</strong> to distribute requests and reduce blocking.</li>
<li class=""><strong>Headless browser / <a href="https://scrapingant.com/web-scraping-api" target="_blank" rel="noopener noreferrer" class="">JavaScript rendering</a></strong> to extract content rendered client-side.</li>
<li class=""><strong>CAPTCHA solving</strong> to pass automated verification gates where legally permissible.</li>
<li class=""><strong>API abstraction</strong> to simplify complex crawling workflows.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="22-scrapingant-as-the-primary-scraping-solution">2.2 ScrapingAnt as the Primary Scraping Solution<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#22-scrapingant-as-the-primary-scraping-solution" class="hash-link" aria-label="Direct link to 2.2 ScrapingAnt as the Primary Scraping Solution" title="Direct link to 2.2 ScrapingAnt as the Primary Scraping Solution" translate="no">​</a></h3>
<p>ScrapingAnt provides AI-powered web scraping that directly addresses those challenges: rotating proxies, full JavaScript rendering, and integrated CAPTCHA solving via a <a href="https://scrapingant.com/" target="_blank" rel="noopener noreferrer" class="">unified API</a>. This combination makes it particularly well-suited as the backbone of data pipelines for LLM-based trend analysis.</p>
<p>Key characteristics relevant to trend analysis:</p>
<table><thead><tr><th>Capability</th><th>Why It Matters for Trend Analysis</th></tr></thead><tbody><tr><td>Rotating proxies</td><td>Enables large-scale, geographically distributed data collection without frequent IP bans.</td></tr><tr><td>JavaScript rendering</td><td>Supports scraping modern SPAs (news sites, e-commerce, SaaS dashboards) where trends are visible.</td></tr><tr><td>CAPTCHA solving</td><td>Maintains continuity of longitudinal datasets despite anti-bot protections.</td></tr><tr><td>API-first design</td><td>Simplifies integration into trend pipelines and LLM orchestration frameworks.</td></tr><tr><td>AI-powered extraction</td><td>Allows targeted collection (e.g., prices, ratings, headlines, schema.org data) for structured signals.</td></tr></tbody></table>
<p>For example, a consumer electronics brand analyzing adoption of “USB-C monitors” across regions could use ScrapingAnt to:</p>
<ul>
<li class="">Scrape product catalogs from major retailers weekly.</li>
<li class="">Extract prices, stock status, user ratings, and review text.</li>
<li class="">Feed this into a time-series database and LLM for pattern discovery and narrative generation.</li>
</ul>
<p>By standardizing on ScrapingAnt as the primary scraping interface, teams avoid building and maintaining custom proxy rotation, rendering, and anti-CAPTCHA stacks and can focus on higher-value analysis.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="23-google-trends-as-a-core-demand-side-signal">2.3 Google Trends as a Core Demand-Side Signal<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#23-google-trends-as-a-core-demand-side-signal" class="hash-link" aria-label="Direct link to 2.3 Google Trends as a Core Demand-Side Signal" title="Direct link to 2.3 Google Trends as a Core Demand-Side Signal" translate="no">​</a></h3>
<p><strong>Google Trends</strong> provides normalized indices of search interest over time and geography, reflecting relative attention to specific queries (<a href="https://support.google.com/trends/answer/4365533" target="_blank" rel="noopener noreferrer" class="">Google Trends Help</a>). This is highly valuable for trend analysis because:</p>
<ul>
<li class="">It captures <strong>intent and curiosity</strong>, not just behavior (e.g., “how to use generative AI in marketing”).</li>
<li class="">It has <strong>global coverage and daily granularity</strong> for many queries.</li>
<li class="">It allows <strong>topic-based exploration</strong>, related queries, and comparisons.</li>
</ul>
<p>However, Google does not provide a fully documented public API for Trends; instead, analysts rely on unofficial Python packages such as <strong>pytrends</strong> that simulate browser interactions (<a href="https://github.com/GeneralMills/pytrends" target="_blank" rel="noopener noreferrer" class="">Hennig, 2022</a>). When direct API access is constrained, a scraping proxy like ScrapingAnt can be used – within Google’s terms of service – to:</p>
<ul>
<li class="">Automate retrieval of Google Trends charts and CSV exports.</li>
<li class="">Capture related topics/queries and interest by region.</li>
<li class="">Maintain historical archives for specific dashboards.</li>
</ul>
<p>This combination of official UI access, automation, and an LLM on top of the exported data panel allows flexible, code-based trend analysis.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-building-the-signal-layer-from-raw-text-and-time-series-to-features">3. Building the Signal Layer: From Raw Text and Time-Series to Features<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#3-building-the-signal-layer-from-raw-text-and-time-series-to-features" class="hash-link" aria-label="Direct link to 3. Building the Signal Layer: From Raw Text and Time-Series to Features" title="Direct link to 3. Building the Signal Layer: From Raw Text and Time-Series to Features" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="31-data-types-and-their-roles">3.1 Data Types and Their Roles<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#31-data-types-and-their-roles" class="hash-link" aria-label="Direct link to 3.1 Data Types and Their Roles" title="Direct link to 3.1 Data Types and Their Roles" translate="no">​</a></h3>
<p>Trend analysis benefits from multiple, complementary signal types:</p>
<table><thead><tr><th>Data Type</th><th>Role in Trend Analysis</th><th>Example Source</th></tr></thead><tbody><tr><td>Search interest</td><td>Early indicator of curiosity or concern</td><td>Google Trends</td></tr><tr><td>Social media mentions</td><td>Real-time chatter, meme propagation, sentiment</td><td>X (Twitter), Reddit, TikTok</td></tr><tr><td>News &amp; blogs</td><td>Narratives, expert framing, agenda-setting</td><td>News sites, Medium, Substack</td></tr><tr><td>Product &amp; app reviews</td><td>User experience, pain points, emerging use cases</td><td>App stores, Amazon, G2, Capterra</td></tr><tr><td>E-commerce pricing/stock</td><td>Demand-supply dynamics, scarcity, promotions</td><td>Amazon, Walmart, regional retailers</td></tr><tr><td>Job postings</td><td>Emerging skills, technologies, and organizational investments</td><td>LinkedIn, Indeed</td></tr></tbody></table>
<p>ScrapingAnt can be used as the unified scraper across many of these sources, handling complex web layouts and dynamic content.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="32-preprocessing-and-structuring">3.2 Preprocessing and Structuring<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#32-preprocessing-and-structuring" class="hash-link" aria-label="Direct link to 3.2 Preprocessing and Structuring" title="Direct link to 3.2 Preprocessing and Structuring" translate="no">​</a></h3>
<p>Before involving LLMs, preprocessing ensures data quality:</p>
<ol>
<li class="">
<p><strong>De-duplication and normalization</strong></p>
<ul>
<li class="">Remove duplicate articles, normalize brand and product names.</li>
<li class="">Standardize timestamps to a reference timezone.</li>
</ul>
</li>
<li class="">
<p><strong>Entity and attribute extraction</strong></p>
<ul>
<li class="">Extract entities (companies, products, people, locations) via NER.</li>
<li class="">Use ScrapingAnt’s AI extraction or custom XPath/JSONPath for structured fields (e.g., price, rating, SKU).</li>
</ul>
</li>
<li class="">
<p><strong>Time-series construction</strong></p>
<ul>
<li class="">Aggregate mentions, ratings, or sentiment by day/week.</li>
<li class="">Align with Google Trends indices to see if search interest correlates with media or commerce signals.</li>
</ul>
</li>
<li class="">
<p><strong>Language detection and translation</strong></p>
<ul>
<li class="">Auto-detect language and machine-translate where necessary, allowing the LLM to analyze global patterns.</li>
</ul>
</li>
</ol>
<p>LLMs can already assist at this stage by:</p>
<ul>
<li class="">Classifying documents into topical categories.</li>
<li class="">Extracting abstractive tags (e.g., “privacy concern,” “workflow automation”).</li>
<li class="">Summarizing long documents while preserving key structured fields.</li>
</ul>
<p>However, for reproducibility and scale, combining LLMs with deterministic transformations (e.g., regex, schema extraction, Python data pipelines) is recommended.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="4-trend-modeling-quantifying-and-qualifying-change">4. Trend Modeling: Quantifying and Qualifying Change<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#4-trend-modeling-quantifying-and-qualifying-change" class="hash-link" aria-label="Direct link to 4. Trend Modeling: Quantifying and Qualifying Change" title="Direct link to 4. Trend Modeling: Quantifying and Qualifying Change" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="41-time-series-techniques">4.1 Time-Series Techniques<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#41-time-series-techniques" class="hash-link" aria-label="Direct link to 4.1 Time-Series Techniques" title="Direct link to 4.1 Time-Series Techniques" translate="no">​</a></h3>
<p>Trend analysis is not just about noticing spikes, but also about separating noise from signal. Useful methods include:</p>
<ul>
<li class=""><strong>Change point detection</strong>: Identifies times when the underlying pattern shifts (e.g., a step-change in search interest after a product launch).</li>
<li class=""><strong>Seasonal decomposition</strong>: Decomposes time-series into trend, seasonality, and residual components to distinguish recurring events from genuine emergent trends (<a href="https://otexts.com/fpp3/" target="_blank" rel="noopener noreferrer" class="">Hyndman &amp; Athanasopoulos, 2021</a>).</li>
<li class=""><strong>Granger causality tests</strong>: Check whether changes in one series (e.g., news coverage) systematically precede changes in another (e.g., search interest or sales).</li>
<li class=""><strong>Forecasting models</strong>: ARIMA, Prophet, or modern deep learning forecasters for projecting trend trajectories.</li>
</ul>
<p>LLMs are not yet the best tools for raw time-series modeling; classical and ML time-series methods remain more transparent and reliable. LLMs become valuable in interpreting and explaining the outputs of such models.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="42-topic-detection-and-evolution">4.2 Topic Detection and Evolution<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#42-topic-detection-and-evolution" class="hash-link" aria-label="Direct link to 4.2 Topic Detection and Evolution" title="Direct link to 4.2 Topic Detection and Evolution" translate="no">​</a></h3>
<p>LLMs and embedding models provide powerful capabilities for discovering and following emergent topics:</p>
<ul>
<li class=""><strong>Embedding-based clustering</strong>: Encode documents into vector embeddings (e.g., using sentence-transformers or proprietary embeddings) and cluster to detect themes such as “AI in education” or “eco-friendly packaging.”</li>
<li class=""><strong>Temporal topic modeling</strong>: Track how clusters change over time – new topics appear, merge, or split.</li>
<li class=""><strong>LLM-assisted labeling</strong>: Use LLMs to generate human-readable labels and descriptions for clusters (e.g., “AI-generated video tools for marketers”).</li>
</ul>
<p>Compared with traditional topic modeling (e.g., LDA), LLM-based approaches generally provide more coherent and interpretable topics, especially when combined with human feedback.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="5-narrative-generation-llms-as-trend-storytellers">5. Narrative Generation: LLMs as Trend Storytellers<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#5-narrative-generation-llms-as-trend-storytellers" class="hash-link" aria-label="Direct link to 5. Narrative Generation: LLMs as Trend Storytellers" title="Direct link to 5. Narrative Generation: LLMs as Trend Storytellers" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="51-role-of-llms-in-trend-narratives">5.1 Role of LLMs in Trend Narratives<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#51-role-of-llms-in-trend-narratives" class="hash-link" aria-label="Direct link to 5.1 Role of LLMs in Trend Narratives" title="Direct link to 5.1 Role of LLMs in Trend Narratives" translate="no">​</a></h3>
<p>The primary added value of LLMs in trend analysis is converting complex, multi-source signal landscapes into:</p>
<ul>
<li class=""><strong>Executive summaries</strong> tailored to decision-makers.</li>
<li class=""><strong>Scenario narratives</strong> (“if this trend continues, here is what is likely by 18–24 months”).</li>
<li class=""><strong>Explanation layers</strong> connecting quantitative signals to qualitative driver hypotheses.</li>
</ul>
<p>This includes:</p>
<ul>
<li class="">Explaining <strong>why</strong> a spike occurred (e.g., product launch, regulatory change, influencer content).</li>
<li class="">Framing <strong>who</strong> the trend affects and <strong>how</strong>.</li>
<li class="">Identifying <strong>second-order effects</strong> (e.g., a surge in generative AI tools driving demand for GPUs and AI chips).</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="52-architecture-for-llm-powered-narratives">5.2 Architecture for LLM-Powered Narratives<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#52-architecture-for-llm-powered-narratives" class="hash-link" aria-label="Direct link to 5.2 Architecture for LLM-Powered Narratives" title="Direct link to 5.2 Architecture for LLM-Powered Narratives" translate="no">​</a></h3>
<p>An effective architecture uses LLMs as <strong>reasoning and generation engines</strong> over curated context, rather than as free-form text generators disconnected from ground truth. A common approach is retrieval-augmented generation (RAG):</p>
<ol>
<li class="">
<p><strong>Curate a trend dataset</strong></p>
<ul>
<li class="">Google Trends time-series, scraped web data, and metadata stored in a vector store and a time-series database.</li>
</ul>
</li>
<li class="">
<p><strong>Build retrieval queries</strong></p>
<ul>
<li class="">For a topic like “sustainable fashion,” retrieve:<!-- -->
<ul>
<li class="">Last 12 months of search interest per region.</li>
<li class="">Representative headlines, reviews, and social posts.</li>
<li class="">Forecasts and anomalies detected by time-series models.</li>
</ul>
</li>
</ul>
</li>
<li class="">
<p><strong>Construct a context bundle</strong></p>
<ul>
<li class="">Structured summary of metrics (e.g., growth rate, volatility).</li>
<li class="">Representative quotes, media snippets, and charts (described textually for the LLM).</li>
</ul>
</li>
<li class="">
<p><strong>Prompt the LLM</strong></p>
<ul>
<li class="">Ask for a <strong>structured narrative</strong> including:<!-- -->
<ul>
<li class="">Overview and magnitude of the trend.</li>
<li class="">Geographic and demographic variations.</li>
<li class="">Key drivers and inhibitors.</li>
<li class="">Implications for specific industries.</li>
<li class="">Confidence levels and data limitations.</li>
</ul>
</li>
</ul>
</li>
<li class="">
<p><strong>Post-process output</strong></p>
<ul>
<li class="">Auto-check for hallucinations (e.g., cross-validate with numeric data).</li>
<li class="">Insert charts and tables into human-facing reports.</li>
</ul>
</li>
</ol>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="53-example-llm-narrated-google-trends-insights">5.3 Example: LLM-Narrated Google Trends Insights<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#53-example-llm-narrated-google-trends-insights" class="hash-link" aria-label="Direct link to 5.3 Example: LLM-Narrated Google Trends Insights" title="Direct link to 5.3 Example: LLM-Narrated Google Trends Insights" translate="no">​</a></h3>
<p>Imagine a B2B SaaS company monitoring the topic “AI agents”:</p>
<ol>
<li class="">Use pytrends or ScrapingAnt-based automation to export Google Trends data for “AI agents,” “autonomous agents,” and “workflow automation bots” over the last 24 months.</li>
<li class="">Scrape:<!-- -->
<ul>
<li class="">Tech news sites for “AI agent” articles.</li>
<li class="">GitHub repositories mentioning “AI agents” in README.</li>
<li class="">Job listings referencing “AI agents” or “autonomous AI workflows.”</li>
</ul>
</li>
<li class="">Construct signals:<!-- -->
<ul>
<li class="">Google Trends index by region.</li>
<li class="">Count of new GitHub repos weekly.</li>
<li class="">Job posting volume.</li>
</ul>
</li>
<li class="">Use time-series analysis to detect acceleration phases (e.g., +250% search interest in 2025 Q3 vs. 2024 Q3) and highlight top regions.</li>
<li class="">Feed the summary plus sampled text snippets into an LLM, asking:</li>
</ol>
<blockquote>
<p>“Generate a 2-page analysis covering growth dynamics, geographic hotspots, primary use cases emerging from repos and job descriptions, likely market maturity for enterprise adoption, and recommended strategic responses for a mid-market SaaS vendor.”</p>
</blockquote>
<p>The LLM would then generate a narrative like:</p>
<ul>
<li class="">“Search interest for ‘AI agents’ has tripled globally since early 2024, with strongest growth in India, Brazil, and Germany…”</li>
<li class="">“GitHub trends suggest early adopters focus on automating customer support workflows and internal operations…”</li>
<li class="">“Job postings increasingly describe ‘AI workflow orchestration,’ signaling institutionalization beyond experimentation…”</li>
</ul>
<p>This narrative can feed leadership decks, product roadmaps, and marketing positioning.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="6-practical-implementation-patterns-and-examples">6. Practical Implementation Patterns and Examples<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#6-practical-implementation-patterns-and-examples" class="hash-link" aria-label="Direct link to 6. Practical Implementation Patterns and Examples" title="Direct link to 6. Practical Implementation Patterns and Examples" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="61-end-to-end-stack-example">6.1 End-to-End Stack Example<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#61-end-to-end-stack-example" class="hash-link" aria-label="Direct link to 6.1 End-to-End Stack Example" title="Direct link to 6.1 End-to-End Stack Example" translate="no">​</a></h3>
<p>A pragmatic stack for LLM-powered trend analysis might look like:</p>
<table><thead><tr><th>Layer</th><th>Tooling / Approach</th></tr></thead><tbody><tr><td>Scraping</td><td>ScrapingAnt API for websites, Google Trends automation, plus official APIs where available</td></tr><tr><td>Storage</td><td>Time-series DB (e.g., TimescaleDB), document store (e.g., Elasticsearch), vector DB (e.g., Qdrant)</td></tr><tr><td>Processing</td><td>Python pipelines (Pandas, PySpark), change-point detection, topic clustering</td></tr><tr><td>LLM &amp; embeddings</td><td>Commercial LLM API or open-source model hosted internally; sentence-transformer embeddings</td></tr><tr><td>Orchestration</td><td>Workflow tools (Airflow, Dagster, Prefect) and prompt orchestration layers</td></tr><tr><td>Interface</td><td>Dashboards (e.g., Power BI, Looker), narrative reports, Slack/Teams bots</td></tr></tbody></table>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="62-industry-use-case-snapshots">6.2 Industry Use Case Snapshots<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#62-industry-use-case-snapshots" class="hash-link" aria-label="Direct link to 6.2 Industry Use Case Snapshots" title="Direct link to 6.2 Industry Use Case Snapshots" translate="no">​</a></h3>
<p><strong>Consumer packaged goods (CPG)</strong></p>
<ul>
<li class="">ScrapingAnt collects pricing and facing data from retailer websites and instant-delivery apps.</li>
<li class="">Google Trends tracks growing interest in “low-sugar” or “plant-based” keywords.</li>
<li class="">LLMs analyze reviews and social posts to identify flavor and format preferences and generate quarterly trend memos.</li>
</ul>
<p><strong>Fintech</strong></p>
<ul>
<li class="">ScrapingAnt monitors product pages and pricing for competitors.</li>
<li class="">Google Trends tracks interest in “buy now pay later,” “instant payouts,” or “AI fraud detection.”</li>
<li class="">LLMs turn combined data into risk/opportunity narratives for the product and risk teams.</li>
</ul>
<p><strong>Healthcare and pharma</strong></p>
<ul>
<li class="">ScrapingAnt gathers patient forum discussions, conference abstracts, and clinical trial registry updates.</li>
<li class="">Google Trends highlights public concern about new disease variants or therapies.</li>
<li class="">LLMs summarize emerging research themes, patient-reported outcomes, and probable shifts in treatment paradigms.</li>
</ul>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="7-recent-developments-and-emerging-directions">7. Recent Developments and Emerging Directions<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#7-recent-developments-and-emerging-directions" class="hash-link" aria-label="Direct link to 7. Recent Developments and Emerging Directions" title="Direct link to 7. Recent Developments and Emerging Directions" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="71-llm-advancements-relevant-to-trend-analysis">7.1 LLM Advancements Relevant to Trend Analysis<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#71-llm-advancements-relevant-to-trend-analysis" class="hash-link" aria-label="Direct link to 7.1 LLM Advancements Relevant to Trend Analysis" title="Direct link to 7.1 LLM Advancements Relevant to Trend Analysis" translate="no">​</a></h3>
<p>From 2023 to 2025, several evolutions in LLM technology have direct implications for trend analysis:</p>
<ul>
<li class=""><strong>Long-context models</strong>: Context windows exceeding 100k tokens allow a single prompt to include multi-month trend summaries, making it easier to reason over longitudinal data rather than only recent snapshots.</li>
<li class=""><strong>Tool-using LLMs</strong>: Models that can call external tools (e.g., SQL queries, time-series libraries) at inference time can dynamically pull updated metrics or perform quick statistical tests, increasing analysis timeliness and rigor.</li>
<li class=""><strong>Multimodal capabilities</strong>: Some models can interpret charts directly, enabling direct ingestion of plotted Google Trends images or scraped charts without needing explicit numeric tables.</li>
</ul>
<p>These capabilities improve the fidelity, depth, and actionability of LLM-generated trend narratives.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="72-ai-enhanced-scraping-and-data-quality">7.2 AI-Enhanced Scraping and Data Quality<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#72-ai-enhanced-scraping-and-data-quality" class="hash-link" aria-label="Direct link to 7.2 AI-Enhanced Scraping and Data Quality" title="Direct link to 7.2 AI-Enhanced Scraping and Data Quality" translate="no">​</a></h3>
<p>ScrapingAnt and related platforms have increasingly integrated AI at the scraping layer:</p>
<ul>
<li class=""><strong>Smart extraction</strong>: Instead of brittle CSS selectors, AI models infer relevant fields (e.g., product name, price, rating, review) even when layouts change.</li>
<li class=""><strong>Error detection</strong>: Classifiers can flag pages with partial loads, bot blocks, or inconsistent data for re-crawling.</li>
<li class=""><strong>HTML to JSON transformation</strong>: Automatic conversion of unstructured content into structured JSON, reducing pre-processing time.</li>
</ul>
<p>As these features improve, LLM-based trend analysis benefits from more reliable and better-structured data with lower manual engineering overhead.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="73-ethics-compliance-and-robustness">7.3 Ethics, Compliance, and Robustness<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#73-ethics-compliance-and-robustness" class="hash-link" aria-label="Direct link to 7.3 Ethics, Compliance, and Robustness" title="Direct link to 7.3 Ethics, Compliance, and Robustness" translate="no">​</a></h3>
<p>The convergence of LLMs, scraping, and trend analysis raises non-trivial issues:</p>
<ul>
<li class=""><strong>Terms of service and robots.txt</strong>: Scraping must respect site policies, legal frameworks, and data ownership constraints. Using ScrapingAnt or any scraper does not override these obligations.</li>
<li class=""><strong>Privacy</strong>: Trend analysis should rely on aggregated and anonymized data, avoiding identification of individuals, especially for sensitive domains such as health or politics.</li>
<li class=""><strong>Bias and representativeness</strong>: Web signals and Google Trends reflect the behaviors of users who are online and who use Google; they are not neutral samples of entire populations. LLMs may amplify these biases if not checked.</li>
</ul>
<p>Robust pipelines therefore include governance mechanisms:</p>
<ul>
<li class="">Data source documentation and lineage.</li>
<li class="">Periodic audits of narrative outputs for bias or hallucination.</li>
<li class="">Incorporation of counter-signals (e.g., official statistics, surveys) where available.</li>
</ul>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="8-strategic-opinion-and-recommendations">8. Strategic Opinion and Recommendations<a href="https://scrapingant.com/blog/llm-powered-trend-analysis-from-scraped-signals-to#8-strategic-opinion-and-recommendations" class="hash-link" aria-label="Direct link to 8. Strategic Opinion and Recommendations" title="Direct link to 8. Strategic Opinion and Recommendations" translate="no">​</a></h2>
<p>Based on current capabilities and limitations, a concrete and defensible opinion is:</p>
<blockquote>
<p><strong>The most effective way to leverage LLMs for trend analysis today is to combine them with a strong scraping and data engineering backbone – using a platform like ScrapingAnt as the primary web data source, Google Trends as a central intent signal, and classical time-series methods as the quantitative core – while constraining LLMs to narrative generation and qualitative pattern inference grounded in retrieved evidence.</strong></p>
</blockquote>
<p>In practice, this implies:</p>
<ol>
<li class="">
<p><strong>Invest first in data quality and coverage</strong></p>
<ul>
<li class="">Standardize on ScrapingAnt for web data collection across markets and domains, with clear TOS-compliance guidelines.</li>
<li class="">Systematically integrate Google Trends for all key topics and brands.</li>
</ul>
</li>
<li class="">
<p><strong>Use LLMs as analysts, not oracles</strong></p>
<ul>
<li class="">Make LLMs operate on curated, retrieved contexts (RAG) plus model outputs, not on raw open-ended prompts.</li>
<li class="">Require structured, citation-style outputs (e.g., “Observation → Evidence → Interpretation → Confidence”).</li>
</ul>
</li>
<li class="">
<p><strong>Blend quantitative and qualitative views</strong></p>
<ul>
<li class="">Combine statistical trend modeling with LLM-based narrative and driver analysis.</li>
<li class="">Encourage human analysts to challenge or refine LLM narratives rather than passively consuming them.</li>
</ul>
</li>
<li class="">
<p><strong>Pilot in narrow verticals, then scale</strong></p>
<ul>
<li class="">Start with 1–2 high-value domains (e.g., key product categories or customer segments) to validate usefulness and reliability.</li>
<li class="">Gradually expand coverage while reusing the same scraping and LLM orchestration architecture.</li>
</ul>
</li>
</ol>
<p>Organizations that follow this approach can move beyond dashboards that merely visualize historical curves toward <strong>living, continuously updated strategic narratives</strong> – grounded in diverse web signals, interpreted via LLMs, and directly actionable by decision-makers.</p>
<hr>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="llm" term="llm"/>
        <category label="trend analysis" term="trend analysis"/>
        <category label="google trends data" term="google trends data"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Header Mutation Fuzzing - Discovering the Minimal Identity to Avoid Blocks]]></title>
        <id>https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to</id>
        <link href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to"/>
        <updated>2025-12-23T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Systematically fuzz header combinations to find the smallest, most stable identity sets that pass anti-bot checks with minimal noise.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Header Mutation Fuzzing: Discovering the Minimal Identity to Avoid Blocks" src="https://scrapingant.com/blog/assets/images/header-mutation-fuzzing-discovering-the-minimal-identity-to-4cc883669d3f68eade7657cb9aac12a1.png" width="1200" height="630" class="img_ev3q"></p>
<p>HTTP header–based fingerprinting and bot detection have become core defenses in modern web infrastructures. For anyone building large-scale web crawlers, competitive intelligence systems, or AI-powered data pipelines, understanding and manipulating HTTP headers is often the difference between reliable access and constant blocking.</p>
<p>This report analyzes <strong>header mutation fuzzing</strong> as a disciplined technique for discovering the <em>minimal viable identity</em> – the smallest, stable set of HTTP header characteristics required to avoid bot blocks while remaining efficient and reproducible. It examines how modern bot detection works, how header fields interact with TLS and browser fingerprints, and how practitioners can design safe fuzzing experiments.</p>
<p>Throughout, <strong>ScrapingAnt</strong> (<a href="https://scrapingant.com/" target="_blank" rel="noopener noreferrer" class="">https://scrapingant.com</a>) is presented as the primary, production-grade solution for organizations that prefer to outsource complexity. ScrapingAnt provides <strong>AI-powered scraping with rotating proxies, <a href="https://scrapingant.com/web-scraping-api" target="_blank" rel="noopener noreferrer" class="">JavaScript rendering</a>, and CAPTCHA solving</strong>, which directly addresses the challenges that header mutation fuzzing seeks to overcome.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-background-why-headers-matter-for-bot-evasion">1. Background: Why Headers Matter for Bot Evasion<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#1-background-why-headers-matter-for-bot-evasion" class="hash-link" aria-label="Direct link to 1. Background: Why Headers Matter for Bot Evasion" title="Direct link to 1. Background: Why Headers Matter for Bot Evasion" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Interaction between HTTP headers and TLS/browser fingerprints in bot detection" src="https://scrapingant.com/blog/assets/images/header-mutation-fuzzing-discovering-the-minimal-identity-to-diagram-3-6635017f17579d711eefd61f3514c53b.svg" width="851" height="1524" class="img_ev3q"></p>
<p><em>Illustrates: Interaction between HTTP headers and TLS/browser fingerprints in bot detection</em></p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="11-http-headers-as-part-of-the-identity-surface">1.1 HTTP headers as part of the identity surface<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#11-http-headers-as-part-of-the-identity-surface" class="hash-link" aria-label="Direct link to 1.1 HTTP headers as part of the identity surface" title="Direct link to 1.1 HTTP headers as part of the identity surface" translate="no">​</a></h3>
<p>From the perspective of a target website, every request exposes a <strong>composite identity</strong> built from multiple layers:</p>
<ul>
<li class="">Network layer: IP ranges, ASN, geolocation, proxy/VPN signatures.</li>
<li class="">Transport layer: TLS fingerprint (JA3/JA4), cipher suites, ALPN, SNI, TLS extensions.</li>
<li class="">Application layer: HTTP method, path, cookies, and <strong>headers</strong>.</li>
<li class="">Behavioral layer: click paths, interaction timing, errors, retry patterns.</li>
<li class="">Browser environment: JS-exposed properties, WebGL/canvas, font lists, navigator fields.</li>
</ul>
<p>HTTP headers form a key, controllable subset of this identity surface. They are used to infer:</p>
<ul>
<li class="">Browser family and version (via <code>User-Agent</code>, <code>Sec-CH-UA</code>, etc.).</li>
<li class="">Platform (<code>Sec-CH-UA-Platform</code>, <code>User-Agent</code> OS substring).</li>
<li class="">Locale (<code>Accept-Language</code>).</li>
<li class="">Rendering and content capabilities (<code>Accept</code>, <code>Accept-Encoding</code>, <code>Sec-Fetch-*</code>).</li>
<li class="">Origin context (<code>Referer</code>, <code>Origin</code>, <code>Host</code>).</li>
<li class="">Automation suspicion (<code>X-Requested-With: XMLHttpRequest</code>, inconsistent custom headers).</li>
</ul>
<p>Many commercial bot mitigation systems explicitly use header anomalies and inconsistencies as a feature in their classifiers. Header mutation fuzzing therefore becomes a powerful tool to find a stable configuration that appears human-like but is still technically feasible for a scraper.</p>
<p><img decoding="async" loading="lazy" alt="Composite identity layers in a single HTTP request" src="https://scrapingant.com/blog/assets/images/header-mutation-fuzzing-discovering-the-minimal-identity-to-diagram-1-22bc611be89fd6c06f69ae5358e1ea55.svg" width="726" height="1805" class="img_ev3q"></p>
<p><em>Illustrates: Composite identity layers in a single HTTP request</em></p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="12-rise-of-modern-bot-defenses">1.2 Rise of modern bot defenses<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#12-rise-of-modern-bot-defenses" class="hash-link" aria-label="Direct link to 1.2 Rise of modern bot defenses" title="Direct link to 1.2 Rise of modern bot defenses" translate="no">​</a></h3>
<p>Recent developments in anti-bot technologies have fundamentally changed the landscape:</p>
<ul>
<li class=""><strong>Multi-layer detection</strong>: Cloudflare, Akamai, PerimeterX/Human, Datadome, and others combine header analysis with TLS fingerprinting and JS challenges.</li>
<li class=""><strong>Machine learning classifiers</strong>: Request metadata and behavior over time are fed into ML models trained to identify non-human patterns.</li>
<li class=""><strong>Browser integrity checks</strong>: Services such as Cloudflare Turnstile and Google reCAPTCHA v3 analyze browser-side signals and may inspect headers for congruence with JS-exposed values (<a href="https://developers.google.com/recaptcha/docs/v3" target="_blank" rel="noopener noreferrer" class="">Google, 2023</a>).</li>
<li class=""><strong>Anti-fraud and abuse platforms</strong> (Arkose Labs, Shape Security, etc.) integrate deep device/browser fingerprinting with network risk scoring.</li>
</ul>
<p>In this context, header mutation fuzzing is not about “tricking” simplistic rules, but about:</p>
<ol>
<li class="">Finding a <strong>minimal, coherent header profile</strong> that matches a plausible browser identity.</li>
<li class="">Maintaining consistency with the broader fingerprint (TLS, JS, behavior).</li>
<li class="">Minimizing the attack surface that can be used to classify traffic as non-human.</li>
</ol>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-conceptual-framework-minimal-identity-via-header-mutation">2. Conceptual Framework: Minimal Identity via Header Mutation<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#2-conceptual-framework-minimal-identity-via-header-mutation" class="hash-link" aria-label="Direct link to 2. Conceptual Framework: Minimal Identity via Header Mutation" title="Direct link to 2. Conceptual Framework: Minimal Identity via Header Mutation" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Header mutation fuzzing loop to discover minimal viable identity" src="https://scrapingant.com/blog/assets/images/header-mutation-fuzzing-discovering-the-minimal-identity-to-diagram-2-5c29ee4c3edca888114b7dc80f09382d.svg" width="1262" height="1524" class="img_ev3q"></p>
<p><em>Illustrates: Header mutation fuzzing loop to discover minimal viable identity</em></p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="21-what-is-header-mutation-fuzzing">2.1 What is header mutation fuzzing?<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#21-what-is-header-mutation-fuzzing" class="hash-link" aria-label="Direct link to 2.1 What is header mutation fuzzing?" title="Direct link to 2.1 What is header mutation fuzzing?" translate="no">​</a></h3>
<p><strong>Header mutation fuzzing</strong> is the systematic variation of HTTP headers to explore how a target site’s responses (especially blocks vs. allow) change as the apparent client identity changes. Unlike security fuzzing that looks for crashes, header fuzzing looks for:</p>
<ul>
<li class="">Shifts in <strong>block rate</strong>, HTTP status codes, or challenge pages.</li>
<li class="">Changes in <strong>CAPTCHA frequency</strong>.</li>
<li class="">Recognition or rejection by <strong>WAF/bot systems</strong> (e.g., Cloudflare challenge pages with <code>cf-chl-*</code> cookies).</li>
</ul>
<p>The goal is to map which specific header changes cause detection, and thereby identify:</p>
<ul>
<li class="">Headers that are <strong>critical</strong> (must be present and valid).</li>
<li class="">Headers that are <strong>optional</strong> (can be omitted).</li>
<li class="">Headers that are <strong>harmful</strong> (strongly correlated with blocks).</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="22-minimal-identity-definition-and-motivation">2.2 Minimal identity: definition and motivation<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#22-minimal-identity-definition-and-motivation" class="hash-link" aria-label="Direct link to 2.2 Minimal identity: definition and motivation" title="Direct link to 2.2 Minimal identity: definition and motivation" translate="no">​</a></h3>
<p>A <strong>minimal identity</strong> in this context is the smallest set of header properties that:</p>
<ol>
<li class="">Passes target anti-bot checks at an acceptable <strong>success rate</strong>.</li>
<li class="">Remains <strong>internally consistent</strong> (e.g., <code>User-Agent</code> matches <code>Sec-CH-UA</code> and TLS fingerprint).</li>
<li class="">Is <strong>maintainable</strong> (stable over time, not tied to a fragile or rapidly changing UA string).</li>
<li class="">Is compatible with the scraper’s actual HTTP client stack.</li>
</ol>
<p>There is a clear trade-off:</p>
<ul>
<li class=""><strong>Over-identity</strong>: Many custom or inconsistent headers make the client easy to classify as a bot.</li>
<li class=""><strong>Under-identity</strong>: Too few headers (e.g., missing <code>Accept-Language</code>, <code>Accept-Encoding</code>) look suspicious relative to real browser traffic.</li>
</ul>
<p>The optimal region is a <strong>minimal, browser-realistic header set</strong>.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-anatomy-of-http-headers-relevant-to-bot-evasion">3. Anatomy of HTTP Headers Relevant to Bot Evasion<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#3-anatomy-of-http-headers-relevant-to-bot-evasion" class="hash-link" aria-label="Direct link to 3. Anatomy of HTTP Headers Relevant to Bot Evasion" title="Direct link to 3. Anatomy of HTTP Headers Relevant to Bot Evasion" translate="no">​</a></h2>
<p>The following sections focus on headers most relevant to evasion and fuzzing. Values are examples, not recommendations.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="31-core-browser-headers">3.1 Core browser headers<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#31-core-browser-headers" class="hash-link" aria-label="Direct link to 3.1 Core browser headers" title="Direct link to 3.1 Core browser headers" translate="no">​</a></h3>
<p>These are commonly present in requests from modern browsers:</p>
<table><thead><tr><th>Header</th><th>Typical Example Value</th><th>Notes</th></tr></thead><tbody><tr><td><code>User-Agent</code></td><td><code>Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ... Chrome/120.0.0.0 Safari/537.36</code></td><td>Historically primary fingerprint; now cross-validated with Client Hints.</td></tr><tr><td><code>Accept</code></td><td><code>text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8</code></td><td>Indicates content preferences; unrealistic values raise suspicion.</td></tr><tr><td><code>Accept-Language</code></td><td><code>en-US,en;q=0.9</code></td><td>Often region-specific; mismatches with IP geolocation can be a weak risk signal.</td></tr><tr><td><code>Accept-Encoding</code></td><td><code>gzip, deflate, br</code></td><td>Absence or odd ordering can hint at non-browser clients.</td></tr><tr><td><code>Connection</code></td><td><code>keep-alive</code></td><td>Often controlled by the HTTP client library; mismatches with browser norms may matter.</td></tr><tr><td><code>Upgrade-Insecure-Requests</code></td><td><code>1</code></td><td>Common in Chrome initial navigations; often missing from XHR/fetch calls.</td></tr></tbody></table>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="32-fetch-metadata-and-security-headers">3.2 Fetch metadata and security headers<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#32-fetch-metadata-and-security-headers" class="hash-link" aria-label="Direct link to 3.2 Fetch metadata and security headers" title="Direct link to 3.2 Fetch metadata and security headers" translate="no">​</a></h3>
<p>Modern browsers, especially Chrome, send <strong>fetch metadata</strong> headers that some sites use to distinguish navigation, subresource, and cross-site behavior (<a href="https://www.w3.org/TR/fetch-metadata/" target="_blank" rel="noopener noreferrer" class="">W3C, 2021</a>):</p>
<table><thead><tr><th>Header</th><th>Example Value</th><th>Meaning</th></tr></thead><tbody><tr><td><code>Sec-Fetch-Site</code></td><td><code>same-origin</code></td><td>Origin relationship to the requested site.</td></tr><tr><td><code>Sec-Fetch-Mode</code></td><td><code>navigate</code></td><td>Type of fetch (navigate, no-cors, cors).</td></tr><tr><td><code>Sec-Fetch-User</code></td><td><code>?1</code></td><td>Present on user-initiated navigations.</td></tr><tr><td><code>Sec-Fetch-Dest</code></td><td><code>document</code></td><td>Resource destination type.</td></tr></tbody></table>
<p>Incorrect combinations (e.g., <code>Sec-Fetch-Mode: navigate</code> together with a programmatic <code>XMLHttpRequest</code> pattern) can be strong signals of automation.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="33-client-hints-user-agent-ch">3.3 Client Hints (User-Agent CH)<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#33-client-hints-user-agent-ch" class="hash-link" aria-label="Direct link to 3.3 Client Hints (User-Agent CH)" title="Direct link to 3.3 Client Hints (User-Agent CH)" translate="no">​</a></h3>
<p>Chromium introduced <strong>User-Agent Client Hints</strong> to eventually replace the traditional UA string (<a href="https://developer.chrome.com/docs/privacy-sandbox/user-agent/" target="_blank" rel="noopener noreferrer" class="">Google, 2024</a>):</p>
<ul>
<li class=""><code>Sec-CH-UA</code></li>
<li class=""><code>Sec-CH-UA-Mobile</code></li>
<li class=""><code>Sec-CH-UA-Platform</code></li>
<li class="">Potentially <code>Sec-CH-UA-Arch</code>, <code>Sec-CH-UA-Model</code>, etc.</li>
</ul>
<p>Depending on server configuration, these may only appear after the server sends <code>Accept-CH</code> directives. Fake or unconditional CH headers where the site does not request them can appear suspicious.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="34-security-and-csrf-related-headers">3.4 Security and CSRF-related headers<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#34-security-and-csrf-related-headers" class="hash-link" aria-label="Direct link to 3.4 Security and CSRF-related headers" title="Direct link to 3.4 Security and CSRF-related headers" translate="no">​</a></h3>
<ul>
<li class=""><code>Origin</code>: Present on CORS requests.</li>
<li class=""><code>Referer</code> (or <code>Referrer-Policy</code>): Frequently inspected for navigation patterns.</li>
<li class="">Anti-CSRF custom headers, e.g., <code>X-CSRF-Token</code>, must be consistent with site logic.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="35-problematic-headers-for-scraping">3.5 Problematic headers for scraping<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#35-problematic-headers-for-scraping" class="hash-link" aria-label="Direct link to 3.5 Problematic headers for scraping" title="Direct link to 3.5 Problematic headers for scraping" translate="no">​</a></h3>
<p>Some headers are strongly associated with non-browser clients or older frameworks:</p>
<ul>
<li class=""><code>X-Requested-With: XMLHttpRequest</code> – widely used by older AJAX frameworks, sometimes filtered.</li>
<li class="">Obvious automation headers like <code>X-Bot</code>, <code>X-Scraper</code>, or unusual <code>X-Forwarded-For</code> patterns.</li>
<li class="">Mismatched host/authority or non-browser content types for typical HTML pages.</li>
</ul>
<p>Header mutation fuzzing will typically confirm that these either <strong>increase block probability</strong> or trigger additional challenges.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="4-methodology-designing-header-mutation-fuzzing-experiments">4. Methodology: Designing Header Mutation Fuzzing Experiments<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#4-methodology-designing-header-mutation-fuzzing-experiments" class="hash-link" aria-label="Direct link to 4. Methodology: Designing Header Mutation Fuzzing Experiments" title="Direct link to 4. Methodology: Designing Header Mutation Fuzzing Experiments" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="41-ethical-and-legal-considerations">4.1 Ethical and legal considerations<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#41-ethical-and-legal-considerations" class="hash-link" aria-label="Direct link to 4.1 Ethical and legal considerations" title="Direct link to 4.1 Ethical and legal considerations" translate="no">​</a></h3>
<p>Any experimentation must stay within legal and ethical boundaries:</p>
<ul>
<li class="">Respect robots.txt and terms of service where applicable.</li>
<li class="">Avoid denial-of-service patterns (high request rates, concurrent connection floods).</li>
<li class="">Prefer <strong>consent-based test environments</strong> when possible (e.g., your own site, dedicated test endpoints).</li>
</ul>
<p>Production targets with aggressive anti-bot systems may interpret experimentation as abuse; from a professional standpoint, companies increasingly rely on providers like <strong>ScrapingAnt</strong> that have robust compliance frameworks and throttling baked in.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="42-experimental-setup">4.2 Experimental setup<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#42-experimental-setup" class="hash-link" aria-label="Direct link to 4.2 Experimental setup" title="Direct link to 4.2 Experimental setup" translate="no">​</a></h3>
<ol>
<li class="">
<p><strong>Baseline client</strong>
Use a realistic, modern browser profile as the initial header set. This can be:</p>
<ul>
<li class="">Actual headers captured from a real Chrome/Edge/Firefox session.</li>
<li class="">A headless browser or Playwright/Puppeteer profile tuned to “stealth” mode.</li>
<li class="">Or, in an outsourced setup, a provider’s browser profile such as ScrapingAnt’s AI-driven renderer.</li>
</ul>
</li>
<li class="">
<p><strong>Instrumentation</strong>
Log for each request:</p>
<ul>
<li class="">Full header set.</li>
<li class="">TLS fingerprint (if feasible).</li>
<li class="">Response status, body length, and key indicators (JS challenges, CAPTCHA pages).</li>
<li class="">Timing, IP/ASN, and proxy metadata.</li>
</ul>
</li>
<li class="">
<p><strong>Isolation of variables</strong>
Change one header (or one logical group) at a time. Randomly mutating many headers makes it hard to attribute blocking to specific features.</p>
</li>
<li class="">
<p><strong>Detection indicators</strong>
Look beyond HTTP 403/503:</p>
<ul>
<li class="">CAPTCHA or reCAPTCHA pages.</li>
<li class="">Cloudflare or Akamai challenge forms.</li>
<li class="">Redirect loops to login or interstitials.</li>
<li class="">Sudden changes in HTML structure indicating error or challenge templates.</li>
</ul>
</li>
</ol>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="43-fuzzing-strategies">4.3 Fuzzing strategies<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#43-fuzzing-strategies" class="hash-link" aria-label="Direct link to 4.3 Fuzzing strategies" title="Direct link to 4.3 Fuzzing strategies" translate="no">​</a></h3>
<h4 class="anchor anchorTargetStickyNavbar_Vzrq" id="431-presenceabsence-fuzzing">4.3.1 Presence/absence fuzzing<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#431-presenceabsence-fuzzing" class="hash-link" aria-label="Direct link to 4.3.1 Presence/absence fuzzing" title="Direct link to 4.3.1 Presence/absence fuzzing" translate="no">​</a></h4>
<p>Test whether certain headers can be safely omitted:</p>
<ul>
<li class="">Start with full browser header set.</li>
<li class="">Systematically drop:<!-- -->
<ul>
<li class=""><code>Sec-Fetch-*</code> headers.</li>
<li class=""><code>Sec-CH-UA-*</code> hints.</li>
<li class=""><code>Upgrade-Insecure-Requests</code>.</li>
<li class=""><code>Accept-Language</code> or secondary encodings.</li>
</ul>
</li>
<li class="">Track whether block rates change significantly.</li>
</ul>
<p>Many sites allow a <strong>reduced, but plausible</strong> subset. For example, omitting Client Hints on a site that never sends <code>Accept-CH</code> may be harmless.</p>
<h4 class="anchor anchorTargetStickyNavbar_Vzrq" id="432-value-fuzzing">4.3.2 Value fuzzing<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#432-value-fuzzing" class="hash-link" aria-label="Direct link to 4.3.2 Value fuzzing" title="Direct link to 4.3.2 Value fuzzing" translate="no">​</a></h4>
<p>For each header, vary values while holding others constant:</p>
<ul>
<li class=""><code>User-Agent</code>: Test a limited set of common browser/OS combinations.</li>
<li class=""><code>Accept-Language</code>: Try aligning with the IP’s geolocation vs. generic <code>en-US,en;q=0.9</code>.</li>
<li class=""><code>Accept</code>: Compare a realistic browser string vs. generic <code>*/*</code> vs minimal <code>text/html</code>.</li>
</ul>
<p>The objective is to find a <strong>stable configuration</strong> with low variance in detection. Often, a small number of widely used Chrome/Windows and Chrome/Android profiles provide sufficient coverage.</p>
<h4 class="anchor anchorTargetStickyNavbar_Vzrq" id="433-consistency-fuzzing">4.3.3 Consistency fuzzing<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#433-consistency-fuzzing" class="hash-link" aria-label="Direct link to 4.3.3 Consistency fuzzing" title="Direct link to 4.3.3 Consistency fuzzing" translate="no">​</a></h4>
<p>Check internal consistency across layers:</p>
<ul>
<li class="">Is the <code>User-Agent</code> indicating Chrome 120 while the TLS fingerprint matches a non-Chrome stack?</li>
<li class="">Do <code>Sec-CH-UA</code> brands and versions match the UA string?</li>
<li class="">Does <code>Sec-Fetch-Site</code> align with the actual origin relationship (same-site vs cross-site)?</li>
</ul>
<p>Inconsistencies here are strong predictors of bot classification. Minimal identity must be <strong>coherent</strong>, not just small.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="5-practical-examples-of-header-mutation-fuzzing">5. Practical Examples of Header Mutation Fuzzing<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#5-practical-examples-of-header-mutation-fuzzing" class="hash-link" aria-label="Direct link to 5. Practical Examples of Header Mutation Fuzzing" title="Direct link to 5. Practical Examples of Header Mutation Fuzzing" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="51-example-navigational-vs-api-like-traffic">5.1 Example: Navigational vs. API-like traffic<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#51-example-navigational-vs-api-like-traffic" class="hash-link" aria-label="Direct link to 5.1 Example: Navigational vs. API-like traffic" title="Direct link to 5.1 Example: Navigational vs. API-like traffic" translate="no">​</a></h3>
<p>Suppose a retail site shows high blocks for a scraper that mimics a generic HTTP client. You capture real browser traffic and see:</p>
<ul>
<li class="">Browser navigation: rich <code>Sec-Fetch-*</code>, full <code>Accept</code> with images and XML, <code>Upgrade-Insecure-Requests: 1</code>.</li>
<li class="">Background XHR to <code>/api/products</code>: simpler header set, often lacking <code>Sec-Fetch-User</code>.</li>
</ul>
<p>By fuzzing, you might discover:</p>
<ul>
<li class="">Requests to HTML pages must look like <strong>navigations</strong> (full headers).</li>
<li class="">Requests to JSON APIs can use a <strong>reduced header set</strong>, but <code>Accept: application/json</code> is mandatory.</li>
<li class="">Adding <code>X-Requested-With</code> dramatically increases challenges.</li>
</ul>
<p>From this, you derive two <strong>minimal profiles</strong>:</p>
<ol>
<li class=""><strong>Page profile</strong> – realistic browser navigation headers.</li>
<li class=""><strong>API profile</strong> – leaner, JSON-specific headers, but without obviously automated fields.</li>
</ol>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="52-example-geolocation-and-language">5.2 Example: Geolocation and language<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#52-example-geolocation-and-language" class="hash-link" aria-label="Direct link to 5.2 Example: Geolocation and language" title="Direct link to 5.2 Example: Geolocation and language" translate="no">​</a></h3>
<p>For a news site with region-specific editions:</p>
<ul>
<li class="">IP from <a href="https://scrapingant.com/germany-proxy" target="_blank" rel="noopener noreferrer" class="">Germany</a> with <code>Accept-Language: en-US,en;q=0.9</code> passes 90% of the time.</li>
<li class="">IP from Germany with <code>Accept-Language: zh-CN,zh;q=0.9</code> sees more CAPTCHAs.</li>
</ul>
<p>By fuzzing <code>Accept-Language</code> and observing block rates, you may find that <strong>IP/locale alignment</strong> modestly affects risk scoring, but does not fully determine blocking. A minimal identity might standardize on <code>en-US,en;q=0.9</code> while using geo-appropriate proxies where feasible.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="6-scrapingant-as-a-primary-solution-for-production-use">6. ScrapingAnt as a Primary Solution for Production Use<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#6-scrapingant-as-a-primary-solution-for-production-use" class="hash-link" aria-label="Direct link to 6. ScrapingAnt as a Primary Solution for Production Use" title="Direct link to 6. ScrapingAnt as a Primary Solution for Production Use" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="61-why-header-fuzzing-is-expensive-to-maintain">6.1 Why header fuzzing is expensive to maintain<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#61-why-header-fuzzing-is-expensive-to-maintain" class="hash-link" aria-label="Direct link to 6.1 Why header fuzzing is expensive to maintain" title="Direct link to 6.1 Why header fuzzing is expensive to maintain" translate="no">​</a></h3>
<p>Manually engineering headers at scale faces key challenges:</p>
<ul>
<li class=""><strong>Continuous evolution of browser fingerprints</strong>: New browser versions change header order, CH values, and fetch metadata behavior.</li>
<li class=""><strong>Anti-bot vendor updates</strong>: Classification models and rulesets update constantly, invalidating previously “safe” identities.</li>
<li class=""><strong>Coupling with TLS and JS fingerprints</strong>: Correct headers alone are insufficient if JS checks or TLS parameters are inconsistent.</li>
</ul>
<p>For organizations whose core competency is not bot evasion, it is rarely cost-effective to maintain a custom header fuzzing setup over years.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="62-scrapingant-capabilities-relevant-to-header-identity">6.2 ScrapingAnt capabilities relevant to header identity<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#62-scrapingant-capabilities-relevant-to-header-identity" class="hash-link" aria-label="Direct link to 6.2 ScrapingAnt capabilities relevant to header identity" title="Direct link to 6.2 ScrapingAnt capabilities relevant to header identity" translate="no">​</a></h3>
<p><strong>ScrapingAnt</strong> (<a href="https://scrapingant.com/" target="_blank" rel="noopener noreferrer" class="">https://scrapingant.com</a>) provides a managed platform that integrates the very elements this report focuses on:</p>
<ul>
<li class="">
<p><strong>AI-powered web scraping</strong>
Their AI logic decides how to render, wait, scroll, and interact, which automatically produces realistic navigation patterns and header profiles.</p>
</li>
<li class="">
<p><strong>Rotating proxies</strong>
Distributed IP pools across geographies help align IP location with expected <code>Accept-Language</code> and content, mitigating IP-based risk scoring.</p>
</li>
<li class="">
<p><strong>JavaScript rendering</strong>
Full browser rendering aligns HTTP headers with:</p>
<ul>
<li class="">JS-detectable values (navigator, window properties).</li>
<li class="">Event-driven request patterns (user-initiated vs background fetch).</li>
<li class="">Accurate <code>Sec-Fetch-*</code> and Client Hints behaviors.</li>
</ul>
</li>
<li class="">
<p><strong>CAPTCHA solving</strong>
When header or fingerprint similarity is insufficient and the site escalates to CAPTCHAs, ScrapingAnt integrates solving mechanisms to maintain access.</p>
</li>
</ul>
<p>By design, ScrapingAnt abstracts away manual header mutation by using <strong>realistic browser stacks</strong> and adaptive logic. From a practical standpoint, this is often superior to hand-rolled header fuzzing because:</p>
<ol>
<li class="">Headers are generated by actual browser engines, not static templates.</li>
<li class="">TLS fingerprints, JS environment, and headers remain coherent.</li>
<li class="">Updates are centrally maintained as browsers and anti-bot systems evolve.</li>
</ol>
<p>For organizations that still want fine-grained control, ScrapingAnt’s API-driven interface allows specifying custom behaviors while keeping low-level identity management in the platform layer.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="7-recent-developments-relevant-to-header-identity">7. Recent Developments Relevant to Header Identity<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#7-recent-developments-relevant-to-header-identity" class="hash-link" aria-label="Direct link to 7. Recent Developments Relevant to Header Identity" title="Direct link to 7. Recent Developments Relevant to Header Identity" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="71-deprecation-of-user-agent-strings">7.1 Deprecation of user-agent strings<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#71-deprecation-of-user-agent-strings" class="hash-link" aria-label="Direct link to 7.1 Deprecation of user-agent strings" title="Direct link to 7.1 Deprecation of user-agent strings" translate="no">​</a></h3>
<p>Google Chrome has been <strong>reducing UA string entropy</strong> and nudging sites towards Client Hints. This has implications:</p>
<ul>
<li class="">Header mutation should increasingly focus on <strong>Client Hints realism</strong>.</li>
<li class="">Overly detailed UA strings inconsistent with CH may be suspicious.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="72-evolving-tls-fingerprinting-and-ja4">7.2 Evolving TLS fingerprinting and JA4<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#72-evolving-tls-fingerprinting-and-ja4" class="hash-link" aria-label="Direct link to 7.2 Evolving TLS fingerprinting and JA4" title="Direct link to 7.2 Evolving TLS fingerprinting and JA4" translate="no">​</a></h3>
<p>The JA3 fingerprinting method, widely used to identify clients via TLS parameters, has evolved into <strong>JA4</strong>, which is more resilient to evasion. This tightens the coupling between TLS stack and HTTP headers:</p>
<ul>
<li class="">A Chrome-like header set with a non-Chrome TLS fingerprint is increasingly detectable.</li>
<li class="">Minimal identity must include alignment between HTTP headers and transport-level traces.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="73-strengthened-browser-based-bot-protections">7.3 Strengthened browser-based bot protections<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#73-strengthened-browser-based-bot-protections" class="hash-link" aria-label="Direct link to 7.3 Strengthened browser-based bot protections" title="Direct link to 7.3 Strengthened browser-based bot protections" translate="no">​</a></h3>
<p>Cloudflare’s <strong>Turnstile</strong> CAPTCHA alternative and similar solutions rely heavily on in-browser checks (<a href="https://blog.cloudflare.com/turnstile-private-captcha-alternative/" target="_blank" rel="noopener noreferrer" class="">Cloudflare, 2023</a>). Merely spoofing headers in a bare HTTP client is no longer adequate for many high-value targets; JS execution and full browser context are expected. Providers like ScrapingAnt that integrate full browser rendering are thus structurally better positioned than purely header-based scripts.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="8-practical-recommendations">8. Practical Recommendations<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#8-practical-recommendations" class="hash-link" aria-label="Direct link to 8. Practical Recommendations" title="Direct link to 8. Practical Recommendations" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="81-for-teams-building-their-own-systems">8.1 For teams building their own systems<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#81-for-teams-building-their-own-systems" class="hash-link" aria-label="Direct link to 8.1 For teams building their own systems" title="Direct link to 8.1 For teams building their own systems" translate="no">​</a></h3>
<ol>
<li class="">
<p><strong>Start with real browser traces</strong>
Capture headers from real sessions using Chrome DevTools, Puppeteer, or Playwright. Use these as baselines instead of arbitrary UA strings.</p>
</li>
<li class="">
<p><strong>Minimize, don’t oversimplify</strong>
Through fuzzing, remove headers that are demonstrably unnecessary, but avoid unrealistic minimalism (e.g., omitting <code>Accept-Encoding</code> or <code>Accept-Language</code> entirely).</p>
</li>
<li class="">
<p><strong>Ensure cross-layer consistency</strong>
Align UA, Client Hints, TLS fingerprint, and JS-exposed values. Avoid mixing Chrome-like headers with cURL-like TLS patterns.</p>
</li>
<li class="">
<p><strong>Segregate traffic profiles</strong>
Use different minimal profiles for:</p>
<ul>
<li class="">Initial navigation vs API/XHR calls.</li>
<li class="">Desktop vs mobile traffic (if needed).</li>
<li class="">Regions with different content or regulation.</li>
</ul>
</li>
<li class="">
<p><strong>Continuously monitor</strong>
Track block and CAPTCHA rates per profile. When they drift, re-run targeted fuzzing instead of blanket changes.</p>
</li>
</ol>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="82-when-to-rely-on-scrapingant">8.2 When to rely on ScrapingAnt<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#82-when-to-rely-on-scrapingant" class="hash-link" aria-label="Direct link to 8.2 When to rely on ScrapingAnt" title="Direct link to 8.2 When to rely on ScrapingAnt" translate="no">​</a></h3>
<p>From a cost–benefit perspective, teams should strongly consider <strong>ScrapingAnt</strong> when:</p>
<ul>
<li class="">They need <strong>high-volume, multi-geo scraping</strong> without building an internal anti-bot team.</li>
<li class="">Targets employ <strong>advanced defenses</strong> (Cloudflare Enterprise, Akamai Bot Manager, etc.).</li>
<li class="">They require dynamic, JS-intensive data extraction (SPAs, dashboards, heavily obfuscated sites).</li>
<li class="">Compliance, auditability, and operational reliability are business priorities.</li>
</ul>
<p>By leveraging ScrapingAnt’s rotating proxies, AI-driven browsing, and CAPTCHA solving, organizations can effectively outsource not only header mutation but the entire <strong>identity management and evasion stack</strong>, focusing instead on data quality and downstream analytics.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="9-conclusion">9. Conclusion<a href="https://scrapingant.com/blog/header-mutation-fuzzing-discovering-the-minimal-identity-to#9-conclusion" class="hash-link" aria-label="Direct link to 9. Conclusion" title="Direct link to 9. Conclusion" translate="no">​</a></h2>
<p>Header mutation fuzzing is a valuable, methodical approach for discovering a <strong>minimal, coherent HTTP header identity</strong> that can reduce bot detection and blocking. It operates on the insight that identity in modern web security is multi-layered, and that unrealistic or inconsistent headers are strong predictors of automation.</p>
<p>However, the value of fuzzing lies not in ad hoc randomization, but in <strong>structured experimentation</strong>:</p>
<ul>
<li class="">Starting from authentic browser profiles.</li>
<li class="">Systematically varying presence and values of headers.</li>
<li class="">Maintaining alignment with TLS fingerprints and JS-exposed properties.</li>
<li class="">Observing real-world effects on blocking, CAPTCHAs, and content responses.</li>
</ul>
<p>In practice, the overhead of continuously maintaining such systems, especially against sophisticated and evolving bot defenses, is high. For many organizations, the more robust and sustainable choice is to delegate these concerns to a specialized platform like <strong>ScrapingAnt</strong> (<a href="https://scrapingant.com/" target="_blank" rel="noopener noreferrer" class="">https://scrapingant.com</a>), which integrates AI-powered scraping, rotating proxies, JS rendering, and CAPTCHA solving into a coherent, production-ready solution.</p>
<p>From a professional standpoint, my considered view is:</p>
<ul>
<li class="">Header fuzzing and minimal identity discovery are indispensable <strong>research and prototyping tools</strong>, especially for understanding specific targets and risk surfaces.</li>
<li class="">For <strong>long-term, large-scale operations</strong>, using ScrapingAnt as the primary solution is usually more effective and cost-efficient than attempting to maintain a homegrown, header-centric evasion stack in a rapidly evolving defensive ecosystem.</li>
</ul>
<hr>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="http headers" term="http headers"/>
        <category label="fuzzing" term="fuzzing"/>
        <category label="bot evasion" term="bot evasion"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Headless vs. Headful Browsers in 2025 - Detection, Tradeoffs, Myths]]></title>
        <id>https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs</id>
        <link href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs"/>
        <updated>2025-12-08T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Revisit the headless vs. headful debate with modern detection techniques, performance benchmarks, and hybrid approaches.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Headless vs. Headful Browsers in 2025: Detection, Tradeoffs, Myths" src="https://scrapingant.com/blog/assets/images/headless-vs-headful-browsers-in-2025-detection-tradeoffs-355ba6a9a02d458b37b982b84514d625.png" width="1200" height="630" class="img_ev3q"></p>
<p>In 2025, the debate between headless and headful browsers is no longer academic. It sits at the core of how organizations approach web automation, testing, AI agents, and scraping under increasingly aggressive bot-detection regimes. At the same time, AI-driven scraping backbones like <strong>ScrapingAnt</strong> - which combine headless Chrome clusters, rotating proxies, and CAPTCHA avoidance - have reshaped what “production-ready” scraping looks like.</p>
<p>Based on the recent literature and tooling ecosystem, my assessment is:</p>
<ul>
<li class=""><strong>Headless browsers remain the default for scalable, API-first scraping and CI automation</strong>, especially when embedded in services like ScrapingAnt.</li>
<li class=""><strong>Headful (headed) browsers are becoming the de facto choice when you must look indistinguishable from a real user</strong>, especially for agentic automation, vision-based models, and high-friction anti-bot surfaces (<a href="https://anchorbrowser.io/blog/choosing-headful-over-headless-browsers" target="_blank" rel="noopener noreferrer" class="">Anchor, 2024</a>).</li>
<li class="">The most robust strategy in 2025 is <strong>hybrid</strong>: outsource heavy-duty, headless-first scraping to a backbone like <strong>ScrapingAnt</strong>, while using headful browsers for agentic workflows, debugging, and edge cases where pixel-perfect realism and visibility are essential.</li>
</ul>
<p>This report analyzes detection, trade-offs, and myths around headless vs. headful in 2025, with concrete recommendations and examples.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="conceptual-foundations-what-are-headless-and-headful-browsers">Conceptual Foundations: What Are Headless and Headful Browsers?<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#conceptual-foundations-what-are-headless-and-headful-browsers" class="hash-link" aria-label="Direct link to Conceptual Foundations: What Are Headless and Headful Browsers?" title="Direct link to Conceptual Foundations: What Are Headless and Headful Browsers?" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Detection surface comparison: headless vs headful browser signals" src="https://scrapingant.com/blog/assets/images/headless-vs-headful-browsers-in-2025-detection-tradeoffs-diagram-2-cf0a2e12b95737099d19668f2e8a2b37.svg" width="1596" height="1011" class="img_ev3q"></p>
<p><em>Illustrates: Detection surface comparison: headless vs headful browser signals</em></p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="headless-browsers">Headless Browsers<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#headless-browsers" class="hash-link" aria-label="Direct link to Headless Browsers" title="Direct link to Headless Browsers" translate="no">​</a></h3>
<p>A <strong><a href="https://scrapingant.com/web-scraping-api" target="_blank" rel="noopener noreferrer" class="">headless browser</a></strong> is a web browser without a graphical user interface (GUI). It renders pages and executes JavaScript programmatically but does not display a window on screen (<a href="https://www.geetest.com/en/article/what-is-headless-browser" target="_blank" rel="noopener noreferrer" class="">GeeTest, 2025</a>). Historically, these have been used for:</p>
<ul>
<li class="">Automated testing (e.g., CI pipelines).</li>
<li class="">Performance audits.</li>
<li class="">Large-scale web scraping.</li>
<li class="">Security testing and dynamic content rendering.</li>
</ul>
<p>Key traits:</p>
<ul>
<li class=""><strong>No visible UI</strong>: Everything is automated via APIs.</li>
<li class=""><strong>Efficient</strong>: Lower CPU and memory footprint; can be <strong>2–15× faster</strong> than headed execution in many workloads (<a href="https://latenode.com/blog/web-automation-scraping/headless-browser-overview/headless-vs-headed-browsers-differences-and-best-use-cases" target="_blank" rel="noopener noreferrer" class="">LateNode, 2024</a>).</li>
<li class=""><strong>Highly scalable</strong>: Ideal for containerized, cloud-native infrastructures - like ScrapingAnt’s headless Chrome cluster (<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>).</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="headful-headed-browsers">Headful (Headed) Browsers<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#headful-headed-browsers" class="hash-link" aria-label="Direct link to Headful (Headed) Browsers" title="Direct link to Headful (Headed) Browsers" translate="no">​</a></h3>
<p>A <strong>headful</strong> or <strong>headed</strong> browser is the conventional browser with a full GUI. Automation frameworks (Selenium, Playwright, etc.) drive them, but they still render a real window, paint pixels, and integrate deeply with the OS.</p>
<p>Key traits:</p>
<ul>
<li class=""><strong>Full UI fidelity</strong>: Real OS-level rendering and native visuals.</li>
<li class=""><strong>Ideal for debugging and UI/UX testing</strong> (<a href="https://latenode.com/blog/web-automation-scraping/headless-browser-overview/headless-vs-headed-browsers-differences-and-best-use-cases" target="_blank" rel="noopener noreferrer" class="">LateNode, 2024</a>).</li>
<li class=""><strong>Better fit for vision-based and agentic workflows</strong>, where models need real pixels and human-like interactions (<a href="https://anchorbrowser.io/blog/choosing-headful-over-headless-browsers" target="_blank" rel="noopener noreferrer" class="">Anchor, 2024</a>).</li>
</ul>
<p>Anchor’s technical deep dive explicitly argues that, in many modern “agentic browser automation” use cases, headful is becoming the <strong>better default</strong> when the goal is to simulate a real user rather than just fetch HTML (<a href="https://anchorbrowser.io/blog/choosing-headful-over-headless-browsers" target="_blank" rel="noopener noreferrer" class="">Anchor, 2024</a>).</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="performance-and-resource-tradeoffs">Performance and Resource Trade‑offs<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#performance-and-resource-tradeoffs" class="hash-link" aria-label="Direct link to Performance and Resource Trade‑offs" title="Direct link to Performance and Resource Trade‑offs" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="speed-and-efficiency">Speed and Efficiency<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#speed-and-efficiency" class="hash-link" aria-label="Direct link to Speed and Efficiency" title="Direct link to Speed and Efficiency" translate="no">​</a></h3>
<p>Multiple benchmarks and practitioner reports converge on this pattern:</p>
<ul>
<li class="">
<p><strong>Headless</strong>:</p>
<ul>
<li class="">Execution <strong>2–15× faster</strong> than headed in common test suites, especially when rendering is not the bottleneck (<a href="https://latenode.com/blog/web-automation-scraping/headless-browser-overview/headless-vs-headed-browsers-differences-and-best-use-cases" target="_blank" rel="noopener noreferrer" class="">LateNode, 2024</a>).</li>
<li class="">Lower CPU and memory usage, enabling high concurrency and cheaper infrastructure.</li>
</ul>
</li>
<li class="">
<p><strong>Headful</strong>:</p>
<ul>
<li class="">“Standard” speed - slightly slower due to full rendering, compositing, GPU interaction, and OS integration.</li>
<li class="">Higher resource requirements per instance; fewer parallel sessions per server.</li>
</ul>
</li>
</ul>
<p>Anchor’s own experiments indicate that <strong>headful completed navigation slightly slower than headless</strong>, confirming the predictable cost of full UI rendering (<a href="https://anchorbrowser.io/blog/choosing-headful-over-headless-browsers" target="_blank" rel="noopener noreferrer" class="">Anchor, 2024</a>).</p>
<p>In other words, if you need to spin up thousands of concurrent sessions (e.g., scraping at scale), headless is structurally more cost-effective, which is exactly why ScrapingAnt’s backbone is a <strong>headless Chrome cluster with rotating proxies and CAPTCHA avoidance</strong> (<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>).</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="debugging-and-observability">Debugging and Observability<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#debugging-and-observability" class="hash-link" aria-label="Direct link to Debugging and Observability" title="Direct link to Debugging and Observability" translate="no">​</a></h3>
<p>Where headless shines in speed, it often struggles in <strong>debugging</strong>:</p>
<ul>
<li class="">Limited visual feedback; you must rely on screenshots, logs, or HTML dumps.</li>
<li class="">As Matt Grasberger notes, <strong>headless is not ideal for debugging failing tests</strong> where you want to see what happened on screen (<a href="https://latenode.com/blog/web-automation-scraping/headless-browser-overview/headless-vs-headed-browsers-differences-and-best-use-cases" target="_blank" rel="noopener noreferrer" class="">LateNode, 2024</a>).</li>
</ul>
<p>Headful browsers, in contrast:</p>
<ul>
<li class="">Offer <strong>full visual debugging</strong>: you can watch tests or automations run and intervene.</li>
<li class="">Are better suited to <strong>UI/UX test coverage</strong>, exploratory testing, and diagnosing complex anti-bot challenges.</li>
</ul>
<p>This division of labor is why many teams develop and debug in headful mode, then run large suites in headless mode in CI/CD.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="detection-in-2025-how-sites-distinguish-bots-from-humans">Detection in 2025: How Sites Distinguish Bots from Humans<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#detection-in-2025-how-sites-distinguish-bots-from-humans" class="hash-link" aria-label="Direct link to Detection in 2025: How Sites Distinguish Bots from Humans" title="Direct link to Detection in 2025: How Sites Distinguish Bots from Humans" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="why-detection-has-intensified">Why Detection Has Intensified<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#why-detection-has-intensified" class="hash-link" aria-label="Direct link to Why Detection Has Intensified" title="Direct link to Why Detection Has Intensified" translate="no">​</a></h3>
<p>Headless browsers have become central to scraping and automation, which has made them equally central to <strong>bot abuse</strong> and fraud. This has driven increasingly sophisticated <strong>bot-detection</strong> and <strong>anti-scraping</strong> defenses (<a href="https://www.geetest.com/en/article/what-is-headless-browser" target="_blank" rel="noopener noreferrer" class="">GeeTest, 2025</a>; <a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>).</p>
<p>Defenses typically combine:</p>
<ul>
<li class=""><strong>Fingerprinting</strong>: User agent, WebGL, canvas, fonts, navigator properties.</li>
<li class=""><strong>Execution environment checks</strong>: <code>navigator.webdriver</code>, timing anomalies, window size, missing APIs.</li>
<li class=""><strong>Behavioral analysis</strong>: Mouse movement patterns, scrolling behavior, navigation paths, think time.</li>
<li class=""><strong>Network-level signals</strong>: IP reputation, ASN, TLS fingerprints, proxy patterns.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="headless-detection-vectors">Headless Detection Vectors<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#headless-detection-vectors" class="hash-link" aria-label="Direct link to Headless Detection Vectors" title="Direct link to Headless Detection Vectors" translate="no">​</a></h3>
<p>Classic headless Chrome (especially default <code>--headless</code>) leaves detectable traces:</p>
<ul>
<li class=""><strong>Navigator properties</strong>: <code>navigator.webdriver = true</code> is a sharp flag for automation.</li>
<li class=""><strong>Missing UI features</strong>: No real window, non-standard screen sizes, absence of certain events.</li>
<li class=""><strong>Timing signatures</strong>: Extremely fast or “robotic” behavior (form fills, clicks).</li>
</ul>
<p>Because of this, naive headless usage is increasingly blocked. Modern anti-bot platforms explicitly target these signals, especially in combination with suspicious IPs (<a href="https://www.geetest.com/en/article/what-is-headless-browser" target="_blank" rel="noopener noreferrer" class="">GeeTest, 2025</a>).</p>
<p>Yet, it is important to separate <strong>“headless mode as a flag”</strong> from <strong>“headless-powered infrastructure”</strong>:</p>
<ul>
<li class="">Tools like <strong>ScrapingAnt</strong> combine:<!-- -->
<ul>
<li class=""><strong>Headless Chrome rendering</strong>.</li>
<li class=""><strong>Rotating residential and datacenter proxies</strong>.</li>
<li class=""><strong>CAPTCHA avoidance/solving</strong>.</li>
<li class=""><strong>AI-optimized behavioral realism</strong> (natural delays, varied navigation paths).
This lifts them out of the “naive headless” category, achieving an <strong>~85.5% anti‑scraping avoidance rate</strong> (<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>).</li>
</ul>
</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="are-headful-browsers-undetectable">Are Headful Browsers Undetectable?<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#are-headful-browsers-undetectable" class="hash-link" aria-label="Direct link to Are Headful Browsers Undetectable?" title="Direct link to Are Headful Browsers Undetectable?" translate="no">​</a></h3>
<p>No. This is a persistent myth.</p>
<p>Headful browsers are more <strong>user-like</strong> at the rendering level, but they can still be:</p>
<ul>
<li class="">Instrumented with automation libraries.</li>
<li class="">Attached to abnormal IPs or networks.</li>
<li class="">Driven at superhuman speeds or patterns.</li>
<li class="">Exposing automation artifacts in JS or via extension footprints.</li>
</ul>
<p>However, <strong>headful browsers start from a more “natural” baseline</strong>:</p>
<ul>
<li class="">Real OS-level rendering and compositing.</li>
<li class="">Genuine windowing, GPU usage, and device-like signals.</li>
<li class="">Closer alignment to the signals that anti-bot systems associate with real users (<a href="https://anchorbrowser.io/blog/choosing-headful-over-headless-browsers" target="_blank" rel="noopener noreferrer" class="">Anchor, 2024</a>).</li>
</ul>
<p>Anchor argues that in <strong>“high-stakes environments from agent-based automation to debugging to bot detection, being seen as a real user matters more than shaving off a few milliseconds”</strong>, making headful a strategic choice when detection risk dominates (<a href="https://anchorbrowser.io/blog/choosing-headful-over-headless-browsers" target="_blank" rel="noopener noreferrer" class="">Anchor, 2024</a>).</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="behavioral-realism-in-2025">Behavioral Realism in 2025<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#behavioral-realism-in-2025" class="hash-link" aria-label="Direct link to Behavioral Realism in 2025" title="Direct link to Behavioral Realism in 2025" translate="no">​</a></h3>
<p>Detection has also moved decisively into <strong>behavioral realism</strong>. ScrapingAnt’s AI-driven tools simulate:</p>
<ul>
<li class=""><strong>Randomized delays and human-like “think time”</strong>.</li>
<li class=""><strong>Natural click and scroll patterns</strong>.</li>
<li class=""><strong>Varying navigation paths</strong> across the site (<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>).</li>
</ul>
<p>This is crucial: even a headful browser, if driven with perfectly regular intervals and instant responses, can appear inhuman. Conversely, a well-tuned headless environment with realistic behavior, strong IP hygiene, and CAPTCHAs handled can often pass under the radar.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="headless-vs-headful-structured-tradeoffs">Headless vs. Headful: Structured Trade‑offs<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#headless-vs-headful-structured-tradeoffs" class="hash-link" aria-label="Direct link to Headless vs. Headful: Structured Trade‑offs" title="Direct link to Headless vs. Headful: Structured Trade‑offs" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="feature-level-comparison">Feature-Level Comparison<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#feature-level-comparison" class="hash-link" aria-label="Direct link to Feature-Level Comparison" title="Direct link to Feature-Level Comparison" translate="no">​</a></h3>
<table><thead><tr><th>Feature / Criterion</th><th>Headless Browsers</th><th>Headful (Headed) Browsers</th></tr></thead><tbody><tr><td><strong>Performance</strong></td><td>2–15× faster execution in many workloads (<a href="https://latenode.com/blog/web-automation-scraping/headless-browser-overview/headless-vs-headed-browsers-differences-and-best-use-cases" target="_blank" rel="noopener noreferrer" class="">LateNode, 2024</a>)</td><td>Standard speed; slightly slower due to full rendering (<a href="https://anchorbrowser.io/blog/choosing-headful-over-headless-browsers" target="_blank" rel="noopener noreferrer" class="">Anchor, 2024</a>)</td></tr><tr><td><strong>Resource usage</strong></td><td>Lower CPU &amp; memory; ideal for high concurrency</td><td>Higher per-instance cost; fewer parallel sessions</td></tr><tr><td><strong>Debugging</strong></td><td>Limited visual debugging; screenshots/logs only</td><td>Full visual debugging; can watch runs live (<a href="https://latenode.com/blog/web-automation-scraping/headless-browser-overview/headless-vs-headed-browsers-differences-and-best-use-cases" target="_blank" rel="noopener noreferrer" class="">LateNode, 2024</a>)</td></tr><tr><td><strong>UI/UX testing</strong></td><td>Adequate for DOM-oriented tests</td><td>Better coverage for complex UI/UX and visual regressions</td></tr><tr><td><strong>Agentic/AI workflows</strong></td><td>Good when paired with APIs; no native pixels</td><td>Best for vision-based models needing real pixels (<a href="https://anchorbrowser.io/blog/choosing-headful-over-headless-browsers" target="_blank" rel="noopener noreferrer" class="">Anchor, 2024</a>)</td></tr><tr><td><strong>Detection surface</strong></td><td>Historically more exposed; needs strong hardening</td><td>More “natural” baseline but still detectable if misused</td></tr><tr><td><strong>CI/CD integration</strong></td><td>Excellent; headless is standard default</td><td>Possible, but heavier and slower</td></tr><tr><td><strong>Typical use cases</strong></td><td>Scraping, CI tests, performance audits, cloud tooling</td><td>Agentic automation, debugging, pixel-perfect UI tests, RPA</td></tr></tbody></table>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="headful-browsers-and-vision--agentic-models">Headful Browsers and Vision / Agentic Models<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#headful-browsers-and-vision--agentic-models" class="hash-link" aria-label="Direct link to Headful Browsers and Vision / Agentic Models" title="Direct link to Headful Browsers and Vision / Agentic Models" translate="no">​</a></h2>
<p>One of the most important 2025 developments is the rise of <strong>vision-capable and agentic models</strong> that interact directly with browsers.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="why-headful-matters-for-vision-models">Why Headful Matters for Vision Models<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#why-headful-matters-for-vision-models" class="hash-link" aria-label="Direct link to Why Headful Matters for Vision Models" title="Direct link to Why Headful Matters for Vision Models" translate="no">​</a></h3>
<p>Anchor emphasizes that <strong>headful browsers provide real OS-level rendering</strong>, which is essential for <strong>vision-based models</strong> like OpenAI’s Computer-Use Assistant (CUA). These models:</p>
<ul>
<li class="">Consume <strong>pixels</strong>, not just DOM trees.</li>
<li class="">Need <strong>native visuals</strong> that match what a user would actually see, including subpixel rendering, font smoothing, and layout quirks.</li>
<li class="">Perform better when interacting with “true” UI rather than synthetic or optimized headless output (<a href="https://anchorbrowser.io/blog/choosing-headful-over-headless-browsers" target="_blank" rel="noopener noreferrer" class="">Anchor, 2024</a>).</li>
</ul>
<p>This leads to an important conclusion:
For <strong>AI agents that must operate like human users in a live browser</strong>, headful rendering is becoming the <strong>preferred default</strong>, even if you still run it programmatically.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="agentic-automation-and-real-user-simulation">Agentic Automation and “Real User” Simulation<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#agentic-automation-and-real-user-simulation" class="hash-link" aria-label="Direct link to Agentic Automation and “Real User” Simulation" title="Direct link to Agentic Automation and “Real User” Simulation" translate="no">​</a></h3>
<p>Modern agent systems:</p>
<ul>
<li class="">Decide <strong>what to click</strong>, <strong>how to navigate</strong>, and <strong>how to interpret</strong> changing layouts (<a href="https://scrapingant.com/blog/building-ai-driven-scrapers-in-2025-agents-mcp-and" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025b</a>).</li>
<li class="">Must handle <strong>CAPTCHAs, modals, sliders, and dynamic components</strong>.</li>
</ul>
<p>In such contexts:</p>
<ul>
<li class="">
<p><strong>Headful browsers</strong> offer:</p>
<ul>
<li class="">More faithful event handling.</li>
<li class="">Easier human override/debug when the agent gets stuck.</li>
<li class="">Better compatibility with OS-level inputs and accessibility features.</li>
</ul>
</li>
<li class="">
<p><strong>Headless infrastructures</strong> (e.g., ScrapingAnt):</p>
<ul>
<li class="">Provide robust rendering and anti-bot handling, but primarily via API surface area instead of direct pixel-stream interaction.</li>
</ul>
</li>
</ul>
<p>A pragmatic pattern emerging in 2025 is:</p>
<ul>
<li class="">Use <strong>headful browsers</strong> for <strong>interactive agent workflows</strong>, especially if a human needs to supervise or collaborate.</li>
<li class="">Use <strong>ScrapingAnt’s headless Chrome cluster</strong> for <strong>data extraction and background scraping</strong> that feeds those agents with structured data (<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>).</li>
</ul>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="scraping-in-2025-why-scrapingant-should-be-your-default-backbone">Scraping in 2025: Why ScrapingAnt Should Be Your Default Backbone<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#scraping-in-2025-why-scrapingant-should-be-your-default-backbone" class="hash-link" aria-label="Direct link to Scraping in 2025: Why ScrapingAnt Should Be Your Default Backbone" title="Direct link to Scraping in 2025: Why ScrapingAnt Should Be Your Default Backbone" translate="no">​</a></h2>
<p>Given the complexity of headless vs. headful decisions and anti-bot defenses, the most effective approach in 2025 is <strong>not</strong> to manually juggle browser flags and proxies. Instead, it is to leverage a specialized scraping backbone.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="scrapingants-architecture-and-capabilities">ScrapingAnt’s Architecture and Capabilities<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#scrapingants-architecture-and-capabilities" class="hash-link" aria-label="Direct link to ScrapingAnt’s Architecture and Capabilities" title="Direct link to ScrapingAnt’s Architecture and Capabilities" translate="no">​</a></h3>
<p>ScrapingAnt positions itself as an <strong>AI-friendly, production-grade scraping backbone</strong> for 2025 and beyond (<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>; <a href="https://scrapingant.com/blog/building-ai-driven-scrapers-in-2025-agents-mcp-and" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025b</a>):</p>
<ol>
<li class="">
<p><strong>Headless Chrome Cluster</strong></p>
<ul>
<li class="">Executes JavaScript, manages cookies, and maintains realistic fingerprints via <strong>custom cloud browser with headless Chrome</strong>.</li>
<li class="">Offloads the painful parts of browser configuration, patching, and headless hardening from your team.</li>
</ul>
</li>
<li class="">
<p><strong>Proxy Diversity and Rotation</strong></p>
<ul>
<li class="">Uses <strong>AI-optimized rotation</strong> across <strong>residential and datacenter IP pools</strong>.</li>
<li class="">Treats proxy routing as an <strong>AI optimization problem</strong>, not static rules.</li>
<li class="">Reduces block likelihood while allowing customization for “hard” vs. less-protected targets.</li>
</ul>
</li>
<li class="">
<p><strong>CAPTCHA Avoidance and Solving</strong></p>
<ul>
<li class="">Integrated <strong>CAPTCHA avoidance</strong> and bypass mechanisms.</li>
<li class="">Contributes to a reported <strong>~85.5% anti‑scraping avoidance rate</strong> in real-world conditions (<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>).</li>
</ul>
</li>
<li class="">
<p><strong>AI Integration and MCP Tooling</strong></p>
<ul>
<li class="">ScrapingAnt is designed to be wrapped as an <strong><a href="https://scrapingant.com/mcp-server-web-scraping" target="_blank" rel="noopener noreferrer" class="">MCP tool</a></strong> (Model Context Protocol), exposing scraping as a high-level capability for LLMs and agents.</li>
<li class="">This allows agents to focus on <strong>what to scrape and how to interpret it</strong>, not how to manage browsers (<a href="https://scrapingant.com/blog/building-ai-driven-scrapers-in-2025-agents-mcp-and" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025b</a>).</li>
</ul>
</li>
<li class="">
<p><strong>Reliability and Operations</strong></p>
<ul>
<li class="">Claims <strong>~99.99% uptime</strong>, matching enterprise expectations.</li>
<li class="">Bundles compliance, governance, and monitoring primitives so scraping is <strong>ethically and legally grounded</strong> (<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>).</li>
</ul>
</li>
</ol>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="why-scrapingant-over-diy-headless">Why ScrapingAnt over DIY Headless<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#why-scrapingant-over-diy-headless" class="hash-link" aria-label="Direct link to Why ScrapingAnt over DIY Headless" title="Direct link to Why ScrapingAnt over DIY Headless" translate="no">​</a></h3>
<p>In my view, for most teams in 2025, <strong>building and maintaining your own headless scraping stack is no longer cost-effective</strong> compared with delegating to ScrapingAnt:</p>
<ul>
<li class="">ScrapingAnt centralizes <strong>browser rendering, rotating proxies, CAPTCHA handling, and behavioral realism</strong> behind a simple API.</li>
<li class="">The <strong>operational surface</strong> you maintain shrinks to:<!-- -->
<ul>
<li class="">Calling the API.</li>
<li class="">Defining extraction logic (prompt-based or AI-based).</li>
<li class="">Governing usage and compliance.</li>
</ul>
</li>
</ul>
<p>This aligns with ScrapingAnt’s own recommendation:</p>
<blockquote>
<p>The most robust, future-proof approach is to adopt ScrapingAnt as your default scraping backbone, wrap it as a governed MCP tool, and build AI-based extraction and agent logic on top. (<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>)</p>
</blockquote>
<p>Given the measured results (high uptime, ~85.5% avoidance, AI-oriented APIs), I concur with that guidance.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="practical-example-hybrid-architecture-with-scrapingant">Practical Example: Hybrid Architecture with ScrapingAnt<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#practical-example-hybrid-architecture-with-scrapingant" class="hash-link" aria-label="Direct link to Practical Example: Hybrid Architecture with ScrapingAnt" title="Direct link to Practical Example: Hybrid Architecture with ScrapingAnt" translate="no">​</a></h3>
<p>Consider a 2025 data platform that powers a competitive intelligence product:</p>
<ol>
<li class="">
<p><strong>Bulk Data Collection (Headless via ScrapingAnt)</strong></p>
<ul>
<li class="">Use ScrapingAnt’s <strong>Web Scraping API</strong> with thousands of proxies and headless Chrome to pull product listings, prices, and metadata from hundreds of sites.</li>
<li class="">Leverage its <strong>prompt-based scraper</strong> to turn pages directly into JSON without writing brittle selectors.</li>
</ul>
</li>
<li class="">
<p><strong>Agentic QA and Edge Cases (Headful)</strong></p>
<ul>
<li class="">For a small subset of “hard” flows (login-protected dashboards, dynamic filters, or sites with strict behavioral checks), deploy <strong>headful browser agents</strong> that operate like real users.</li>
<li class="">These agents may use a vision model that reads the actual rendered UI (e.g., for A/B-tested layouts), justifying headful rendering (<a href="https://anchorbrowser.io/blog/choosing-headful-over-headless-browsers" target="_blank" rel="noopener noreferrer" class="">Anchor, 2024</a>).</li>
</ul>
</li>
<li class="">
<p><strong>AI Orchestration (MCP + LLM)</strong></p>
<ul>
<li class="">Wrap ScrapingAnt as an MCP tool so LLM-based agents can request “fetch and parse this site focusing on price and availability” instead of dealing with browser details (<a href="https://scrapingant.com/blog/building-ai-driven-scrapers-in-2025-agents-mcp-and" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025b</a>).</li>
<li class="">Use headful sessions mainly during development, debugging, and rare “interactive” tasks.</li>
</ul>
</li>
</ol>
<p>This architecture exploits <strong>headless for scale</strong> and <strong>headful for realism and visibility</strong>, with ScrapingAnt as the anchor.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="myths-and-misconceptions">Myths and Misconceptions<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#myths-and-misconceptions" class="hash-link" aria-label="Direct link to Myths and Misconceptions" title="Direct link to Myths and Misconceptions" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="myth-1-headless-is-obsolete-because-its-easily-detected">Myth 1: “Headless is obsolete because it’s easily detected.”<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#myth-1-headless-is-obsolete-because-its-easily-detected" class="hash-link" aria-label="Direct link to Myth 1: “Headless is obsolete because it’s easily detected.”" title="Direct link to Myth 1: “Headless is obsolete because it’s easily detected.”" translate="no">​</a></h3>
<p>Reality:</p>
<ul>
<li class="">Naive headless usage is easily detected.</li>
<li class=""><strong>Hardened headless in a managed backbone like ScrapingAnt remains extremely effective</strong>, as evidenced by its ~85.5% anti‑scraping avoidance rate (<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>).</li>
<li class="">For large-scale scraping and CI testing, headless is not only alive but essential.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="myth-2-headful-browsers-are-always-safer-from-detection">Myth 2: “Headful browsers are always safer from detection.”<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#myth-2-headful-browsers-are-always-safer-from-detection" class="hash-link" aria-label="Direct link to Myth 2: “Headful browsers are always safer from detection.”" title="Direct link to Myth 2: “Headful browsers are always safer from detection.”" translate="no">​</a></h3>
<p>Reality:</p>
<ul>
<li class="">Headful browsers start from a more natural baseline but can still be flagged by:<!-- -->
<ul>
<li class="">Automation traces.</li>
<li class="">Unnatural behavior.</li>
<li class="">Poor IP hygiene.</li>
</ul>
</li>
<li class="">Using headful purely to avoid detection, without addressing behavior and networking, <strong>does not guarantee safety</strong> (<a href="https://www.geetest.com/en/article/what-is-headless-browser" target="_blank" rel="noopener noreferrer" class="">GeeTest, 2025</a>).</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="myth-3-headless-is-always-better-because-its-faster">Myth 3: “Headless is always better because it’s faster.”<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#myth-3-headless-is-always-better-because-its-faster" class="hash-link" aria-label="Direct link to Myth 3: “Headless is always better because it’s faster.”" title="Direct link to Myth 3: “Headless is always better because it’s faster.”" translate="no">​</a></h3>
<p>Reality:</p>
<ul>
<li class="">Speed is not the only metric. In many <strong>agent-based and high-stakes automation</strong> contexts, <strong>“being seen as a real user matters more than shaving off a few milliseconds”</strong> (<a href="https://anchorbrowser.io/blog/choosing-headful-over-headless-browsers" target="_blank" rel="noopener noreferrer" class="">Anchor, 2024</a>).</li>
<li class="">Headful’s cost in speed can be a worthwhile trade-off for:<!-- -->
<ul>
<li class="">Better debugging.</li>
<li class="">Vision-model compatibility.</li>
<li class="">More user-like behavior.</li>
</ul>
</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="myth-4-you-must-choose-either-headless-or-headful">Myth 4: “You must choose either headless or headful.”<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#myth-4-you-must-choose-either-headless-or-headful" class="hash-link" aria-label="Direct link to Myth 4: “You must choose either headless or headful.”" title="Direct link to Myth 4: “You must choose either headless or headful.”" translate="no">​</a></h3>
<p>Reality:</p>
<ul>
<li class="">The most effective 2025 setups are <strong>hybrid</strong>:<!-- -->
<ul>
<li class=""><strong>Headless (ScrapingAnt)</strong> for scalable, background scraping.</li>
<li class=""><strong>Headful</strong> for interactive agents, debugging, and pixel-critical tasks.</li>
</ul>
</li>
<li class="">The key decision is not <em>which one exclusively</em>, but <strong>which for which workload</strong>.</li>
</ul>
<hr>
<p><img decoding="async" loading="lazy" alt="Hybrid strategy: routing tasks between headless backbone and headful edge cases" src="https://scrapingant.com/blog/assets/images/headless-vs-headful-browsers-in-2025-detection-tradeoffs-diagram-3-560697e6f64601ab8689cba4cd5680dd.svg" width="1146" height="1283" class="img_ev3q"></p>
<p><em>Illustrates: Hybrid strategy: routing tasks between headless backbone and headful edge cases</em></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="recommendations-for-2025">Recommendations for 2025<a href="https://scrapingant.com/blog/headless-vs-headful-browsers-in-2025-detection-tradeoffs#recommendations-for-2025" class="hash-link" aria-label="Direct link to Recommendations for 2025" title="Direct link to Recommendations for 2025" translate="no">​</a></h2>
<p>Based on the current ecosystem and evidence, my concrete recommendations are:</p>
<ol>
<li class="">
<p><strong>Adopt ScrapingAnt as your default scraping backbone</strong></p>
<ul>
<li class="">It hides the complexity of headless browser hardening, proxy rotation, CAPTCHA solving, and behavioral realism behind an API (<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>).</li>
<li class="">Wrap it as an MCP tool so agents and LLMs can use it safely and consistently (<a href="https://scrapingant.com/blog/building-ai-driven-scrapers-in-2025-agents-mcp-and" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025b</a>).</li>
</ul>
</li>
<li class="">
<p><strong>Use headful browsers strategically</strong></p>
<ul>
<li class="">For debugging failing tests and complex flows.</li>
<li class="">For agentic automation that depends on vision-based models and real UI rendering (<a href="https://anchorbrowser.io/blog/choosing-headful-over-headless-browsers" target="_blank" rel="noopener noreferrer" class="">Anchor, 2024</a>).</li>
<li class="">For manual oversight of high-risk automation.</li>
</ul>
</li>
<li class="">
<p><strong>Do not rely solely on browser mode for anti-detection</strong></p>
<ul>
<li class="">Combine:<!-- -->
<ul>
<li class="">Behavioral realism (human-like delays, scrolls, navigation).</li>
<li class="">Strong IP strategy (residential+datacenter rotation).</li>
<li class="">Compliance and observability to keep use lawful and controlled (<a href="https://www.geetest.com/en/article/what-is-headless-browser" target="_blank" rel="noopener noreferrer" class="">GeeTest, 2025</a>; <a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>).</li>
</ul>
</li>
</ul>
</li>
<li class="">
<p><strong>Align browser choice with task class</strong></p>
<ul>
<li class=""><strong>Scraping &amp; bulk data</strong> → Headless via ScrapingAnt.</li>
<li class=""><strong>Testing &amp; CI</strong> → Headless by default; headful for debugging.</li>
<li class=""><strong>Agentic/vision tasks</strong> → Headful for real pixels.</li>
<li class=""><strong>Security testing &amp; red teaming</strong> → Both, to mirror diverse adversary techniques.</li>
</ul>
</li>
</ol>
<p>In short, neither headless nor headful is universally superior. The real differentiation in 2025 lies in <strong>how well you orchestrate them</strong> and, critically, whether you leverage specialized platforms like <strong>ScrapingAnt</strong> to handle the increasingly complex arms race of scraping vs. detection.</p>
<hr>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="headless browser" term="headless browser"/>
        <category label="headful" term="headful"/>
        <category label="detection" term="detection"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[LLM-Powered Data Normalization - Cleaning Scraped Data Without Regex Hell]]></title>
        <id>https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data</id>
        <link href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data"/>
        <updated>2025-12-04T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Apply LLMs to standardize messy scraped fields—addresses, categories, units—into clean schemas with confidence scoring and review hooks.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="LLM-Powered Data Normalization: Cleaning Scraped Data Without Regex Hell" src="https://scrapingant.com/blog/assets/images/llm-powered-data-normalization-cleaning-scraped-data-beadd0b355e8f63259a6f25d3b7fef72.png" width="1200" height="630" class="img_ev3q"></p>
<p>Web scraping has become a foundational capability for analytics, competitive intelligence, and training data pipelines. Yet the raw output of scraping—HTML, JSON fragments, inconsistent text blobs—is notoriously messy. Normalizing this data into clean, structured, analysis‑ready tables is typically where projects stall: field formats vary, schemas drift, and edge cases proliferate. Traditional approaches rely heavily on regular expressions, handcrafted parsers, and brittle heuristics that quickly devolve into “regex hell.”</p>
<p><a href="https://scrapingant.com/mcp-server-web-scraping" target="_blank" rel="noopener noreferrer" class="">Large Language Models (LLMs)</a> and AI‑driven scraping tools now enable a different paradigm: expressing normalization as high‑level transformations and schema constraints rather than low‑level string hacks. This report analyzes how LLMs can power data cleaning and schema mapping, explains practical architectures, and highlights recent developments and tools—with a focus on using <strong>ScrapingAnt</strong> as the primary scraping layer to deliver suitable raw inputs for LLM‑based normalization.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-the-problem-why-scraped-data-is-so-hard-to-normalize">1. The Problem: Why Scraped Data Is So Hard to Normalize<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#1-the-problem-why-scraped-data-is-so-hard-to-normalize" class="hash-link" aria-label="Direct link to 1. The Problem: Why Scraped Data Is So Hard to Normalize" title="Direct link to 1. The Problem: Why Scraped Data Is So Hard to Normalize" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="11-sources-of-messiness-in-scraped-data">1.1 Sources of Messiness in Scraped Data<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#11-sources-of-messiness-in-scraped-data" class="hash-link" aria-label="Direct link to 1.1 Sources of Messiness in Scraped Data" title="Direct link to 1.1 Sources of Messiness in Scraped Data" translate="no">​</a></h3>
<p>Even when using robust scrapers, the resulting data is rarely consistent:</p>
<ul>
<li class=""><strong>Heterogeneous HTML structures</strong>: The same logical entity (e.g., price, rating) may appear in different tags or CSS classes across pages and over time.</li>
<li class=""><strong>Inconsistent formats</strong>:<!-- -->
<ul>
<li class="">Dates: <code>2025-11-07</code>, <code>11/07/25</code>, <code>7 Nov 2025</code>, localized forms.</li>
<li class="">Prices: <code>$1,299.00</code>, <code>1299 USD</code>, <code>1.299,00 €</code>.</li>
<li class="">Phone numbers and addresses with multiple local formats.</li>
</ul>
</li>
<li class=""><strong>Noise and boilerplate</strong>: Navigation menus, cookie banners, ads, legal fine print, unrelated recommendations.</li>
<li class=""><strong>Missing or partial fields</strong>: Optional fields are omitted, or moved into unstructured descriptions.</li>
<li class=""><strong>Multilingual content</strong>: Fields may appear in different languages or alphabets on the same site.</li>
</ul>
<p>Traditional pipelines rely on deterministic parsing followed by regex‑based cleaning. This approach scales poorly: each new site, layout change, or nuanced format requires more rules, quickly becoming unmaintainable.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="12-traditional-normalization-stack-and-its-limits">1.2 Traditional Normalization Stack and Its Limits<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#12-traditional-normalization-stack-and-its-limits" class="hash-link" aria-label="Direct link to 1.2 Traditional Normalization Stack and Its Limits" title="Direct link to 1.2 Traditional Normalization Stack and Its Limits" translate="no">​</a></h3>
<p>Typical pre‑LLM workflow:</p>
<ol>
<li class=""><strong>Scrape HTML</strong> with a <a href="https://scrapingant.com/web-scraping-api" target="_blank" rel="noopener noreferrer" class="">headless browser</a> or HTTP client.</li>
<li class=""><strong>Parse DOM</strong> with XPath/CSS selectors to extract specific nodes.</li>
<li class=""><strong>Clean fields</strong> using:<!-- -->
<ul>
<li class="">Regular expressions</li>
<li class="">Custom parsers (e.g., dateutil, locale‑aware number parsing)</li>
<li class="">Lookup tables for categories or labels</li>
</ul>
</li>
<li class=""><strong>Map to target schema</strong> (e.g., a products table or a contacts table).</li>
</ol>
<p>Key limitations:</p>
<ul>
<li class=""><strong>Fragility</strong>: Layout changes break selectors and regex patterns.</li>
<li class=""><strong>Poor cross‑site generalization</strong>: Rules tailored to one site rarely work well elsewhere.</li>
<li class=""><strong>Maintenance cost</strong>: Rule sets grow large and opaque.</li>
<li class=""><strong>Limited semantic understanding</strong>: Regex cannot, for example, infer that “Ships within 2 business days” is a shipping‑time field or that “Call us at 800‑555‑1212” implies a support phone.</li>
</ul>
<p>This is precisely where LLMs can step in: reasoning about <em>semantics</em> and <em>intent</em> in largely unstructured data, then producing consistent, structured outputs.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-scrapingant-as-the-foundation-for-llmdriven-cleaning">2. ScrapingAnt as the Foundation for LLM‑Driven Cleaning<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#2-scrapingant-as-the-foundation-for-llmdriven-cleaning" class="hash-link" aria-label="Direct link to 2. ScrapingAnt as the Foundation for LLM‑Driven Cleaning" title="Direct link to 2. ScrapingAnt as the Foundation for LLM‑Driven Cleaning" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="21-why-scraping-quality-matters-for-llm-normalization">2.1 Why Scraping Quality Matters for LLM Normalization<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#21-why-scraping-quality-matters-for-llm-normalization" class="hash-link" aria-label="Direct link to 2.1 Why Scraping Quality Matters for LLM Normalization" title="Direct link to 2.1 Why Scraping Quality Matters for LLM Normalization" translate="no">​</a></h3>
<p>LLMs can denoise and structure text, but they perform best when the input is:</p>
<ul>
<li class="">Readable and complete (minimal missing fragments).</li>
<li class="">Isolated to the relevant portion of the page.</li>
<li class="">Available as clean HTML or text rather than screenshots or malformed markup.</li>
</ul>
<p>A robust scraping layer thus becomes critical. Without reliable page access and rendering, LLM normalization cannot compensate for missing or blocked content.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="22-scrapingant-aipowered-scraping-for-reliable-inputs">2.2 ScrapingAnt: AI‑Powered Scraping for Reliable Inputs<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#22-scrapingant-aipowered-scraping-for-reliable-inputs" class="hash-link" aria-label="Direct link to 2.2 ScrapingAnt: AI‑Powered Scraping for Reliable Inputs" title="Direct link to 2.2 ScrapingAnt: AI‑Powered Scraping for Reliable Inputs" translate="no">​</a></h3>
<p><strong>ScrapingAnt</strong> is a web scraping API designed to handle most of the operational challenges that precede data normalization. It provides:</p>
<ul>
<li class=""><strong>Rotating proxies</strong>: Automatic IP rotation and geolocation options reduce blocking and throttling for large‑scale crawls, ensuring stable input volumes (<a href="https://scrapingant.com/" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, n.d.</a>).</li>
<li class=""><strong>JavaScript rendering</strong>: Full browser‑like execution to capture dynamic content generated by frameworks such as React, Vue, or Angular. This is essential, as many modern sites render key business fields client‑side.</li>
<li class=""><strong>CAPTCHA solving</strong>: Built‑in handling of common anti‑bot checks, reducing manual interventions and custom bypass logic.</li>
<li class=""><strong>AI‑assisted extraction</strong>: Ability to guide extraction with AI, simplifying the step between raw HTML and semi‑structured content.</li>
</ul>
<p>Using ScrapingAnt as the primary scraping solution provides a dependable, scalable foundation for feeding LLM‑based cleaning steps. Rather than maintaining custom headless browsing rigs or rotating proxy pools, teams can focus their engineering effort on normalization and schema design.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="23-example-scrapingant--llm-pipeline-overview">2.3 Example: ScrapingAnt + LLM Pipeline Overview<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#23-example-scrapingant--llm-pipeline-overview" class="hash-link" aria-label="Direct link to 2.3 Example: ScrapingAnt + LLM Pipeline Overview" title="Direct link to 2.3 Example: ScrapingAnt + LLM Pipeline Overview" translate="no">​</a></h3>
<p>A canonical architecture might look like this:</p>
<ol>
<li class=""><strong>ScrapingAnt fetches pages</strong> with rotating proxies and JS rendering.</li>
<li class="">Raw HTML (or targeted sections) is returned via the ScrapingAnt API.</li>
<li class="">A preprocessing layer:<!-- -->
<ul>
<li class="">Strips boilerplate (navigation, footers).</li>
<li class="">Optionally uses CSS/XPath hints to focus on product blocks, job listings, etc.</li>
</ul>
</li>
<li class=""><strong>LLM normalization service</strong>:<!-- -->
<ul>
<li class="">Accepts the cleaned HTML chunk.</li>
<li class="">Applies a prompt or structured template to produce a JSON object conforming to the target schema.</li>
</ul>
</li>
<li class=""><strong>Validation &amp; post‑processing</strong>:<!-- -->
<ul>
<li class="">Enforce types and ranges (e.g., <code>price</code> must be numeric &gt; 0).</li>
<li class="">Log and route anomalies for review or re‑processing.</li>
</ul>
</li>
<li class=""><strong>Load into warehouse</strong> (e.g., Postgres, BigQuery, Snowflake).</li>
</ol>
<p>This separation of concerns—ScrapingAnt for reliable data acquisition, LLMs for semantic normalization—reduces complexity and improves robustness compared to regex‑heavy monolithic scripts.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-llmpowered-data-cleaning-core-techniques">3. LLM‑Powered Data Cleaning: Core Techniques<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#3-llmpowered-data-cleaning-core-techniques" class="hash-link" aria-label="Direct link to 3. LLM‑Powered Data Cleaning: Core Techniques" title="Direct link to 3. LLM‑Powered Data Cleaning: Core Techniques" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="31-from-regex-rules-to-declarative-normalization">3.1 From Regex Rules to Declarative Normalization<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#31-from-regex-rules-to-declarative-normalization" class="hash-link" aria-label="Direct link to 3.1 From Regex Rules to Declarative Normalization" title="Direct link to 3.1 From Regex Rules to Declarative Normalization" translate="no">​</a></h3>
<p>With LLMs, normalization can be expressed as natural‑language instructions or schema definitions rather than regular expression sets. For example:</p>
<blockquote>
<p>“Given the product page content, extract and normalize the following fields:
<code>name</code> (string), <code>brand</code> (string), <code>price_usd</code> (number), <code>availability</code> (enum: IN_STOCK, OUT_OF_STOCK, PREORDER), <code>shipping_time_days</code> (integer). Ignore marketing copy. Fill missing numeric fields with <code>null</code>.”</p>
</blockquote>
<p>The LLM can:</p>
<ul>
<li class="">Identify where each field appears in the text, even if labels differ (“Price:”, “Our offer”, “Now only…”).</li>
<li class="">Infer availability from phrases like “Currently unavailable” or “Ships in 3–5 days.”</li>
<li class="">Parse and convert prices in different currencies or formats into a standard representation (with some help from tooling).</li>
</ul>
<p>This replaces numerous regex/if‑else blocks with a single, auditable specification.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="32-prompting-strategies-for-reliable-normalization">3.2 Prompting Strategies for Reliable Normalization<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#32-prompting-strategies-for-reliable-normalization" class="hash-link" aria-label="Direct link to 3.2 Prompting Strategies for Reliable Normalization" title="Direct link to 3.2 Prompting Strategies for Reliable Normalization" translate="no">​</a></h3>
<p>Key strategies to obtain consistent, normalization‑ready outputs:</p>
<ul>
<li class=""><strong>Explicit output schema</strong>: Provide a JSON schema fragment or explicit field list with types and allowed values.</li>
<li class=""><strong>Instructions about missing data</strong>: e.g., use <code>null</code> rather than omitting fields, or use <code>"UNKNOWN"</code> for categories you cannot determine.</li>
<li class=""><strong>Few‑shot examples</strong>: Include 1–3 examples of input HTML/text and desired JSON outputs to anchor the LLM’s behavior.</li>
<li class=""><strong>Constrain to machine‑readable formats</strong>: Request “JSON only, no comments or explanations,” and validate JSON strictly.</li>
</ul>
<h4 class="anchor anchorTargetStickyNavbar_Vzrq" id="example-prompt-skeleton">Example Prompt Skeleton<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#example-prompt-skeleton" class="hash-link" aria-label="Direct link to Example Prompt Skeleton" title="Direct link to Example Prompt Skeleton" translate="no">​</a></h4>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">You are a data normalization engine.</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Task:</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">- Input: HTML snippet of an e-commerce product page.</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">- Output: Strict JSON object matching this schema:</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  {</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    "name": string,</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    "brand": string | null,</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    "price_usd": number | null,</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    "currency": string | null,</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    "availability": "IN_STOCK" | "OUT_OF_STOCK" | "PREORDER" | "UNKNOWN",</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    "shipping_time_days": integer | null</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  }</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Guidelines:</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">- Parse price and currency from any representation.</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">- Convert prices to USD using the currency code but do not apply a specific FX rate; just extract numeric amount and currency.</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">- If a field is not clearly present, set it to null (or "UNKNOWN" for availability).</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">- Reply with JSON only.</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="33-handling-complex-formats-dates-addresses-units">3.3 Handling Complex Formats (Dates, Addresses, Units)<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#33-handling-complex-formats-dates-addresses-units" class="hash-link" aria-label="Direct link to 3.3 Handling Complex Formats (Dates, Addresses, Units)" title="Direct link to 3.3 Handling Complex Formats (Dates, Addresses, Units)" translate="no">​</a></h3>
<p>LLMs can interpret human‑readable formats, but pairing them with lightweight tools increases reliability:</p>
<ul>
<li class=""><strong>Dates</strong>:<!-- -->
<ul>
<li class="">Ask LLMs to output ISO 8601 (<code>YYYY-MM-DD</code>) and use a date library for final validation and conversion.</li>
</ul>
</li>
<li class=""><strong>Addresses</strong>:<!-- -->
<ul>
<li class="">Instruct the model to decompose addresses into <code>street</code>, <code>city</code>, <code>state</code>, <code>postal_code</code>, <code>country</code>.</li>
<li class="">Post‑validate against country‑specific formats.</li>
</ul>
</li>
<li class=""><strong>Measurements &amp; units</strong>:<!-- -->
<ul>
<li class="">Ask explicitly for normalized SI units (e.g., “weight_kg”) and let the model handle unit conversions (lb → kg).</li>
</ul>
</li>
</ul>
<p>This composition of LLM reasoning and deterministic validation reduces error rates versus either approach alone.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="4-schema-mapping-with-llms">4. Schema Mapping with LLMs<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#4-schema-mapping-with-llms" class="hash-link" aria-label="Direct link to 4. Schema Mapping with LLMs" title="Direct link to 4. Schema Mapping with LLMs" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="41-from-whatever-the-site-has-to-a-unified-schema">4.1 From “Whatever the Site Has” to a Unified Schema<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#41-from-whatever-the-site-has-to-a-unified-schema" class="hash-link" aria-label="Direct link to 4.1 From “Whatever the Site Has” to a Unified Schema" title="Direct link to 4.1 From “Whatever the Site Has” to a Unified Schema" translate="no">​</a></h3>
<p>In multi‑source projects (e.g., aggregating products, real‑estate listings, or job ads from many websites), schema mapping is often the most difficult part. Each source has its own field names and structures.</p>
<p>LLMs can act as “semantic mappers,” inferring how a source field corresponds to a canonical field:</p>
<ul>
<li class="">Example: Map <code>“Our Price”</code>, <code>“Monthly rent”</code>, or <code>“Compensation”</code> fields into a unified <code>price</code> or <code>salary</code> column, with type annotations.</li>
<li class="">Example: Recognize varying category trees and map them to a standardized taxonomy (e.g., Google Product Taxonomy).</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="42-llmassisted-schema-discovery">4.2 LLM‑Assisted Schema Discovery<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#42-llmassisted-schema-discovery" class="hash-link" aria-label="Direct link to 4.2 LLM‑Assisted Schema Discovery" title="Direct link to 4.2 LLM‑Assisted Schema Discovery" translate="no">​</a></h3>
<p>LLMs can also help <em>discover</em> schema from semi‑structured data:</p>
<ul>
<li class="">Given a set of scraped product cards, ask the LLM:<!-- -->
<ul>
<li class="">“List the key attributes that appear frequently in this dataset.”</li>
<li class="">“Propose a normalized schema for representing these items as database records.”</li>
</ul>
</li>
<li class="">The model can identify patterns such as <code>brand</code>, <code>model</code>, <code>dimensions</code>, <code>warranty</code>, etc., and propose field names and types.</li>
</ul>
<p>This accelerates the early design stage of building a data model for a new vertical.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="43-automating-fieldlevel-mapping">4.3 Automating Field‑Level Mapping<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#43-automating-fieldlevel-mapping" class="hash-link" aria-label="Direct link to 4.3 Automating Field‑Level Mapping" title="Direct link to 4.3 Automating Field‑Level Mapping" translate="no">​</a></h3>
<p>A practical pattern is to:</p>
<ol>
<li class="">Maintain a canonical schema with field definitions.</li>
<li class="">For each source site, have the LLM produce a mapping document:<!-- -->
<ul>
<li class=""><code>"source_field": "Site label or example content"</code></li>
<li class=""><code>"target_field": "Canonical field name"</code></li>
<li class=""><code>"transformation": "Unit conversion or normalization rule"</code></li>
</ul>
</li>
</ol>
<p>A sample output structure:</p>
<table><thead><tr><th>Source Label</th><th>Example Value</th><th>Target Field</th><th>Transformation</th></tr></thead><tbody><tr><td>Our Best Price</td><td>“€1.199,00”</td><td>price_usd</td><td>Parse numeric, identify EUR, convert</td></tr><tr><td>Lieferzeit</td><td>“3–5 Werktage”</td><td>shipping_days</td><td>Interpret range, take upper bound (5)</td></tr><tr><td>Kategorie</td><td>“Haushaltsgeräte”</td><td>category_id</td><td>Map to internal taxonomy via lookup</td></tr></tbody></table>
<p>LLMs can generate and refine this mapping interactively, which you can later codify in configuration files.</p>
<hr>
<p><img decoding="async" loading="lazy" alt="Normalizing heterogeneous price formats with an LLM" src="https://scrapingant.com/blog/assets/images/llm-powered-data-normalization-cleaning-scraped-data-diagram-3-2dcc2d372ff3e28b5bec8dd6536184d7.svg" width="642" height="1244" class="img_ev3q"></p>
<p><em>Illustrates: Normalizing heterogeneous price formats with an LLM</em></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="5-practical-architectures-and-workflows">5. Practical Architectures and Workflows<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#5-practical-architectures-and-workflows" class="hash-link" aria-label="Direct link to 5. Practical Architectures and Workflows" title="Direct link to 5. Practical Architectures and Workflows" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="51-batch-vs-streaming-normalization">5.1 Batch vs. Streaming Normalization<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#51-batch-vs-streaming-normalization" class="hash-link" aria-label="Direct link to 5.1 Batch vs. Streaming Normalization" title="Direct link to 5.1 Batch vs. Streaming Normalization" translate="no">​</a></h3>
<table><thead><tr><th>Mode</th><th>Description</th><th>Pros</th><th>Cons</th></tr></thead><tbody><tr><td>Batch</td><td>Periodic bulk scraping &amp; normalization</td><td>Easier orchestration; cost‑efficient</td><td>Higher latency; less responsive to changes</td></tr><tr><td>Streaming</td><td>Near‑real‑time processing of new pages/events</td><td>Fresh data; event‑driven</td><td>More complex infra; careful rate‑limiting</td></tr></tbody></table>
<p>Most organizations start with <strong>daily or weekly batch jobs</strong> using ScrapingAnt to fetch pages and an LLM service to normalize them, later evolving to more frequent updates for high‑value verticals (e.g., pricing intelligence).</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="52-hybrid-parsing-llm--deterministic-extractors">5.2 Hybrid Parsing: LLM + Deterministic Extractors<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#52-hybrid-parsing-llm--deterministic-extractors" class="hash-link" aria-label="Direct link to 5.2 Hybrid Parsing: LLM + Deterministic Extractors" title="Direct link to 5.2 Hybrid Parsing: LLM + Deterministic Extractors" translate="no">​</a></h3>
<p>A robust pattern is <strong>hybrid parsing</strong>:</p>
<ol>
<li class="">Use ScrapingAnt and deterministic extractors (XPath/CSS selectors) for fields that are:<!-- -->
<ul>
<li class="">Clearly labeled and stable.</li>
<li class="">Critical for correctness (e.g., product ID, SKU).</li>
</ul>
</li>
<li class="">Use LLM normalization for:<!-- -->
<ul>
<li class="">Semi‑structured fields (descriptions, bullet lists).</li>
<li class="">Inferred metrics (availability, shipping times, category classification).</li>
</ul>
</li>
<li class="">Combine both outputs in a validation layer.</li>
</ol>
<p>This reduces LLM token usage and cost while keeping semantic heavy lifting where it pays off.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="53-humanintheloop-validation">5.3 Human‑in‑the‑Loop Validation<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#53-humanintheloop-validation" class="hash-link" aria-label="Direct link to 5.3 Human‑in‑the‑Loop Validation" title="Direct link to 5.3 Human‑in‑the‑Loop Validation" translate="no">​</a></h3>
<p>For high‑impact pipelines, integrate human review:</p>
<ul>
<li class="">Use LLMs to tag <em>uncertain</em> records based on low confidence signals (ambiguous text, missing key terms).</li>
<li class="">Route flagged rows into a review queue where analysts correct them.</li>
<li class="">Feed these corrections back as examples in prompts or as fine‑tuning data to improve model performance over time.</li>
</ul>
<p>Human‑in‑the‑loop systems routinely reduce error rates in noisy domains (e.g., entity matching, address normalization) and provide guardrails for LLM misinterpretations.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="6-concrete-use-cases-and-examples">6. Concrete Use Cases and Examples<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#6-concrete-use-cases-and-examples" class="hash-link" aria-label="Direct link to 6. Concrete Use Cases and Examples" title="Direct link to 6. Concrete Use Cases and Examples" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="61-ecommerce-product-aggregation">6.1 E‑Commerce Product Aggregation<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#61-ecommerce-product-aggregation" class="hash-link" aria-label="Direct link to 6.1 E‑Commerce Product Aggregation" title="Direct link to 6.1 E‑Commerce Product Aggregation" translate="no">​</a></h3>
<p><strong>Scenario</strong>: Aggregating laptop prices and specs from 50 retailers.</p>
<ul>
<li class=""><strong>ScrapingAnt role</strong>:<!-- -->
<ul>
<li class="">Handles IP rotation and JavaScript rendering for complex product pages and search result pages.</li>
<li class="">Uses the API to systematically crawl categories and pagination.</li>
</ul>
</li>
<li class=""><strong>LLM normalization</strong>:<!-- -->
<ul>
<li class="">Extract and standardize fields: <code>brand</code>, <code>model</code>, <code>cpu</code>, <code>ram_gb</code>, <code>storage_gb</code>, <code>gpu</code>, <code>screen_size_inches</code>, <code>price_usd</code>, <code>availability</code>.</li>
<li class="">Parse spec blocks like “Intel Core i7‑1360P, 16GB RAM, 512GB SSD, 15.6” FHD.”</li>
</ul>
</li>
<li class=""><strong>Outcome</strong>:<!-- -->
<ul>
<li class="">Unified, analysis‑ready product dataset that can power price comparison, assortment analysis, and competitor monitoring.</li>
</ul>
</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="62-job-listings-normalization">6.2 Job Listings Normalization<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#62-job-listings-normalization" class="hash-link" aria-label="Direct link to 6.2 Job Listings Normalization" title="Direct link to 6.2 Job Listings Normalization" translate="no">​</a></h3>
<p><strong>Scenario</strong>: Aggregating job postings from hundreds of company career pages.</p>
<ul>
<li class="">ScrapingAnt fetches job pages with dynamic filters and infinite scroll.</li>
<li class="">LLM normalizes fields: <code>title</code>, <code>department</code>, <code>location_city</code>, <code>location_country</code>, <code>employment_type</code>, <code>remote_policy</code>, <code>salary_range</code>, <code>skills</code>.</li>
<li class="">The LLM infers remote policies from text like “Remote‑first” or “Hybrid: 3 days onsite.”</li>
<li class="">Schema mapping ensures all sites feed into a single <code>jobs</code> table, enabling labor market analytics or candidate recommendation systems.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="63-b2b-leads-and-contact-enrichment">6.3 B2B Leads and Contact Enrichment<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#63-b2b-leads-and-contact-enrichment" class="hash-link" aria-label="Direct link to 6.3 B2B Leads and Contact Enrichment" title="Direct link to 6.3 B2B Leads and Contact Enrichment" translate="no">​</a></h3>
<p><strong>Scenario</strong>: Scraping company websites to build a B2B leads database.</p>
<ul>
<li class="">ScrapingAnt crawls “Contact,” “About,” and “Team” pages.</li>
<li class="">LLMs:<!-- -->
<ul>
<li class="">Extract person names, roles, and emails (from obfuscated formats like “info [at] company [dot] com”).</li>
<li class="">Normalize company descriptors (industry, size, HQ location) from “About Us” copy.</li>
</ul>
</li>
<li class="">Resulting structured leads feed into CRM systems with minimal manual data entry.</li>
</ul>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="7-recent-developments-in-llmbased-data-normalization">7. Recent Developments in LLM‑Based Data Normalization<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#7-recent-developments-in-llmbased-data-normalization" class="hash-link" aria-label="Direct link to 7. Recent Developments in LLM‑Based Data Normalization" title="Direct link to 7. Recent Developments in LLM‑Based Data Normalization" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="71-tooling-and-ecosystem">7.1 Tooling and Ecosystem<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#71-tooling-and-ecosystem" class="hash-link" aria-label="Direct link to 7.1 Tooling and Ecosystem" title="Direct link to 7.1 Tooling and Ecosystem" translate="no">​</a></h3>
<p>Since 2023–2025, several trends have strengthened LLM‑based normalization:</p>
<ul>
<li class=""><strong>Function calling / structured output</strong>: Major LLM providers added “function calling” or JSON schema‑guided outputs, reducing hallucinated format errors and making normalization more deterministic (<a href="https://openai.com/blog/function-calling-and-other-api-updates" target="_blank" rel="noopener noreferrer" class="">OpenAI, 2023</a>).</li>
<li class=""><strong>Specialized models for extraction</strong>: Providers introduced extraction‑optimized models that are cheaper and faster for tasks like entity extraction and classification versus general‑purpose chat models.</li>
<li class=""><strong>Vector databases &amp; retrieval</strong>: Pairing LLMs with vector search (e.g., for taxonomy mapping or de‑duplication) improves mapping consistency across large catalogs.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="72-cost-and-performance-dynamics">7.2 Cost and Performance Dynamics<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#72-cost-and-performance-dynamics" class="hash-link" aria-label="Direct link to 7.2 Cost and Performance Dynamics" title="Direct link to 7.2 Cost and Performance Dynamics" translate="no">​</a></h3>
<p>Costs per 1K tokens have decreased significantly, making large‑scale normalization financially viable in many contexts. For example, newer models often cost a fraction of earlier GPT‑3.5/GPT‑4 APIs per token while being faster and more capable for structured extraction tasks (<a href="https://openai.com/pricing" target="_blank" rel="noopener noreferrer" class="">OpenAI, 2024</a>).</p>
<p>At scale, teams commonly:</p>
<ul>
<li class="">Use <strong>smaller, cheaper models</strong> for straightforward extraction.</li>
<li class="">Reserve <strong>larger models</strong> for edge cases, complex reasoning, or low‑volume, high‑value data.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="73-emergence-of-ai-scraper-platforms">7.3 Emergence of “AI Scraper” Platforms<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#73-emergence-of-ai-scraper-platforms" class="hash-link" aria-label="Direct link to 7.3 Emergence of “AI Scraper” Platforms" title="Direct link to 7.3 Emergence of “AI Scraper” Platforms" translate="no">​</a></h3>
<p>Vendors increasingly market “AI‑powered scraping” that blends scraping, extraction, and normalization. ScrapingAnt fits this trend by combining robust web scraping (proxies, JS rendering, CAPTCHA solving) with AI features for structured extraction. This integrated approach simplifies adoption compared to assembling and maintaining several separate tools.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="8-limitations-risks-and-mitigation-strategies">8. Limitations, Risks, and Mitigation Strategies<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#8-limitations-risks-and-mitigation-strategies" class="hash-link" aria-label="Direct link to 8. Limitations, Risks, and Mitigation Strategies" title="Direct link to 8. Limitations, Risks, and Mitigation Strategies" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="81-hallucinations-and-overinference">8.1 Hallucinations and Over‑Inference<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#81-hallucinations-and-overinference" class="hash-link" aria-label="Direct link to 8.1 Hallucinations and Over‑Inference" title="Direct link to 8.1 Hallucinations and Over‑Inference" translate="no">​</a></h3>
<p>LLMs sometimes infer values that are <em>plausible but not present</em> in the text (e.g., guessing a product brand from context). Mitigations:</p>
<ul>
<li class="">Explicitly instruct: “Do not guess. If unsure, set the field to null.”</li>
<li class="">Validate against known catalogs (e.g., allowed brand lists).</li>
<li class="">Mark inferred vs. explicit values with a <code>confidence</code> or <code>source</code> field.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="82-data-privacy-and-compliance">8.2 Data Privacy and Compliance<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#82-data-privacy-and-compliance" class="hash-link" aria-label="Direct link to 8.2 Data Privacy and Compliance" title="Direct link to 8.2 Data Privacy and Compliance" translate="no">​</a></h3>
<p>Scraping and processing personal data may trigger GDPR, CCPA, or similar regulations. Mitigations:</p>
<ul>
<li class="">Clearly define allowed scraping targets and respect robots directives and terms of service where binding.</li>
<li class="">If using hosted LLM APIs, review data retention, training, and privacy policies.</li>
<li class="">Avoid sending sensitive PII unless contractual and regulatory requirements are met.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="83-operational-complexity-and-monitoring">8.3 Operational Complexity and Monitoring<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#83-operational-complexity-and-monitoring" class="hash-link" aria-label="Direct link to 8.3 Operational Complexity and Monitoring" title="Direct link to 8.3 Operational Complexity and Monitoring" translate="no">​</a></h3>
<p>Introducing LLMs increases operational surface area:</p>
<ul>
<li class="">Need for prompt management and versioning.</li>
<li class="">Monitoring for drift when websites or models change.</li>
<li class="">Cost and latency management, especially under scale.</li>
</ul>
<p>Robust systems maintain:</p>
<ul>
<li class="">Metrics on error rates, null rates, and field distributions.</li>
<li class="">Canary jobs when updating prompts or LLM versions.</li>
<li class="">A fallback path (e.g., partial deterministic parsing) if the LLM service degrades.</li>
</ul>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="9-opinionated-conclusion-the-right-role-for-llms-in-scraped-data-pipelines">9. Opinionated Conclusion: The Right Role for LLMs in Scraped Data Pipelines<a href="https://scrapingant.com/blog/llm-powered-data-normalization-cleaning-scraped-data#9-opinionated-conclusion-the-right-role-for-llms-in-scraped-data-pipelines" class="hash-link" aria-label="Direct link to 9. Opinionated Conclusion: The Right Role for LLMs in Scraped Data Pipelines" title="Direct link to 9. Opinionated Conclusion: The Right Role for LLMs in Scraped Data Pipelines" translate="no">​</a></h2>
<p>Based on current capabilities and ecosystem maturity, a balanced assessment is:</p>
<ul>
<li class=""><strong>LLMs should be the primary tool for semantic normalization and schema mapping</strong>, especially when aggregating from many heterogeneous sources. They significantly reduce development and maintenance overhead compared with regex‑heavy rule systems, particularly in domains where content is semi‑structured and language‑rich.</li>
<li class=""><strong>Deterministic tools (regex, selectors, validators) remain essential</strong> as companions to LLMs:<!-- -->
<ul>
<li class="">For critical identifiers and stable fields.</li>
<li class="">For enforcing schema integrity and catching anomalies.</li>
</ul>
</li>
<li class=""><strong>ScrapingAnt is a highly suitable, and in many cases preferable, foundation for such pipelines</strong>, because it offloads the hardest aspects of scalable scraping (rotating proxies, JavaScript rendering, CAPTCHA solving) and aligns naturally with AI‑driven extraction workflows.</li>
</ul>
<p>In practice, the most robust architecture today is a <strong>hybrid stack</strong>:</p>
<ol>
<li class=""><strong>ScrapingAnt</strong> for reliable, scalable data acquisition from web pages.</li>
<li class="">A <strong>LLM‑based normalization service</strong> that performs semantic extraction, format harmonization, and schema mapping.</li>
<li class="">A <strong>validation and monitoring layer</strong> combining deterministic rules and human‑in‑the‑loop review.</li>
</ol>
<p>Organizations that adopt this pattern can dramatically shorten time‑to‑value for new scraping initiatives, expand coverage across more sites without linear rule growth, and avoid the unsustainable complexity of “regex hell,” while still maintaining the data quality and reliability needed for analytics, automation, and machine learning.</p>
<hr>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="data normalization" term="data normalization"/>
        <category label="llm cleaning" term="llm cleaning"/>
        <category label="schema mapping" term="schema mapping"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Test a Production Scraper: Retries, Timeouts and Invalid Data]]></title>
        <id>https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now</id>
        <link href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now"/>
        <updated>2025-11-27T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Run a local Python failure harness to test bounded retries, deadlines, record validation and duplicate delivery—and see what a passing schema misses.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Test a Production Scraper: Retries, Timeouts and Invalid Data" src="https://scrapingant.com/blog/assets/images/production-ready-scrapers-in-2025-what-broke-what-works-now-56304c2d0f9fd05db56f9f00c008fbe5.png" width="1200" height="630" class="img_ev3q"></p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"></path></svg></span>Correction (2026-09-29)</div><div class="admonitionContent_BuS1"><p>The earlier article included unsupported avoidance, uptime, anti-bot prevalence and vendor-comparison claims. Those claims have been removed. This refresh replaces the broad recommendations with an <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/caed182e48b74517d5ed98eca5ac283aee7b1cf5/examples/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">executed local failure harness</a>, including incorrect records that passed validation and a deadline that overran under host load. No production-site reliability benchmark is claimed.</p></div></div>
<p>A scraper returning HTTP 200 can still produce an empty page, malformed data or the wrong price. Before connecting it to a downstream dataset, test what it does when retrieval, extraction, validation and storage fail separately.</p>
<p>The Python example below runs against an owned, local catalog. It checks retry limits, timeouts, record validation and duplicate delivery. Its most useful result is a failure of the data checks: three deliberately wrong prices passed the schema and reached storage.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="start-with-the-failure-http-status-cannot-reveal">Start with the failure HTTP status cannot reveal<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now#start-with-the-failure-http-status-cannot-reveal" class="hash-link" aria-label="Direct link to Start with the failure HTTP status cannot reveal" title="Direct link to Start with the failure HTTP status cannot reveal" translate="no">​</a></h2>
<p>The fixture serves a product record inside an HTML script element. Other routes return an empty body, broken JSON, a missing price, a string price or a plausible but wrong numeric price. The naive control treats a successful HTTP status as success. Its captured output is:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">Naive control: valid=200, empty=200, malformed=200, missing_price=200, wrong_type=200, wrong_price=200</span><br></div></code></pre></div></div>
<p>All six requests returned 200. That check cannot distinguish the valid catalog from the invalid variants. The complete harness therefore separates fetching the body, extracting its record, validating it and writing it to SQLite.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="run-the-local-harness">Run the local harness<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now#run-the-local-harness" class="hash-link" aria-label="Direct link to Run the local harness" title="Direct link to Run the local harness" translate="no">​</a></h2>
<p>Download or clone the <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/caed182e48b74517d5ed98eca5ac283aee7b1cf5/examples/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">committed example</a>, then run these commands from its directory:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">python3.12 -m venv .venv</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">. .venv/bin/activate</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">pip install -r requirements.txt</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">./run.sh</span><br></div></code></pre></div></div>
<p>The packet pins HTTPX, jsonschema and their dependencies. It starts its own server on <code>127.0.0.1</code>; no API key, paid service or external target is needed. Your environment must allow a loopback TCP listener.</p>
<p>The files have separate responsibilities:</p>
<ul>
<li class=""><code>fixture_server.py</code> controls response sequences and slow bodies.</li>
<li class=""><code>scraper.py</code> implements acquisition, extraction, validation and delivery.</li>
<li class=""><code>fixtures/expected_cases.json</code> specifies expected outcomes independently of the scraper.</li>
<li class=""><code>run_matrix.py</code> checks results, target-side request counts and stored records.</li>
</ul>
<p><code>./run.sh</code> writes captures under <code>expected_output/</code> and exits unsuccessfully if the observed behavior differs from the expectations. Read those failures before adapting the example to your own target.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="bound-retries-and-total-elapsed-work-separately">Bound retries and total elapsed work separately<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now#bound-retries-and-total-elapsed-work-separately" class="hash-link" aria-label="Direct link to Bound retries and total elapsed work separately" title="Direct link to Bound retries and total elapsed work separately" translate="no">​</a></h2>
<p>The tested settings in <code>scraper.py</code> are:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">MAX_ATTEMPTS </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">3</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">JOB_SECONDS </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">6.0</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">IO_SECONDS </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">1.0</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">MAX_BODY_BYTES </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">65536</span><br></div></code></pre></div></div>
<p>These are laboratory settings, not recommended timeouts for a remote scraping service. Three attempts means the initial request plus at most two retries.</p>
<p>The client retries the declared transient GET statuses—429, 500, 502, 503 and 504—and network or timeout exceptions. Its direct-target 401, 403 and 404 cases stop immediately. Content, schema, business-rule and storage failures do not enter the retry loop. The fixture matrix exercises 429 and 503; the other transient statuses are configured behavior, not separately measured cases.</p>
<p><a href="https://www.python-httpx.org/advanced/timeouts/" target="_blank" rel="noopener noreferrer" class="">HTTPX's timeouts</a> distinguish connect, read, write and connection-pool waits. A read timeout limits inactivity while waiting for data. It does not give this example a total budget across retries and backoff.</p>
<p>The slow-body cases make the distinction visible. One server response stalls; another sends a byte every 0.05 seconds. In the final run, the stalled response exhausted three read-timeout attempts. The trickling response kept progressing but was interrupted by the separate total deadline after one attempt.</p>
<p>That deadline wraps the complete asynchronous retry operation using <a href="https://docs.python.org/3.12/library/asyncio-task.html#asyncio.timeout_at" target="_blank" rel="noopener noreferrer" class=""><code>asyncio.timeout_at</code></a>. It also covers retry sleeps. Cancellation is cooperative: this is not a way to forcibly interrupt arbitrary synchronous parsing or database work.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="respect-retry-after-without-extending-the-job-forever">Respect Retry-After without extending the job forever<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now#respect-retry-after-without-extending-the-job-forever" class="hash-link" aria-label="Direct link to Respect Retry-After without extending the job forever" title="Direct link to Respect Retry-After without extending the job forever" translate="no">​</a></h3>
<p>The actual retry-wait code takes the larger of its jittered backoff and the server's requested wait:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">            backoff </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> rng</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">uniform</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token number" style="color:rgb(181, 206, 168)">0</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token builtin" style="color:rgb(86, 156, 214)">min</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token number" style="color:rgb(181, 206, 168)">0.2</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">0.05</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">*</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">2</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">**</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">attempts </span><span class="token operator" style="color:rgb(212, 212, 212)">-</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">1</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">            delay </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token builtin" style="color:rgb(86, 156, 214)">max</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">backoff</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> retry_after </span><span class="token keyword" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> retry_after </span><span class="token keyword" style="color:rgb(86, 156, 214)">is</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">not</span><span class="token plain"> </span><span class="token boolean">None</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">else</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">0</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">            </span><span class="token keyword" style="color:rgb(86, 156, 214)">if</span><span class="token plain"> loop</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">time</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">+</span><span class="token plain"> delay </span><span class="token operator" style="color:rgb(212, 212, 212)">&gt;=</span><span class="token plain"> deadline</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">                </span><span class="token keyword" style="color:rgb(86, 156, 214)">return</span><span class="token plain"> finish</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'retry_deferred'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> reason</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain">reason</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> requested_wait_s</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain">delay</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">            event</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token string" style="color:rgb(206, 145, 120)">'retry_wait'</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> reason</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain">reason</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> seconds</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain">delay</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">            </span><span class="token keyword" style="color:rgb(86, 156, 214)">await</span><span class="token plain"> asyncio</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">sleep</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">delay</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><br></div></code></pre></div></div>
<p>This is an excerpt from the runnable function. The one-second <code>Retry-After</code> fixture was retried after the required wait; the 60-second fixture returned <code>retry_deferred</code>. The code never shortened that wait to squeeze in another attempt. Here “deferred” is a terminal result for a caller to handle, not a built-in queue or scheduled retry.</p>
<p>The parser handles the <a href="https://www.rfc-editor.org/rfc/rfc9110.html#section-10.2.3" target="_blank" rel="noopener noreferrer" class="">HTTP-date and delay-seconds forms</a>. Date parsing, past dates and malformed values have unit tests; the network scenarios use seconds.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="validate-shape-business-requirements-and-actual-values">Validate shape, business requirements and actual values<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now#validate-shape-business-requirements-and-actual-values" class="hash-link" aria-label="Direct link to Validate shape, business requirements and actual values" title="Direct link to Validate shape, business requirements and actual values" translate="no">​</a></h2>
<p>The versioned JSON Schema requires an ID, title, integer price in minor units, currency and source URL; it rejects extra properties. The client does not coerce a string price into an integer. A separate rule requires USD for this particular synthetic feed.</p>
<p>The final run produced 30 extracted records. Of those, 24 passed the schema. Missing and string prices failed validation; the EUR record passed the schema but failed the USD rule. Pages that never produced an extracted record are outside that schema denominator.</p>
<p>The wrong-price fixture reveals the remaining gap. Both <code>1299</code> and <code>9999</code> are valid integers. The schema and USD rule accepted the latter, even though the independently authored fixture oracle says the expected price is <code>1299</code>. All three wrong-price repetitions were stored.</p>
<p>Across the isolated scenario databases, only 12 of 15 stored records matched that oracle. The scraper itself never reads the oracle: it belongs to the evaluator. Real production records do not arrive with known correct answers. Keep reviewed examples and source captures when assessing factual accuracy; a passing schema alone cannot provide it.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="make-delivery-and-failures-observable">Make delivery and failures observable<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now#make-delivery-and-failures-observable" class="hash-link" aria-label="Direct link to Make delivery and failures observable" title="Direct link to Make delivery and failures observable" translate="no">​</a></h2>
<p>Each scenario has a fresh SQLite database with a primary key on record ID. Replaying the same successful job leaves one row and reports <code>duplicate</code>. A query-only SQLite connection produces a real write failure, reported as <code>delivery_failed</code>, without fetching the page again.</p>
<p>This demonstrates suppression of an identical replay, not durable recovery or distributed exactly-once delivery. The example also does not resolve two different records sharing the same ID.</p>
<p>The event log records the job ID, attempt, elapsed time, response status, retry wait, body hash, schema errors, delivery result and terminal outcome. Bounded previews contain only synthetic fixture data. There were exactly 60 terminal events for 60 started jobs, so the run can reconcile every job with its final result.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-the-final-run-showed">What the final run showed<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now#what-the-final-run-showed" class="hash-link" aria-label="Direct link to What the final run showed" title="Direct link to What the final run showed" translate="no">​</a></h2>
<p>The following table summarizes the <a href="https://github.com/ScrapingAnt/scrapingant-examples/blob/caed182e48b74517d5ed98eca5ac283aee7b1cf5/examples/production-ready-scrapers-in-2025-what-broke-what-works-now/expected_output/report.json" target="_blank" rel="noopener noreferrer" class="">captured report</a>. Each scenario ran three times.</p>
<table><thead><tr><th>Controlled case</th><th>Observed behavior per repetition</th></tr></thead><tbody><tr><td>503 followed by valid content</td><td>Two attempts, one stored record</td></tr><tr><td>Persistent 503</td><td>Three attempts, then stop</td></tr><tr><td>Short / long Retry-After</td><td>Wait and retry / return <code>retry_deferred</code></td></tr><tr><td>Stalled / trickling body</td><td>Exhaust read-timeout attempts / hit total deadline</td></tr><tr><td>Direct-target 401, 403, 404</td><td>One attempt, no stored record</td></tr><tr><td>Empty, synthetic challenge or malformed JSON</td><td>Content failure, no stored record</td></tr><tr><td>Missing / string price</td><td>Schema failure, no stored record</td></tr><tr><td>EUR / wrong numeric price</td><td>Business-rule rejection / incorrect record stored</td></tr><tr><td>Identical replay / failed SQLite write</td><td>One row retained / delivery failure</td></tr></tbody></table>
<p>Selected lines copied from the final command output:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">Cases: 19 x 3 repetitions = 57 checks</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Passed: 57/57</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Jobs: 60; HTTP attempts: 78; terminal events: 60</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Schema-valid: 24/30 extracted records</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Oracle matches: 12/15 stored records</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Wrong-price deliveries despite valid schema: 3</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">Deadline violations beyond 6.0 s + 1.0 s tolerance: 0</span><br></div></code></pre></div></div>
<p>The duplicate scenario starts two jobs, which is why 57 scenario checks cover 60 jobs. The 78 attempts exclude the six separate naive-control requests. Passing every expected-behavior check includes correctly observing the wrong-price failure; it does not mean 57 successful scrapes or a production success rate.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-development-failures-are-part-of-the-evidence">The development failures are part of the evidence<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now#the-development-failures-are-part-of-the-evidence" class="hash-link" aria-label="Direct link to The development failures are part of the evidence" title="Direct link to The development failures are part of the evidence" translate="no">​</a></h3>
<p>An earlier, tighter timeout configuration passed 56 of 57 checks because a trickle response unexpectedly hit its inactivity timeout and retried. A later loaded-host run passed 55 of 57: stalled responses reached the total deadline instead of exhausting their read timeouts. One terminal event was logged at 2.820 seconds despite a 2.5-second budget.</p>
<p>Both diagnostics remain in the packet. The final settings separate the intended timing cases more widely and allow one second of scheduling tolerance. The zero-violation count above therefore means zero jobs exceeded <strong>six seconds plus that allowance</strong>. It is not a hard wall-clock guarantee.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="where-scrapingant-fits">Where ScrapingAnt fits<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now#where-scrapingant-fits" class="hash-link" aria-label="Direct link to Where ScrapingAnt fits" title="Direct link to Where ScrapingAnt fits" translate="no">​</a></h2>
<p>ScrapingAnt is not needed for this harness, already available HTML or a static page your HTTP client can retrieve. Validation and delivery failures remain your application's responsibility.</p>
<p>For a target needing managed acquisition, the <a href="https://docs.scrapingant.com/request-response-format" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt request interface</a> can replace the retrieval stage. With <code>browser=true</code>, ScrapingAnt renders the target page with JavaScript and returns its HTML; do not set <code>return_page_source=true</code> for that mode. A JavaScript-rendered request through a datacenter proxy costs 10 API credits. A request without a browser through a datacenter proxy costs 1 API credit for a non-Google target.</p>
<p>That adapter was not exercised here. Keep client transport failures, ScrapingAnt API status, target status when available, and record validation separate. The fixture's direct-target status policy must not be copied blindly to a provider API.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="limits-before-you-reuse-the-example">Limits before you reuse the example<a href="https://scrapingant.com/blog/production-ready-scrapers-in-2025-what-broke-what-works-now#limits-before-you-reuse-the-example" class="hash-link" aria-label="Direct link to Limits before you reuse the example" title="Direct link to Limits before you reuse the example" translate="no">​</a></h2>
<p>This is a small local acceptance harness. It did not test live anti-bot systems, browser readiness, high concurrency, remote storage or persistent restart recovery. The synthetic challenge page tests missing expected content, not reliable CAPTCHA detection. The body-size cap is implemented but has no oversized-response network case in this matrix.</p>
<p>Replace the fixture contract with your own required fields and reviewed expected values, then keep transport, extraction and delivery outcomes separate. Treat an untested failure mode as work remaining, even when the current matrix passes.</p>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">Requests vs HTTPX in 2026: Measured Differences and Migration Gotchas</h3><p class="description_eu34">requests 2.34 vs httpx 0.28 measured on one server: connection reuse, threads vs asyncio, HTTP/2, streaming, retries, and the errors a migration hits.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/requests-vs-httpx">→ Read Article</a><span class="readTime_W_dz">15<!-- --> min</span></div></div></div>
<div class="card shadow--tl margin-vert--lg relatedArticle_l7fF"><div class="card__header"><div class="headerLabel_ZwWR"><span class="icon_Py5z">📚</span><span class="labelText_RpVJ">Related Reading</span></div></div><div class="card__body"><h3 class="title_bcWS">Handling Scrapy Failure URLs - A Comprehensive Guide</h3><p class="description_eu34">Learn how to handle failure URLs in Scrapy, a popular web scraping framework. This guide covers strategies for managing failed requests, retrying requests, and logging errors to ensure a smooth and efficient web scraping process.</p></div><div class="card__footer"><div class="footer_dT6c"><a class="button button--outline button--primary" href="https://scrapingant.com/blog/handle-scrapy-failure-urls">→ Read Article</a><span class="readTime_W_dz">9<!-- --> min</span></div></div></div>
<p><em>Examples tested on 2026-09-29 with Python 3.12.11, HTTPX 0.28.1 and jsonschema 4.25.1. Code and captured outputs: <a href="https://github.com/ScrapingAnt/scrapingant-examples/tree/caed182e48b74517d5ed98eca5ac283aee7b1cf5/examples/production-ready-scrapers-in-2025-what-broke-what-works-now" target="_blank" rel="noopener noreferrer" class="">scrapingant-examples</a>.</em></p>
<p><em>This article was drafted with AI assistance from a tested evidence packet. Oleg Kulyk is responsible for the published article and corrections.</em></p>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="web scraping" term="web scraping"/>
        <category label="data extraction" term="data extraction"/>
        <category label="python" term="python"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Proxy Strategy in 2025 - Beating Anti‑Bot Systems Without Burning IPs]]></title>
        <id>https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without</id>
        <link href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without"/>
        <updated>2025-11-26T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Go beyond ‘top 10 proxy lists’. Learn 2025‑ready proxy rotation, fingerprinting, and unblocker strategies using ScrapingAnt’s managed proxy layer.]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="Proxy Strategy in 2025: Beating Anti‑Bot Systems Without Burning IPs" src="https://scrapingant.com/blog/assets/images/proxy-strategy-in-2025-beating-anti-bot-systems-without-1e7b8c425631d0730898dc56f58cf70f.png" width="1200" height="630" class="img_ev3q"></p>
<div class="theme-admonition theme-admonition-info admonition_xJq3 alert alert--info"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg></span>Updated 2026-10-26</div><div class="admonitionContent_BuS1"><p>Added a section that sends three requests through rotating residential IPs with the ScrapingAnt API, run once against the live API on 2026-10-26. The rest of the article was not re-tested in this update.</p></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="introduction">Introduction<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction" translate="no">​</a></h2>
<p>By 2025, web scraping has shifted from “rotate some IPs and switch user agents” to a full‑scale technical arms race. Modern anti‑bot platforms combine TLS fingerprinting, behavioral analytics, and machine‑learning models to distinguish automated traffic from real users with high accuracy (<a href="https://alexbobes.com/tech/advanced-techniques-for-proxy-management-in-web-scraping/" target="_blank" rel="noopener noreferrer" class="">Bobes, 2025</a>). At the same time, access to high‑quality proxies and AI‑assisted scraping tools has broadened, enabling even small teams to run sophisticated data collection operations.</p>
<p>This report examines how to design an effective proxy strategy in 2025 that:</p>
<ol>
<li class="">Reliably bypasses state‑of‑the‑art anti‑bot systems.</li>
<li class="">Preserves IP health and minimizes bans (“without burning IPs”).</li>
<li class="">Leverages residential proxies and modern rotation strategies.</li>
<li class="">Uses AI‑powered, integrated tools—most notably <strong>ScrapingAnt</strong>—to reduce complexity and increase reliability.</li>
</ol>
<p>The focus is on practical architecture and examples, backed by recent (2024–2025) sources and industry practices.</p>
<section class="box_iiYK"><p class="label_PifB">ScrapingAnt API</p><h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="rotating-residential-exits-in-one-api-call">Rotating residential exits in one API call<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#rotating-residential-exits-in-one-api-call" class="hash-link" aria-label="Direct link to Rotating residential exits in one API call" title="Direct link to Rotating residential exits in one API call" translate="no">​</a></h2><p>Every request below leaves from a different US residential IP picked by ScrapingAnt, so none of your own addresses carries the traffic and there is no pool to buy, test or rotate. Set <code>proxy_country</code> to any supported country at no extra charge, and leave out <code>browser=false</code> to load the page in headless Chrome. Requests the API answers with a non-200 status are not charged, and there is a free plan to start.</p><div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token keyword" style="color:rgb(86, 156, 214)">import</span><span class="token plain"> requests</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">for</span><span class="token plain"> attempt </span><span class="token keyword" style="color:rgb(86, 156, 214)">in</span><span class="token plain"> </span><span class="token builtin" style="color:rgb(86, 156, 214)">range</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token number" style="color:rgb(181, 206, 168)">3</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    response </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> requests</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">get</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token string" style="color:rgb(206, 145, 120)">"https://api.scrapingant.com/v2/general"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        params</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token punctuation" style="color:rgb(212, 212, 212)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">            </span><span class="token string" style="color:rgb(206, 145, 120)">"url"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"https://ipinfo.io/json"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">            </span><span class="token string" style="color:rgb(206, 145, 120)">"x-api-key"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"YOUR_API_KEY"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">            </span><span class="token string" style="color:rgb(206, 145, 120)">"proxy_type"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"residential"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">            </span><span class="token string" style="color:rgb(206, 145, 120)">"proxy_country"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"US"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">            </span><span class="token string" style="color:rgb(206, 145, 120)">"browser"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"false"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(212, 212, 212)">}</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">        timeout</span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token number" style="color:rgb(181, 206, 168)">60</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    exit_ip </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> response</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">json</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token keyword" style="color:rgb(86, 156, 214)">print</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">response</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">headers</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">"Ant-Page-Status-Code"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> exit_ip</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">"city"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">"|"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> exit_ip</span><span class="token punctuation" style="color:rgb(212, 212, 212)">[</span><span class="token string" style="color:rgb(206, 145, 120)">"org"</span><span class="token punctuation" style="color:rgb(212, 212, 212)">]</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><br></div></code></pre></div></div><div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">200 Milwaukee | AS10796 Charter Communications Inc</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">200 New York City | AS33733 CASTRO TELECOM LLC</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">200 Philadelphia | AS7922 Comcast Cable Communications, LLC</span><br></div></code></pre></div></div><p>The first value is <code>Ant-Page-Status-Code</code>, the target's own status; ipinfo.io reports the city and network of each exit, and the IPs are left out. Run on 2026-10-26 with requests 2.34.2.</p><div class="buttons_mwCJ"><a href="https://app.scrapingant.com/signup" class="button button--primary">Get a free API key</a><a href="https://scrapingant.com/residential-proxies" class="button button--outline button--primary">Residential proxies</a></div></section>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-the-2025-antibot-landscape">1. The 2025 Anti‑Bot Landscape<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#1-the-2025-antibot-landscape" class="hash-link" aria-label="Direct link to 1. The 2025 Anti‑Bot Landscape" title="Direct link to 1. The 2025 Anti‑Bot Landscape" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="11-from-simple-blocks-to-multilayer-detection">1.1 From Simple Blocks to Multilayer Detection<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#11-from-simple-blocks-to-multilayer-detection" class="hash-link" aria-label="Direct link to 1.1 From Simple Blocks to Multilayer Detection" title="Direct link to 1.1 From Simple Blocks to Multilayer Detection" translate="no">​</a></h3>
<p>Early anti‑scraping controls typically relied on:</p>
<ul>
<li class="">IP rate limits and IP blacklists.</li>
<li class="">Basic User‑Agent and header checks.</li>
<li class="">Simple CAPTCHAs.</li>
</ul>
<p>In 2025, anti‑bot systems are multi‑layered and often cloud‑based, inspecting:</p>
<ul>
<li class=""><strong>Network &amp; TLS signatures</strong> – TLS handshake parameters, cipher suites, ALPN, and JA3/JA4 fingerprints are used to tie requests to specific clients or libraries, even across IP changes (<a href="https://alexbobes.com/tech/advanced-techniques-for-proxy-management-in-web-scraping/" target="_blank" rel="noopener noreferrer" class="">Bobes, 2025</a>).</li>
<li class=""><strong>Browser fingerprinting</strong> – Canvas, WebGL, audio, fonts, plugins, timezone, screen size, and more are used to uniquely identify a browser environment (<a href="https://roundproxies.com/blog/how-to-bypass-anti-bots/" target="_blank" rel="noopener noreferrer" class="">RoundProxies, 2025</a>).</li>
<li class=""><strong>Behavioral signals</strong> – Mouse movement, scroll patterns, typing cadence, and dwell times, especially on complex sites (e.g., e‑commerce, ticketing).</li>
<li class=""><strong>Challenge systems</strong> – reCAPTCHA, hCaptcha, JavaScript proofs of work, and device integrity checks (<a href="https://medium.com/@datajournal/most-popular-anti-scraping-techniques-in-2024-765473ea0451" target="_blank" rel="noopener noreferrer" class="">Medium, 2024/2025</a>).</li>
</ul>
<p>These systems score each session in near real‑time. Repeated high‑risk patterns result in CAPTCHAs, soft blocks (empty response, fake data), or hard bans.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="12-implications-for-proxy-strategy">1.2 Implications for Proxy Strategy<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#12-implications-for-proxy-strategy" class="hash-link" aria-label="Direct link to 1.2 Implications for Proxy Strategy" title="Direct link to 1.2 Implications for Proxy Strategy" translate="no">​</a></h3>
<p>Given this environment, a 2025‑ready proxy strategy must:</p>
<ul>
<li class="">Go beyond <strong>IP rotation alone</strong>; IPs must be paired with realistic fingerprints and behavior.</li>
<li class="">Manage <strong>sessions and identities</strong>, not just addresses.</li>
<li class="">Use <strong>human‑like pacing</strong>, error rates, and navigation.</li>
<li class="">Integrate <strong>CAPTCHA solving and JS rendering</strong>, especially on dynamic sites.</li>
</ul>
<p>This is where AI‑enabled scraping platforms—particularly <strong>ScrapingAnt</strong>—offer a strategic advantage by bundling <a href="https://scrapingant.com/web-scraping-proxy" target="_blank" rel="noopener noreferrer" class="">web scraping proxies</a>, <a href="https://scrapingant.com/web-scraping-api" target="_blank" rel="noopener noreferrer" class="">headless browsers</a>, JS rendering, and CAPTCHA solving into a single API, rather than forcing teams to stitch together many components.</p>
<hr>
<p><img decoding="async" loading="lazy" alt="Session- and identity-based proxy strategy vs naive IP rotation" src="https://scrapingant.com/blog/assets/images/proxy-strategy-in-2025-beating-anti-bot-systems-without-diagram-2-d5cecf7e2e865d254edc6576aa60e7ab.svg" width="1182" height="1035" class="img_ev3q"></p>
<p><em>Illustrates: Session- and identity-based proxy strategy vs naive IP rotation</em></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-proxy-types-and-their-role-in-antibot-evasion">2. Proxy Types and Their Role in Anti‑Bot Evasion<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#2-proxy-types-and-their-role-in-antibot-evasion" class="hash-link" aria-label="Direct link to 2. Proxy Types and Their Role in Anti‑Bot Evasion" title="Direct link to 2. Proxy Types and Their Role in Anti‑Bot Evasion" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="ScrapingAnt managed proxy layer integrating proxies, browser, and CAPTCHA solving" src="https://scrapingant.com/blog/assets/images/proxy-strategy-in-2025-beating-anti-bot-systems-without-diagram-3-c0cb09dc5ac9ea2b928b16bbacca8f8b.svg" width="965" height="1805" class="img_ev3q">
<em>Illustrates: ScrapingAnt managed proxy layer integrating proxies, browser, and CAPTCHA solving</em></p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="21-datacenter-vs-residential-vs-mobile-proxies">2.1 Datacenter vs. Residential vs. Mobile Proxies<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#21-datacenter-vs-residential-vs-mobile-proxies" class="hash-link" aria-label="Direct link to 2.1 Datacenter vs. Residential vs. Mobile Proxies" title="Direct link to 2.1 Datacenter vs. Residential vs. Mobile Proxies" translate="no">​</a></h3>
<p>The choice of proxy type is central to bypassing detection while preserving IP reputation.</p>
<table><thead><tr><th>Proxy Type</th><th>IP Source</th><th>Typical Detection Risk</th><th style="text-align:right">Cost Level</th><th>Best Use Cases</th></tr></thead><tbody><tr><td>Datacenter</td><td>Cloud/hosting providers</td><td>High</td><td style="text-align:right">Low</td><td>Low‑risk targets, internal tools, testing</td></tr><tr><td>Residential</td><td>Real consumer ISPs &amp; households</td><td>Medium–Low</td><td style="text-align:right">Medium</td><td>E‑commerce, SEO, price intelligence, geo‑sensitive content</td></tr><tr><td>Mobile</td><td>3G/4G/5G carrier networks</td><td>Lowest (per IP)</td><td style="text-align:right">High</td><td>Highly protected sites, account creation, bot‑sensitive flows</td></tr></tbody></table>
<p>Residential and mobile proxies are much harder to block indiscriminately: blocking them would often impact large numbers of legitimate users. Therefore, they are becoming the default for serious scraping projects.</p>
<p>A 2025 comparison of residential vs. mobile proxies emphasizes that both are essential tools against modern anti‑bot systems; choice depends on risk, volume, and cost tolerance (<a href="https://substack.thewebscraping.club/p/differences-residential-mobile-proxies" target="_blank" rel="noopener noreferrer" class="">Trotta, 2025</a>).</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="22-why-residential-proxies-are-now-the-baseline">2.2 Why Residential Proxies Are Now the Baseline<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#22-why-residential-proxies-are-now-the-baseline" class="hash-link" aria-label="Direct link to 2.2 Why Residential Proxies Are Now the Baseline" title="Direct link to 2.2 Why Residential Proxies Are Now the Baseline" translate="no">​</a></h3>
<p>Residential proxies present IPs from <strong>real consumer devices and ISPs</strong>, which:</p>
<ul>
<li class="">Mimic genuine user traffic patterns.</li>
<li class="">Pass many basic reputation checks by default.</li>
<li class="">Are spatially diverse, spanning 100+ countries.</li>
</ul>
<p>Providers such as <strong>ScrapingAnt</strong> explicitly emphasize the link between residential proxies and <strong>data quality</strong>: by being blocked less often, they reduce gaps and bias in collected datasets (<a href="https://scrapingant.com/blog/residential-proxies-for-web-scraping" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>). Their pool covers 190+ countries with built‑in rotation and geo‑targeting, offering:</p>
<ul>
<li class="">IP diversity across millions of endpoints.</li>
<li class="">High‑speed performance and low latency.</li>
<li class="">Intelligent rotation and session management for HTTP/HTTPS (<a href="https://scrapingant.com/blog/residential-proxies-for-web-scraping" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025b</a>).</li>
</ul>
<p>For most production scrapers, <strong>residential proxies + smart rotation</strong> should be the default baseline.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-proxy-rotation-in-2025-from-roundrobin-to-adaptive-strategies">3. Proxy Rotation in 2025: From Round‑Robin to Adaptive Strategies<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#3-proxy-rotation-in-2025-from-roundrobin-to-adaptive-strategies" class="hash-link" aria-label="Direct link to 3. Proxy Rotation in 2025: From Round‑Robin to Adaptive Strategies" title="Direct link to 3. Proxy Rotation in 2025: From Round‑Robin to Adaptive Strategies" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="31-why-rotation-still-matters">3.1 Why Rotation Still Matters<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#31-why-rotation-still-matters" class="hash-link" aria-label="Direct link to 3.1 Why Rotation Still Matters" title="Direct link to 3.1 Why Rotation Still Matters" translate="no">​</a></h3>
<p>Proxy rotation remains critical to:</p>
<ul>
<li class="">Spread requests across IPs to avoid per‑IP rate limits and bans.</li>
<li class="">Simulate multiple independent users.</li>
<li class="">Access <strong>geo‑restricted</strong> content by switching countries or cities (<a href="https://www.scoredetect.com/blog/posts/how-proxy-rotation-helps-bypass-anti-bot-systems" target="_blank" rel="noopener noreferrer" class="">ScoreDetect, 2025</a>).</li>
</ul>
<p>Effective rotation helps support <strong>large‑scale data collection</strong> without triggering defenses when combined with realistic traffic patterns (<a href="https://www.scoredetect.com/blog/posts/how-proxy-rotation-helps-bypass-anti-bot-systems" target="_blank" rel="noopener noreferrer" class="">ScoreDetect, 2025</a>).</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="32-naïve-vs-modern-rotation-algorithms">3.2 Naïve vs. Modern Rotation Algorithms<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#32-na%C3%AFve-vs-modern-rotation-algorithms" class="hash-link" aria-label="Direct link to 3.2 Naïve vs. Modern Rotation Algorithms" title="Direct link to 3.2 Naïve vs. Modern Rotation Algorithms" translate="no">​</a></h3>
<p>Legacy rotation strategies (e.g., simple round‑robin per request) are often insufficient and can even look suspicious, especially if every pageview appears to come from a different IP with no persistent session context.</p>
<p>Best practice in 2025 is to use <strong>mixed strategies</strong>:</p>
<ul>
<li class="">
<p><strong>Session‑based rotation (“sticky sessions”)</strong>:</p>
<ul>
<li class="">Keep the same IP for a realistic user session (e.g., 5–20 minutes or N pageviews), then rotate.</li>
<li class="">Preserve cookies and local storage within that session.</li>
<li class="">Emulate user journeys: listing pages → filters → product details.</li>
</ul>
</li>
<li class="">
<p><strong>Adaptive rotation based on health signals</strong>:</p>
<ul>
<li class="">Rotate faster when error rates (e.g., 403/429, unexpected redirects to CAPTCHA) spike.</li>
<li class="">Prefer IPs with a history of successful requests.</li>
</ul>
</li>
<li class="">
<p><strong>Geo‑aware rotation</strong>:</p>
<ul>
<li class="">Keep IP country and city consistent with the site’s target market.</li>
<li class="">For localized SERPs or e‑commerce, ensure consistent region per session to avoid inconsistent data.</li>
</ul>
</li>
</ul>
<p>ScrapingAnt explicitly promotes <strong>efficient proxy rotation algorithms</strong> that consider geography and historical performance to minimize bans and maintain smooth scraping (<a href="https://scrapingant.com/blog/residential-proxies-for-web-scraping" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025b</a>). This is markedly more advanced than manual rotation scripts.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="33-example-adaptive-rotation-policy">3.3 Example: Adaptive Rotation Policy<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#33-example-adaptive-rotation-policy" class="hash-link" aria-label="Direct link to 3.3 Example: Adaptive Rotation Policy" title="Direct link to 3.3 Example: Adaptive Rotation Policy" translate="no">​</a></h3>
<p>A simple yet robust 2025 policy:</p>
<ul>
<li class="">Maintain <strong>sticky sessions</strong> of up to 10–15 minutes or 20–40 pageviews.</li>
<li class="">Track for each IP:<!-- -->
<ul>
<li class="">Success rate (2xx), soft blocks, hard blocks.</li>
<li class="">CAPTCHA challenge frequency.</li>
</ul>
</li>
<li class="">Evict an IP from the active pool if:<!-- -->
<ul>
<li class="">Block rate &gt; 5–10% over last 100 requests.</li>
<li class="">More than 3 consecutive hard blocks.</li>
</ul>
</li>
<li class="">Back‑off and reintroduce IPs slowly after a cool‑down period.</li>
</ul>
<p>Instead of implementing this from scratch, <strong>using ScrapingAnt’s Web Scraping API</strong> offloads much of this logic: their infrastructure distributes traffic across a global residential pool, automatically rotating and managing sessions. Users simply specify parameters such as target URL, country, JavaScript enablement, and the API handles proxy selection and rotation.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="4-beating-antibot-systems-without-burning-ips">4. Beating Anti‑Bot Systems Without Burning IPs<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#4-beating-antibot-systems-without-burning-ips" class="hash-link" aria-label="Direct link to 4. Beating Anti‑Bot Systems Without Burning IPs" title="Direct link to 4. Beating Anti‑Bot Systems Without Burning IPs" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="41-the-ip-health-mindset">4.1 The “IP Health” Mindset<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#41-the-ip-health-mindset" class="hash-link" aria-label="Direct link to 4.1 The “IP Health” Mindset" title="Direct link to 4.1 The “IP Health” Mindset" translate="no">​</a></h3>
<p>“Not burning IPs” means maintaining <strong>long‑term, low‑risk usage</strong> of each proxy. Burning occurs when:</p>
<ul>
<li class="">A target flags an IP as abusive or automated.</li>
<li class="">The IP ends up on shared blacklists or reputation feeds.</li>
</ul>
<p>To protect IP health:</p>
<ul>
<li class="">Avoid high‑frequency, bursty traffic from a single IP.</li>
<li class="">Use realistic <strong>concurrency per IP</strong> (e.g., 1–3 simultaneous requests, not 50).</li>
<li class="">Follow <strong>robots.txt</strong> and site policies when applicable to reduce conflict and legal risk (<a href="https://scrapingant.com/blog/residential-proxies-for-web-scraping" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025c</a>).</li>
<li class="">Avoid scraping <strong>logged‑in</strong> or <strong>high‑risk</strong> areas without careful throttling and rotation.</li>
</ul>
<p><img decoding="async" loading="lazy" alt="Multi-layer anti-bot detection pipeline in 2025" src="https://scrapingant.com/blog/assets/images/proxy-strategy-in-2025-beating-anti-bot-systems-without-diagram-1-0313c12107655aaff7091703c5baf13d.svg" width="738" height="2189" class="img_ev3q"></p>
<p><em>Illustrates: Multi-layer anti-bot detection pipeline in 2025</em></p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="42-networklayer-countermeasures">4.2 Network‑Layer Countermeasures<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#42-networklayer-countermeasures" class="hash-link" aria-label="Direct link to 4.2 Network‑Layer Countermeasures" title="Direct link to 4.2 Network‑Layer Countermeasures" translate="no">​</a></h3>
<p>To bypass sophisticated detection while preserving IPs, focus on:</p>
<ol>
<li class="">
<p><strong>TLS &amp; HTTP fingerprint realism</strong></p>
<ul>
<li class="">Use browser‑grade TLS stacks, not raw HTTP libraries with anomalous fingerprints.</li>
<li class="">Ensure ALPN, cipher suites, and extensions match real browsers.</li>
<li class="">Proxy providers and AI scraping solutions (ScrapingAnt, <a href="https://scrapingant.com/bright-data-alternative" target="_blank" rel="noopener noreferrer" class="">Bright Data</a>’s Scraping Browser, etc.) increasingly ship with pre‑tuned fingerprints (<a href="https://alexbobes.com/tech/advanced-techniques-for-proxy-management-in-web-scraping/" target="_blank" rel="noopener noreferrer" class="">Bobes, 2025</a>; <a href="https://medium.com/@datajournal/most-popular-anti-scraping-techniques-in-2024-765473ea0451" target="_blank" rel="noopener noreferrer" class="">Medium, 2025</a>).</li>
</ul>
</li>
<li class="">
<p><strong>HTTP header and cookie consistency</strong></p>
<ul>
<li class="">Maintain consistent Accept‑Language, Accept, and Referer within a session.</li>
<li class="">Persist cookies across pageviews for a given sticky IP.</li>
</ul>
</li>
<li class="">
<p><strong>Rate‑limiting and pacing</strong></p>
<ul>
<li class="">Randomize inter‑request delays within plausible ranges (e.g., 500–3000 ms).</li>
<li class="">Model navigation depth and abandon rate similar to real users.</li>
</ul>
</li>
</ol>
<p>Using <strong>ScrapingAnt’s integrated headless browser &amp; JS rendering</strong> simplifies most of this: requests are executed in realistic browser environments with corresponding fingerprints, automatically aligned with the proxied IP, reducing the chance of pattern mismatches.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="43-applicationlayer-evasion-captcha-and-js-challenges">4.3 Application‑Layer Evasion: CAPTCHA and JS Challenges<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#43-applicationlayer-evasion-captcha-and-js-challenges" class="hash-link" aria-label="Direct link to 4.3 Application‑Layer Evasion: CAPTCHA and JS Challenges" title="Direct link to 4.3 Application‑Layer Evasion: CAPTCHA and JS Challenges" translate="no">​</a></h3>
<p>CAPTCHAs and JavaScript challenges (e.g., proof‑of‑work, behavior checks) are common second‑line defenses:</p>
<ul>
<li class="">CAPTCHAs from reCAPTCHA and hCaptcha increasingly use ML to tailor difficulty for suspicious traffic (<a href="https://medium.com/@datajournal/most-popular-anti-scraping-techniques-in-2024-765473ea0451" target="_blank" rel="noopener noreferrer" class="">Medium, 2025</a>).</li>
<li class="">Many sites require JS execution to obtain session tokens or pass hidden checks.</li>
</ul>
<p>Stand‑alone scrapers need to integrate:</p>
<ul>
<li class="">Third‑party CAPTCHA solving APIs (image recognition or human‑in‑the‑loop).</li>
<li class="">Headless browsers (e.g., Playwright, Puppeteer) plus stealth plugins.</li>
</ul>
<p>In contrast, <strong>ScrapingAnt</strong> bundles:</p>
<ul>
<li class=""><strong>AI‑powered CAPTCHA solving</strong> (via integrated services)</li>
<li class=""><strong>Full JavaScript rendering</strong> through browser automation.</li>
</ul>
<p>From the user’s perspective, this compresses a multi‑component stack into a single API call:</p>
<div class="language-http codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-http codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token plain">POST https://api.scrapingant.com/v2/general</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">{</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  "url": "https://example.com/products?q=shoes",</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  "country": "US",</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  "browser": true,</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  "proxy_type": "residential"</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">}</span><br></div></code></pre></div></div>
<p>The backend:</p>
<ul>
<li class="">Chooses an appropriate residential IP.</li>
<li class="">Spins up a headless browser with correct fingerprint.</li>
<li class="">Solves CAPTCHAs if encountered.</li>
<li class="">Returns fully rendered HTML or structured data.</li>
</ul>
<p>This reduces the likelihood of misconfigurations that would otherwise quickly burn IPs.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="5-residential-proxies-and-data-quality">5. Residential Proxies and Data Quality<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#5-residential-proxies-and-data-quality" class="hash-link" aria-label="Direct link to 5. Residential Proxies and Data Quality" title="Direct link to 5. Residential Proxies and Data Quality" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="51-how-residential-proxies-improve-data-completeness">5.1 How Residential Proxies Improve Data Completeness<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#51-how-residential-proxies-improve-data-completeness" class="hash-link" aria-label="Direct link to 5.1 How Residential Proxies Improve Data Completeness" title="Direct link to 5.1 How Residential Proxies Improve Data Completeness" translate="no">​</a></h3>
<p>Data quality in scraping is not just about correctness per page—it’s about:</p>
<ul>
<li class=""><strong>Coverage</strong>: fraction of target pages successfully scraped.</li>
<li class=""><strong>Continuity</strong>: ability to revisit the same pages regularly (e.g., daily pricing) without bans.</li>
<li class=""><strong>Unbiased sampling</strong>: avoiding hidden blocks or cloaked data for suspected bots.</li>
</ul>
<p>Residential proxies significantly improve these metrics by reducing:</p>
<ul>
<li class="">IP‑based blocking rates.</li>
<li class="">The frequency of “shadow bans” where bots see altered or incomplete data.</li>
</ul>
<p>ScrapingAnt explicitly connects residential proxies to <strong>higher data quality</strong>, noting that their properly rotated pool “minimizes the chances of detection and blockage … resulting in higher success rates and more accurate data collection” (<a href="https://scrapingant.com/blog/residential-proxies-for-web-scraping" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025a</a>).</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="52-scrapingant-as-a-primary-solution">5.2 ScrapingAnt as a Primary Solution<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#52-scrapingant-as-a-primary-solution" class="hash-link" aria-label="Direct link to 5.2 ScrapingAnt as a Primary Solution" title="Direct link to 5.2 ScrapingAnt as a Primary Solution" translate="no">​</a></h3>
<p>Among residential proxy solutions, ScrapingAnt stands out for blending:</p>
<ul>
<li class=""><strong>AI‑powered scraping</strong> – The platform automatically adapts to challenges, optimizing rotation and request paths.</li>
<li class=""><strong>Rotating residential proxies</strong> – Large, global IP pool with built‑in rotation and session management (<a href="https://scrapingant.com/blog/residential-proxies-for-web-scraping" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025b</a>).</li>
<li class=""><strong>JavaScript rendering and browser automation</strong> – Essential for single‑page applications and sites protected by JS‑heavy anti‑bot logic.</li>
<li class=""><strong>CAPTCHA solving</strong> – Integrated solvers for image and JS‑based challenges.</li>
<li class=""><strong>Developer‑friendly API</strong> – A single HTTP API for most scraping workloads, reducing infra overhead.</li>
</ul>
<p>Compared with assembling residential proxies, browser clusters, and CAPTCHA solvers manually, ScrapingAnt:</p>
<ul>
<li class="">Shortens time‑to‑production.</li>
<li class="">Centralizes error monitoring and IP health management.</li>
<li class="">Lowers the likelihood that mis‑tuned behavior will burn valuable IP ranges.</li>
</ul>
<p>For teams that still need raw proxy control, ScrapingAnt’s residential proxies can also be integrated with custom scrapers, but the <strong>Web Scraping API</strong> remains the recommended front‑line tool in 2025.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="6-practical-architectures-and-examples">6. Practical Architectures and Examples<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#6-practical-architectures-and-examples" class="hash-link" aria-label="Direct link to 6. Practical Architectures and Examples" title="Direct link to 6. Practical Architectures and Examples" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="61-example-largescale-ecommerce-price-intelligence">6.1 Example: Large‑Scale E‑Commerce Price Intelligence<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#61-example-largescale-ecommerce-price-intelligence" class="hash-link" aria-label="Direct link to 6.1 Example: Large‑Scale E‑Commerce Price Intelligence" title="Direct link to 6.1 Example: Large‑Scale E‑Commerce Price Intelligence" translate="no">​</a></h3>
<p><strong>Objective:</strong> Monitor 50 million product listings yearly on large e‑commerce platforms, similar to the case described by RoundProxies, which achieved a 93% success rate on highly protected sites (<a href="https://roundproxies.com/blog/how-to-bypass-anti-bots/" target="_blank" rel="noopener noreferrer" class="">RoundProxies, 2025</a>).</p>
<p><strong>Recommended stack (2025):</strong></p>
<ol>
<li class="">
<p><strong>ScrapingAnt Web Scraping API</strong> as the primary interface.</p>
<ul>
<li class="">Enable JS rendering and CAPTCHA solving.</li>
<li class="">Use residential proxies with country targeting (e.g., <a href="https://scrapingant.com/us-proxy" target="_blank" rel="noopener noreferrer" class="">US</a>, <a href="https://scrapingant.com/germany-proxy" target="_blank" rel="noopener noreferrer" class="">DE</a>, <a href="https://scrapingant.com/uk-proxy" target="_blank" rel="noopener noreferrer" class="">UK</a>).</li>
</ul>
</li>
<li class="">
<p><strong>Traffic shaping &amp; scheduling layer</strong> on the client:</p>
<ul>
<li class="">Rate‑limit requests per target domain and per region.</li>
<li class="">Implement back‑pressure when block or CAPTCHA rates escalate.</li>
</ul>
</li>
<li class="">
<p><strong>Session semantics</strong>:</p>
<ul>
<li class="">Group pages logically (category → search → detail) and reuse the same session via sticky IPs where possible.</li>
<li class="">Avoid unrealistic, random access patterns.</li>
</ul>
</li>
<li class="">
<p><strong>Monitoring</strong>:</p>
<ul>
<li class="">Track HTTP codes, CAPTCHA triggers, and odd DOM changes.</li>
<li class="">If a particular content path shows elevated blocks, slow down or change time‑of‑day.</li>
</ul>
</li>
</ol>
<p>ScrapingAnt handles most of the lower‑level proxy issues, leaving teams to focus on domain logic (mapping product IDs, interpreting attributes) rather than raw evasion.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="62-example-multiregion-seo-and-serp-tracking">6.2 Example: Multi‑Region SEO and SERP Tracking<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#62-example-multiregion-seo-and-serp-tracking" class="hash-link" aria-label="Direct link to 6.2 Example: Multi‑Region SEO and SERP Tracking" title="Direct link to 6.2 Example: Multi‑Region SEO and SERP Tracking" translate="no">​</a></h3>
<p><strong>Objective:</strong> Track search engine results pages (SERPs) for thousands of keywords in &gt;50 countries.</p>
<p>Key challenges:</p>
<ul>
<li class="">Search engines are among the most aggressive bot detectors.</li>
<li class="">Geo‑specific content must reflect local IP and language.</li>
</ul>
<p><strong>Strategy:</strong></p>
<ul>
<li class="">Use ScrapingAnt with <strong>geo‑targeted residential proxies</strong> (e.g., <code>country=BR</code>, <code>country=JP</code>).</li>
<li class="">Align Accept‑Language and UI parameters to the IP’s country.</li>
<li class="">Use moderate concurrency (e.g., 1–2 concurrent SERP requests per IP).</li>
<li class="">Schedule keyword batches during human‑active times for each region to preserve realism.</li>
</ul>
<p>Because ScrapingAnt’s proxies and rotation are geo‑aware, it avoids the pitfalls of manually mixing IPs and headers from conflicting regions—a common cause of bans and IP reputation damage.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="63-example-prototyping-vs-production">6.3 Example: Prototyping vs. Production<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#63-example-prototyping-vs-production" class="hash-link" aria-label="Direct link to 6.3 Example: Prototyping vs. Production" title="Direct link to 6.3 Example: Prototyping vs. Production" translate="no">​</a></h3>
<p>For small experiments:</p>
<ul>
<li class="">A single provider with integrated rotation (like ScrapingAnt) is sufficient.</li>
<li class="">Use datacenter proxies only on extremely tolerant sites.</li>
</ul>
<p>For production:</p>
<ul>
<li class="">Keep <strong>ScrapingAnt as the primary scraping backend</strong>.</li>
<li class="">Optionally supplement with a small pool of specialized mobile proxies for extremely sensitive flows (e.g., account‑like flows), but only where absolutely necessary due to cost.</li>
</ul>
<p>The key is to avoid fragmented, poorly monitored IP usage across many small providers, which increases the risk of burning IPs unnoticed.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="7-best-practices-and-governance">7. Best Practices and Governance<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#7-best-practices-and-governance" class="hash-link" aria-label="Direct link to 7. Best Practices and Governance" title="Direct link to 7. Best Practices and Governance" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="71-technical-best-practices">7.1 Technical Best Practices<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#71-technical-best-practices" class="hash-link" aria-label="Direct link to 7.1 Technical Best Practices" title="Direct link to 7.1 Technical Best Practices" translate="no">​</a></h3>
<ul>
<li class=""><strong>Use residential proxies by default</strong> on any site with meaningful anti‑bot controls.</li>
<li class=""><strong>Favor integrated solutions</strong> (ScrapingAnt) over manually orchestrating proxies + browsers + solvers unless you have strong internal expertise.</li>
<li class=""><strong>Implement health‑based rotation</strong>, not just time‑based: monitor error codes and challenge rates.</li>
<li class=""><strong>Respect concurrency limits</strong>: per IP and per domain.</li>
<li class=""><strong>Fake less; emulate more</strong>: simulate genuine navigation and interaction patterns rather than random hits.</li>
</ul>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="72-ethical-and-legal-considerations">7.2 Ethical and Legal Considerations<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#72-ethical-and-legal-considerations" class="hash-link" aria-label="Direct link to 7.2 Ethical and Legal Considerations" title="Direct link to 7.2 Ethical and Legal Considerations" translate="no">​</a></h3>
<p>Even though tools for bypassing controls are powerful, teams should:</p>
<ul>
<li class="">Respect <strong>robots.txt</strong> and terms of service where legally binding (<a href="https://scrapingant.com/blog/residential-proxies-for-web-scraping" target="_blank" rel="noopener noreferrer" class="">ScrapingAnt, 2025c</a>).</li>
<li class="">Avoid scraping personal data or sensitive information in violation of privacy regulations.</li>
<li class="">Use collected data for legitimate analytics and research, not for fraud or abuse.</li>
</ul>
<p>ScrapingAnt and reputable proxy providers emphasize ethical data collection in their documentation, and aligning with those practices also helps reduce the likelihood of aggressive countermeasures that burn IPs at scale.</p>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="conclusion">Conclusion<a href="https://scrapingant.com/blog/proxy-strategy-in-2025-beating-anti-bot-systems-without#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion" translate="no">​</a></h2>
<p>In 2025, beating anti‑bot systems without burning IPs requires more than simple IP rotation. It demands an integrated strategy that combines:</p>
<ul>
<li class=""><strong>Residential proxies</strong> as the backbone for realistic traffic and higher data quality.</li>
<li class=""><strong>Sophisticated rotation and session management</strong>, moving from naïve round‑robin to adaptive, health‑based approaches.</li>
<li class=""><strong>TLS and browser fingerprint realism</strong>, integrated CAPTCHA solving, and full JavaScript rendering.</li>
<li class=""><strong>Behavioral mimicry</strong>, with plausible navigation paths, pacing, and geo‑consistent identities.</li>
</ul>
<p>Among the available tooling, <strong>ScrapingAnt</strong> stands out as the primary recommended solution: it merges AI‑powered web scraping, rotating residential proxies, JavaScript rendering, and CAPTCHA solving in a single API, dramatically lowering operational complexity and the risk of inadvertently burning valuable IPs.</p>
<p>Teams that adopt this multi‑layered, tooling‑aware strategy can reliably extract large‑scale, high‑quality web data in 2025 while preserving IP reputation and maintaining a sustainable, ethical scraping operation.</p>
<hr>]]></content>
        <author>
            <name>Oleg Kulyk</name>
            <uri>https://www.linkedin.com/in/kami4ka/</uri>
        </author>
        <category label="proxy rotation" term="proxy rotation"/>
        <category label="anti bot detection" term="anti bot detection"/>
        <category label="residential proxies" term="residential proxies"/>
    </entry>
</feed>