# URL Wrangler (`rl1987/url-wrangler`) Actor Join relative URLs, decompose URLs into a node/edge tree, replace/remove query params, and extract fields — batch URL wrangling toolkit. - **URL**: https://apify.com/rl1987/url-wrangler.md - **Developed by:** [R.L.](https://apify.com/rl1987) (community) - **Categories:** Developer tools - **Features:** Limited permissions - **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README **URL Wrangler** is a batch URL-processing toolkit for the [Apify platform](https://apify.com). Feed it a list of URLs and pick one of four operations — resolve relative URLs, decompose a URL into a labeled node/edge tree, upsert or strip query parameters, or pull out a single field — and get structured JSON back, one row per input URL. Run it via the [Apify Console](https://console.apify.com), the [API](https://docs.apify.com/api/v2), or on a [schedule](https://docs.apify.com/platform/schedules), with full run history, monitoring, and integrations (Zapier, Make, Google Sheets, webhooks) included. ### Why use URL Wrangler? Working with URLs at scale — cleaning up scraped links, rewriting tracking parameters before storing them, or reverse-engineering an unfamiliar link structure during OSINT/DFIR work — usually means writing one-off scripts around `urllib.parse`. URL Wrangler packages the common operations into a single Actor so you can: - **Deduplicate and clean up crawl output** by resolving every relative link a scraper found against its page URL (`urljoin`). - **Investigate a suspicious or unfamiliar URL** by breaking it into its components and auto-decoding embedded Base64 blobs, hex strings, UUIDs, and Unix timestamps (`unfurl`) — the same idea as the DFIR tool [`unfurl`](https://github.com/obsidianforensics/unfurl), reimplemented here dependency-free. - **Strip tracking parameters or rewrite campaign tags in bulk** before storing or forwarding URLs (`replace_params`). - **Pull out just the domain, path, or a custom-formatted string** from a batch of URLs for reporting or filtering (`extract`). ### How to use URL Wrangler 1. Open the Actor's **Input** tab. 2. Choose an **Operation**: `urljoin`, `unfurl`, `replace_params`, or `extract`. 3. Paste your list of URLs into **URLs**. 4. Fill in the fields relevant to your chosen operation (see [Input](#input) below, and [Examples](#examples) for a full input/output pair per operation — irrelevant fields are ignored). 5. Click **Start**. Results land in the run's default **Dataset**, one row per input URL, downloadable as JSON, CSV, Excel, or HTML. ### Input All fields live in one flat input object — an `operation` selector, the shared `urls` list, and a handful of operation-specific parameters that apply uniformly to every URL in that run. See the **Input** tab for the full schema with descriptions. | Field | Used by | Description | |---|---|---| | `operation` | all | `urljoin` | `unfurl` | `replace_params` | `extract` | | `urls` | all | The URLs to process (relative URLs, for `urljoin`) | | `baseUrl` | `urljoin` | Reference URL every entry in `urls` is resolved against | | `params` | `replace_params` | Key/value pairs to add or overwrite, via a key-value editor (no JSON typing) | | `remove` | `replace_params` | Keys to strip; supports `fnmatch` wildcards like `utm_*` | | `field` | `extract` | `domain` | `apex` | `subdomain` | `tld` | `path` | `keys` | `values` | `keypairs` | `format` | | `format` | `extract` (`field=format`) | Directive string, e.g. `%s://%d%p?%q` | | `unfurlDetail` | `unfurl` | `summary` (default) or `full` (adds a node/edge count summary) | | `outputFormat` | `unfurl` | Include a human-readable ASCII tree alongside the JSON graph | ### Output Every run pushes one dataset item per input URL: `{ operation, input, result, error }`. `error` is `null` on success, so a single malformed URL never aborts the run. ### Examples Each pair below is a complete Actor input alongside the corresponding dataset row it produces. #### `urljoin` Input: ```json { "operation": "urljoin", "urls": ["../c", "/d", "https://other.com/x"], "baseUrl": "https://example.com/a/b/" } ``` Output (first row): ```json { "operation": "urljoin", "input": "../c", "result": "https://example.com/a/c", "error": null } ``` #### `unfurl` Input: ```json { "operation": "unfurl", "urls": ["https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8="], "outputFormat": true } ``` Output: ```json { "operation": "unfurl", "input": "https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8=", "result": { "nodes": [ { "id": 1, "type": "url", "key": null, "value": "https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8=", "label": "https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8=" }, { "id": 2, "type": "url.scheme", "key": null, "value": "https", "label": "https" }, { "id": 3, "type": "url.hostname", "key": null, "value": "sub.example.com", "label": "sub.example.com" }, { "id": 6, "type": "url.query.param", "key": "ts", "value": "1700000000", "label": "ts: 1700000000" }, { "id": 7, "type": "decoded.timestamp", "key": null, "value": "2023-11-14T22:13:20+00:00", "label": "2023-11-14T22:13:20+00:00" }, { "id": 8, "type": "url.query.param", "key": "data", "value": "aGVsbG8=", "label": "data: aGVsbG8=" }, { "id": 9, "type": "decoded.base64", "key": null, "value": "hello", "label": "hello" } ], "edges": [ { "from": 1, "to": 2, "label": "url-parse" }, { "from": 1, "to": 3, "label": "url-parse" }, { "from": 1, "to": 6, "label": "query-split" }, { "from": 6, "to": 7, "label": "epoch-decode" }, { "from": 1, "to": 8, "label": "query-split" }, { "from": 8, "to": 9, "label": "base64-decode" } ], "text": "[1] https://sub.example.com/users/1?ts=1700000000&data=aGVsbG8=\n├─(url-parse)─[2] https\n├─(url-parse)─[3] sub.example.com\n├─(query-split)─[6] ts: 1700000000\n│ └─(epoch-decode)─[7] 2023-11-14T22:13:20+00:00\n└─(query-split)─[8] data: aGVsbG8=\n └─(base64-decode)─[9] hello" }, "error": null } ``` *(path-segment nodes trimmed above for brevity — the full graph includes every path segment as its own node.)* #### `replace_params` Input: ```json { "operation": "replace_params", "urls": ["https://example.com/?a=1&utm_source=x&gclid=z"], "params": [{ "key": "a", "value": "9" }], "remove": ["utm_*", "gclid"] } ``` Output: ```json { "operation": "replace_params", "input": "https://example.com/?a=1&utm_source=x&gclid=z", "result": "https://example.com/?a=9", "error": null } ``` #### `extract` Input: ```json { "operation": "extract", "urls": ["https://sub.example.com/users/1?id=1"], "field": "format", "format": "%s://%d%p?%q" } ``` Output: ```json { "operation": "extract", "input": "https://sub.example.com/users/1?id=1", "result": "https://sub.example.com/users/1?id=1", "error": null } ``` ### Data table | Field | Type | Present when | Description | |---|---|---|---| | `operation` | string | always | Operation that produced this row | | `input` | string | always | The original input URL | | `result` | varies | on success | Absolute URL string (`urljoin`), node/edge object (`unfurl`), rewritten URL string (`replace_params`), or field value/array/string (`extract`) | | `error` | string | null | always | Failure reason for this URL, or `null` | ### Pricing / cost estimation The current price of every event is on the **Pricing** tab of this Actor's page in Apify Store. That tab is generated from the Actor's own pricing record, so it is always current, and your plan's rung is applied automatically. A figure repeated here would be a second copy that drifts out of date. You can cap what any single run may spend with **Max total charge**. ### Tips / advanced options - For `unfurl`, leave `outputFormat` off unless you want the ASCII tree — it roughly doubles the payload size per row for large graphs. - Base64/hex/timestamp/UUID decoding in `unfurl` recurses up to 5 levels deep (e.g. a query param that's itself Base64-encoded JSON containing another encoded value), capped to bound output size on adversarial input. - `replace_params`'s `remove` field supports `fnmatch`-style wildcards (`utm_*`, `*_id`), so you don't need to enumerate every tracking parameter by name. - `apex`/`subdomain`/`tld` in `extract` resolve against the [Public Suffix List](https://publicsuffix.org/), so `www.bbc.co.uk` gives apex `bbc.co.uk` and tld `co.uk`. The list is used from a bundled snapshot — no network call at run time — which means a suffix registered after the snapshot was taken is not known yet. The private section is included, so a user-controlled host like `user.github.io` is its own apex rather than a subdomain of `github.io`. (`unfurl` does not decompose the hostname at all; an earlier note here said it did.) ### Background reading The `unfurl` operation's node/edge tree is modeled on the DFIR community tool of the same name, and `extract`'s field/format model is modeled on a separate, unrelated CLI tool. Related reading: - [obsidianforensics/unfurl](https://github.com/obsidianforensics/unfurl) — the DFIR tool this Actor's `unfurl` operation is modeled on - [SANS ISC: Unfurl v2025.02 released](https://isc.sans.edu/diary/31716) — write-up on `unfurl`'s timestamp/IP/UUID decoding capabilities - [tomnomnom/unfurl](https://github.com/tomnomnom/unfurl) — the flat field-extraction CLI this Actor's `extract` operation is modeled on - [Google Search Central: URL canonicalization](https://developers.google.com/search/docs/crawling-indexing/canonicalization) — background on why URL normalization matters for deduplication - [Pinterest Engineering: Smarter URL Normalization at Scale (MIQPS)](https://medium.com/pinterest-engineering/smarter-url-normalization-at-scale-how-miqps-powers-content-deduplication-at-pinterest-4aa42e807d7d) — a production system for deciding which query parameters are semantically significant ### FAQ, limitations, and support - This Actor only processes URLs you provide — it does not fetch, crawl, or follow redirects for any of them. - `unfurl`'s decoders are heuristic (they detect *plausible* Base64/hex/epoch/UUID values); false negatives on obfuscated data and, rarely, false positives on coincidentally-decodable strings are possible. - Found a bug or want another operation (URL normalization/dedup, punycode/IDNA conversion, tracking-parameter presets)? Open an issue on the Actor's Issues tab — these are tracked as candidate follow-ups. ### Data pipeline toolkit Part of the **Data pipeline toolkit** — small, chainable Actors for cleaning, transforming, and generating data inside a larger pipeline: - [jq Helper – transform JSON with jq](https://apify.com/rl1987/jq-helper) — Run jq programs over inline JSON or a linked Apify dataset. - [DuckDB Helper – SQL over CSV, JSON, Parquet, Excel, SQLite](https://apify.com/rl1987/duckdb-wrapper) — Run a DuckDB SQL query over remote/local files, push results to a dataset. - [Regex Helper](https://apify.com/rl1987/regex-helper) — Apply named regular expressions to strings, extract structured matches. - [ZIP Code Helper](https://apify.com/rl1987/zip-code-helper) — Resolves US ZIP codes into city, state, county, and more. - [Postal Address Normaliser](https://apify.com/rl1987/postal-address-normaliser) — Parses and normalises postal addresses using libpostal. - [Phone Number Wrangler](https://apify.com/rl1987/phone-number-wrangler) — Validate, format, and parse phone numbers using libphonenumber. - [UUID Generator](https://apify.com/rl1987/uuid-generator) — Generate bulk UUIDs (v1, v3, v4, v5, v7) on demand. - [Secure Password & Passphrase Generator](https://apify.com/rl1987/password-generator) — Generate secure passwords and diceware passphrases per NIST guidance. - [Thumbnail Maker](https://apify.com/rl1987/thumbnail-maker) — Generates thumbnails from image URLs using ImageMagick. - [Katana Web Crawler (ProjectDiscovery)](https://apify.com/rl1987/pd-katana) — Crawl websites with Katana, stream results as JSONL. - [ProjectDiscovery Notify](https://apify.com/rl1987/pd-notify) — Stream records to Slack, Discord, Telegram, Email, and more. ### Did you find this useful? Rate this actor on Apify! Your feedback helps other users find it and helps us keep improving it. # Actor input Schema ## `operation` (type: `string`): Which URL-wrangling operation to run over the URL list. ## `urls` (type: `array`): The URLs to process. For 'urljoin', these are the relative URLs resolved against Base URL. For the other operations, these are the URLs operated on directly. ## `baseUrl` (type: `string`): Reference URL every entry in URLs is resolved against. ## `params` (type: `array`): Query-string key/value pairs to add or overwrite on every URL. ## `remove` (type: `array`): Query-string keys to strip from every URL. Supports fnmatch wildcards, e.g. "utm\_\*". ## `field` (type: `string`): Which field to extract from every URL. `apex`, `subdomain` and `tld` resolve against the Public Suffix List, so `www.bbc.co.uk` gives apex `bbc.co.uk`, subdomain `www` and tld `co.uk`; the list's private section is included, so `user.github.io` is its own apex. ## `format` (type: `string`): Printf-style directive string, e.g. "%s://%d%p?%q". Directives: %s scheme, %u userinfo, %d domain, %S subdomain, %r root, %t tld, %P port, %p path, %e extension, %q query, %f fragment, %@ %: %? %# conditional separators, %a authority, %% literal percent. ## `unfurlDetail` (type: `string`): 'summary' returns the node/edge tree; 'full' also includes a node/edge count summary. ## `outputFormat` (type: `boolean`): Also render a human-readable ASCII tree alongside the JSON node/edge graph. ## Actor input object example ```json { "operation": "urljoin", "urls": [ "../c", "/d" ], "baseUrl": "https://example.com/a/b/", "field": "domain", "unfurlDetail": "summary", "outputFormat": false } ``` # Actor output Schema ## `results` (type: `string`): Rows produced by the run. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "../c", "/d" ], "baseUrl": "https://example.com/a/b/" }; // Run the Actor and wait for it to finish const run = await client.actor("rl1987/url-wrangler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "../c", "/d", ], "baseUrl": "https://example.com/a/b/", } # Run the Actor and wait for it to finish run = client.actor("rl1987/url-wrangler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "../c", "/d" ], "baseUrl": "https://example.com/a/b/" }' | apify call rl1987/url-wrangler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,rl1987/url-wrangler" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/jYXgiMDWp7QAbQkmj/builds/YuWbUmheadwzw8Fks/openapi.json