# gau - Get All URLs (`rl1987/gau-wrapper`) Actor

Fetch known URLs from the Wayback Machine, Common Crawl, AlienVault OTX, and URLScan for any domain. A wrapper around the gau OSINT tool for attack-surface and data-pipeline use.

- **URL**: https://apify.com/rl1987/gau-wrapper.md
- **Developed by:** [R.L.](https://apify.com/rl1987) (community)
- **Categories:** Other
- **Features:** Limited permissions
- **Stats:** 5 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does gau - Get All URLs do?

**gau - Get All URLs** is an Apify Actor that wraps the popular open-source OSINT tool [`gau`](https://github.com/lc/gau) (Get All URLs). For any domain you give it, it collects **every URL ever seen** by major web archives and threat-intelligence feeds — the [Wayback Machine](https://web.archive.org), [Common Crawl](https://commoncrawl.org), [AlienVault OTX](https://otx.alienvault.com), and [URLScan](https://urlscan.io) — and returns them as a clean, structured dataset.

Running it on the Apify platform gives you API access, scheduling, monitoring, proxy rotation, and easy integration with the rest of your data pipeline — no need to install Go or manage the binary yourself.

### Why use gau - Get All URLs?

- **Attack-surface mapping** — enumerate historical and current endpoints, parameters, and forgotten paths for a target domain during authorized security assessments.
- **Content & SEO audits** — discover every URL that archives know about, including pages no longer linked from the live site.
- **Pipeline integration** — feed the resulting URLs into other Apify Actors (HTTP scrapers, vulnerability scanners, link checkers) straight from the dataset.
- **No local setup** — schedule recurring runs, call it from the API, and connect it to Make, Zapier, n8n, Google Drive, and more.

### How to use gau - Get All URLs

1. Open the Actor and go to the **Input** tab.
2. Enter one or more **domains** (bare hostnames such as `example.com` — no `https://` and no path).
3. Optionally pick the **providers**, enable **subdomains**, set date ranges, or filter by extension / status code / MIME type.
4. Click **Start** and watch URLs stream into the **Output** tab.
5. Download the dataset in JSON, CSV, or Excel — or fetch a plain newline-delimited URL list directly from the dataset API with `?fields=url&format=csv&clean=true`.

### Input

Configure the run from the **Input** tab or via JSON. Key fields:

| Field | Type | Description |
|-------|------|-------------|
| `domains` | array | **Required.** Bare domains to query, e.g. `["example.com"]`. |
| `providers` | array | Sources to use: `wayback`, `commoncrawl`, `otx`, `urlscan`. |
| `includeSubdomains` | boolean | Include subdomains of the target (`--subs`). |
| `fromDate` / `toDate` | string | Limit by first-seen month, format `YYYYMM`. |
| `blacklistExtensions` | array | Extensions to skip, e.g. `["png","jpg","woff"]`. |
| `matchStatusCodes` / `filterStatusCodes` | array | Keep / drop by archived HTTP status. |
| `matchMimeTypes` / `filterMimeTypes` | array | Keep / drop by archived MIME type. |
| `removeDuplicateParams` | boolean | Collapse endpoints that differ only in parameter values (`--fp`). |
| `threads`, `timeout`, `retries` | integer | HTTP client tuning. |
| `maxResults` | integer | Stop after N URLs (`0` = unlimited). |
| `proxyConfiguration` | object | Route requests through an Apify or custom proxy. |

Example input:

```json
{
    "domains": ["example.com"],
    "providers": ["wayback", "commoncrawl", "otx"],
    "includeSubdomains": true,
    "blacklistExtensions": ["png", "jpg", "css"],
    "maxResults": 5000
}
```

### Output

Each discovered URL becomes one dataset item, **pushed to the dataset the instant gau yields it** — so rows stream into the Output tab live while the run is still in progress, ready for downstream consumers to pick up immediately. You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. Need just a plain newline-delimited URL list (e.g. to pipe into `httpx` or `nuclei`)? Hit the dataset API with `?fields=url&format=csv&clean=true`.

```json
{
    "url": "https://www.example.com/login?next=/account",
    "domain": "example.com",
    "host": "www.example.com",
    "scheme": "https",
    "path": "/login",
    "query": "next=/account",
    "fileExtension": null,
    "provider": "wayback"
}
```

#### Data table

| Field | Description |
|-------|-------------|
| `url` | The full discovered URL. |
| `domain` | Which input domain the URL belongs to. |
| `host` | Hostname of the URL (may be a subdomain). |
| `scheme` | `http` or `https`. |
| `path` | URL path component. |
| `query` | Raw query string, if any. |
| `fileExtension` | Lower-cased file extension of the path, if any. |
| `provider` | Source provider when a single provider is selected, otherwise `null`. |

### How much does it cost?

The Actor is lightweight — it streams text from public archives and does no browser rendering, and it never buffers the full result set in memory, so it runs comfortably at the modest **512 MB default** memory regardless of how many URLs a domain has. Most runs finish in seconds to a few minutes. Cost scales with how many URLs a domain has archived and how many providers you query. Large, popular domains can return hundreds of thousands of URLs; use `maxResults`, `blacklistExtensions`, and date ranges to keep runs bounded.

### Tips and advanced options

- **Speed vs. completeness** — querying all four providers is the most thorough but slowest; pick a subset for quick runs.
- **Rate limits** — if a provider throttles you, enable `proxyConfiguration` and/or raise `retries` and `timeout`.
- **Noise reduction** — blacklist static asset extensions (`png,jpg,css,woff,svg`) and use `removeDuplicateParams` to shrink the result set.
- **Subdomains** — `includeSubdomains` greatly increases coverage (and volume) for an organization.

### Cyber recon toolkit

Part of the **Cyber recon toolkit** — OSINT, attack-surface recon, and vulnerability-intel data collection for security research:

- [Bugcrowd Scraper](https://apify.com/rl1987/bugcrowd-scraper) — Scrapes public Bugcrowd data: disclosed submissions, program directory, leaderboard.
- [HackerOne Scraper](https://apify.com/rl1987/hackerone-scraper) — Scrapes public HackerOne data: Hacktivity disclosed reports, program directory, researcher leaderboard.
- [Intigriti Scraper](https://apify.com/rl1987/intigriti-scraper) — Scrapes public Intigriti data: bug bounty/VDP program directory, researcher leaderboard and profiles.
- [CVE Scraper](https://apify.com/rl1987/cve-scraper) — CVE.org vulnerability info scraper.
- [TLSX - TLS Data Gathering and Analysis](https://apify.com/rl1987/pd-tlsx) — Fast, configurable TLS grabber for TLS-based data collection and analysis.
- [BGP.HE.NET Recon Scraper](https://apify.com/rl1987/bgphenet-scraper) — Scrapes ASN, prefix, IP, and DNS recon data from bgp.he.net.
- [Git Email Extractor](https://apify.com/rl1987/git-email-extractor) — Shallow-clones git repos and extracts contributor emails from commit history.
- [Wappalyzer Technology Lookup](https://apify.com/rl1987/wappalyzer-tech-lookup) — Detect the technology stack behind any website using Wappalyzer.
- [Meta Tags Scraper](https://apify.com/rl1987/meta-tags-scraper) — Web page metadata scraper (meta tags, Open Graph, Twitter Card).
- [Universal Contact Extractor](https://apify.com/rl1987/universal-contact-extractor) — Extract phone numbers, emails and social media URLs from web pages.
- [Web Text Extractor](https://apify.com/rl1987/web-text-extractor) — Extract clean text or Markdown content from web pages.

### Did you find this useful?

Rate this actor on Apify! Your feedback helps other users find it and helps us keep improving it.

### FAQ, disclaimers, and support

**Is this legal?** The Actor only reads from public archives and threat-intel feeds; it does not touch the target's own servers. Use it only against domains you own or are explicitly authorized to test, and comply with the providers' and Apify's Terms of Service.

**Why are some URLs dead or old?** Results come from historical archives, so they include URLs that may no longer exist. That is by design for OSINT and attack-surface work.

**Found a bug or need a custom version?** Open an issue from the Actor's **Issues** tab — feedback and custom-solution requests are welcome.

This Actor wraps the open-source [`gau`](https://github.com/lc/gau) tool by [@lc](https://github.com/lc), distributed under the MIT license.

# Actor input Schema

## `domains` (type: `array`):

List of domains (or apex domains) to fetch known URLs for, e.g. `example.com`. Do not include a scheme (`https://`) or path.

## `providers` (type: `array`):

Which URL sources to query. Leave empty to use gau's defaults (wayback, commoncrawl, otx).

## `includeSubdomains` (type: `boolean`):

Include URLs from subdomains of the target domain (gau `--subs`).

## `fromDate` (type: `string`):

Only fetch URLs first seen on or after this month. Format: YYYYMM (e.g. `202101`).

## `toDate` (type: `string`):

Only fetch URLs first seen on or before this month. Format: YYYYMM (e.g. `202412`).

## `blacklistExtensions` (type: `array`):

File extensions to skip, without the dot (gau `--blacklist`), e.g. `png`, `jpg`, `woff`.

## `matchStatusCodes` (type: `array`):

Only keep URLs whose archived HTTP status code is in this list (gau `--mc`), e.g. `200`, `301`.

## `filterStatusCodes` (type: `array`):

Drop URLs whose archived HTTP status code is in this list (gau `--fc`), e.g. `404`, `302`.

## `matchMimeTypes` (type: `array`):

Only keep URLs whose archived MIME type is in this list (gau `--mt`), e.g. `text/html`.

## `filterMimeTypes` (type: `array`):

Drop URLs whose archived MIME type is in this list (gau `--ft`), e.g. `text/plain`, `image/png`.

## `removeDuplicateParams` (type: `boolean`):

Keep only one representative URL per endpoint that differs only in parameter values (gau `--fp`).

## `threads` (type: `integer`):

Number of worker threads gau spawns.

## `timeout` (type: `integer`):

Timeout for gau's HTTP client, in seconds.

## `retries` (type: `integer`):

Number of retries for gau's HTTP client.

## `maxResults` (type: `integer`):

Stop after pushing this many URLs to the dataset. Set 0 for unlimited.

## `proxyConfiguration` (type: `object`):

Route gau's requests through an Apify or custom HTTP proxy. Useful when archive providers rate-limit your runs.

## Actor input object example

```json
{
  "domains": [
    "trickster.dev"
  ],
  "providers": [
    "wayback",
    "commoncrawl",
    "otx"
  ],
  "includeSubdomains": true,
  "blacklistExtensions": [],
  "matchStatusCodes": [],
  "filterStatusCodes": [],
  "matchMimeTypes": [],
  "filterMimeTypes": [],
  "removeDuplicateParams": false,
  "threads": 5,
  "timeout": 45,
  "retries": 3,
  "maxResults": 0,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `urlsCsv` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "trickster.dev"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("rl1987/gau-wrapper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": ["trickster.dev"] }

# Run the Actor and wait for it to finish
run = client.actor("rl1987/gau-wrapper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "trickster.dev"
  ]
}' |
apify call rl1987/gau-wrapper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rl1987/gau-wrapper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/c3NSSHxL3F7YY6TPN/builds/zLgHP1awavftoDugS/openapi.json