Ants Watch

ants watch is a DHT client monitoring tool. It is able to log the activity of all nodes in a DHT network by carefully placing ants in the DHT keyspace. For nodes to utilize the DHT they need to perform routing table maintenance tasks. These tasks consist of sending requests to several other nodes close to oneself in the DHT keyspace. ants watch ensures that at least one of these requests will always hit one of the deployed ants. When a request hits an ant, we record information about the requesting peer like agent version, supported protocols, IP addresses, and more.

Supported networks (selected via --project/--network):

• Celestia — celestia/mainnet, celestia/arabica-11, celestia/mocha-4
• Avail — avail/mainnet-lc
• AGNTCY — agntcy/mainnet
• Can be extended to support other networks using the libp2p DHT by adding an entry to bootstrappers.go.

• Quickstart
• Methodology
• Configuration
• Usage
  • Queen
  • Health
• Ants Key Generation
• Metrics
• Development
• Deployment
• Related Efforts
• Maintainers
• Contributing
• License

The queen needs:

1. Start ClickHouse in Docker:

   just local-clickhouse

2. Copy the settings, which match that ClickHouse server, and load them:

   cp .env.example .env
   set -a; . ./.env; set +a

3. Start the queen with UPnP:

   go run ./cmd/ants queen --upnp

The queen applies the database migrations on startup.

• An ant is a lightweight libp2p DHT node, participating in the DHT network, and logging incoming requests.
• ants participate in the DHT network as DHT server nodes. ants need to be dialable by other nodes in the network. Hence, ants-watch must run on a public IP address either with port forwarding properly configured (including local and gateway firewalls) or UPnP enabled.
• The tool releases ants (i.e., spawns new ant nodes) at targeted locations in the keyspace in order to occupy and watch the full keyspace.
• The tool's logic is based on the fact that peer routing requests are distributed to k closest nodes in the keyspace and routing table updates by DHT client (and server) nodes need to find the k closest DHT server peers to themselves. Therefore, placing approximately 1 ant node every k DHT server nodes can capture all DHT client nodes over time.
• The routing table update process varies across implementations, but is by default set to 10 mins in the go-libp2p implementation. This means that ants will record the existence of DHT client nodes approximately every 10 mins (or whatever the routing table update interval is).
• Depending on the network size, the number of ants as well as their location in the keyspace is adjusted automatically.
• Network size and peers distribution is obtained by querying an external Nebula database.
• All ants run from within the same process, sharing the same DHT records.
• The ant queen is responsible for spawning, adjusting the number and monitoring the ants as well as gathering their logs and persisting them to a central database.
• ants-watch does not operate like a crawler, where after one run the number of DHT client nodes is captured. ants-watch logs all received DHT requests and therefore, it must run continuously to provide the number of DHT client nodes over time.

Each flag also reads an environment variable. .env.example lists all of them.

The queen applies the embedded migration files from ./db/migrations on startup. It strips the Replicated prefix of the merge tree engines, because a single ClickHouse server, such as the one of just local-clickhouse, cannot run them. On a ClickHouse cluster, such as ClickHouse Cloud, set ANTS_CLICKHOUSE_MIGRATIONS_REPLICATED_TABLE_ENGINES=true.

just local-clickhouse starts the container ants-clickhouse on port 9000 without TLS, with the database ants_local, the user ants_local and the password password.

Once the database is set up and migrations are applied, you can start the honeypot.

ants-watch needs to be dialable by other nodes in the network. Hence, it must run on a public IP address either with port forwarding properly configured (including local and gateway firewalls) or UPnP enabled.

To start the ants queen, you can run the following command:

go run ./cmd/ants queen --project celestia --network mainnet --upnp # for UPnP
# or
go run ./cmd/ants queen --project celestia --network mainnet --first.port=<port> --num.ports=<count> # for port forwarding

Use --project/--network (default celestia/mainnet) to select the network. The queen fails to start if the pair has no configured bootstrap peers (see bootstrappers.go).

When UPnP is disabled, ports from firstPort to firstPort + nPorts - 1 must be forwarded to the machine running ants-watch. ants-watch will be able to spawn at most nPorts distinct ants.

You can run a health check on the honeypot by running the following command:

go run ./cmd/ants health

The command requests /healthz from the metrics server. Its defaults are 127.0.0.1 and port 5999, and the defaults of the metrics server are localhost and port 6060. Set ANTS_METRICS_PORT for both commands, as .env.example does.

The queen ant periodically queries the Nebula database to retrieve the list of connected DHT servers. Kademlia identifiers of these peers are then inserted into a binary trie. Using this binary trie, the queen defines keyspace zones of at most bucket_size - 1 peers. One ant must be present in each of these zones in order to capture all DHT requests reaching the bucket_size closest peers to the target key.

Kademlia identifiers are derived from a libp2p peer id, which itself is derived from a cryptographic key pair. Hence generating a key matching a specific zone of the binary trie isn't trivial and requires bruteforce. All keys generated during the bruteforce are persisted on disk, because they may be useful in the future. When an ant isn't needed anymore, its key is marked as available for reuse. This also allows reusing the same peer ids for the ants across multiple runs of the honeypot.

With ANTS_METRICS_ENABLED=true, the queen serves Prometheus metrics on ANTS_METRICS_HOST:ANTS_METRICS_PORT at /metrics. The names carry the prefix ants_. Units are not set yet, and some names do not follow the ProbeLab naming rules.

The batch_inserter metrics come from the ClickHouse batch inserter of go-commons. The repository has no Grafana dashboard yet.

Run just ci before you open a pull request.

ProbeLab runs ants-watch as 2 AWS ECS services in the cluster prod-use1-cmi-ecs-cluster (us-east-1):

The probelab-infra repository holds the Terraform (tf/ants.tf, tf/modules/ants). Every merge to main builds the image probelab:ants-sha<commit> in ProbeLab's ECR. A pull request with the label push-image pushes its image too.

To deploy:

1. Add the label deploy to the pull request before you merge it. The merge then deploys both services.
2. Or deploy a commit of main by hand: just deploy <commit>, or run the CI workflow from the Actions tab with deploy set.

• hydra-booster - A DHT Indexer node & Peer Router

• @guillaumemichel
• @kasteph
• @dennis-tra.

Feel free to dive in! Open an issue or submit PRs.

Standard Readme follows the Contributor Covenant Code of Conduct.

PolyForm Strict License 1.0.0