elasticsearch

package
v3.100.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 19, 2026 License: AGPL-3.0 Imports: 23 Imported by: 1

README

Elasticsearch Source

Overview

The Elasticsearch source lets TruffleHog scan documents stored in an Elasticsearch cluster for secrets, credentials, and sensitive data. It connects to the cluster, finds matching indices, and reads the documents inside them.

Elasticsearch Fundamentals

What is Elasticsearch?

Elasticsearch is a search and analytics engine that stores data as documents inside indices. It is often used to store and search logs, so many logging tools send their log lines into Elasticsearch as documents.

Key Elasticsearch Terminology
Term Description
Node A single running instance of Elasticsearch. A cluster is made of one or more nodes
Cluster A group of nodes working together and holding the same data
Index A collection of documents, similar to a table in a normal database
Document A single record stored inside an index, in JSON format
Cloud ID An identifier used to connect to a managed Elastic Cloud deployment without listing node addresses
API Key A credential used to authenticate API requests without a username and password
Service Token A credential tied to a specific Elasticsearch service account
Point in Time (PIT) A snapshot of the index state used so a search can page through results safely while data keeps changing

Features

  • Multiple Connection Methods: Connect using node addresses or an Elastic Cloud ID
  • Multiple Authentication Methods: Username and password, API key, or service token
  • Index Filtering: Limit scanning to indices matching a pattern
  • Document Filtering: Limit scanning to documents matching a query, or created after a timestamp
  • Concurrent Processing: Documents are scanned in parallel across a number of workers
  • Continuous Scanning: Optionally keep scanning a live cluster and pick up new documents as they arrive
  • Duplicate Avoidance: Tracks the newest document timestamp per index so already seen documents are not scanned again on the next run. This tracking is kept in memory only, not in a database, so it only lasts for the current run (or the current best effort scan loop). Restarting TruffleHog loses this state, so use --since-timestamp again to avoid rescanning old documents.

Configuration

Connecting to a Cluster

CLI Usage:

You can connect using node addresses:

trufflehog elasticsearch --nodes https://localhost:9200

Or using an Elastic Cloud ID:

trufflehog elasticsearch --cloud-id "my-deployment:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=="

YAML Configuration:

This is the correct shape for an Elasticsearch entry in a --config file:

sources:
- connection:
    '@type': type.googleapis.com/sources.Elasticsearch
    nodes:
    - "https://localhost:9200"
  name: es-scan
  type: SOURCE_TYPE_ELASTICSEARCH
  verify: true

Note: as of this writing, pkg/config/config.go does not have a case for SOURCE_TYPE_ELASTICSEARCH in its source loader, so a config file entry like the one above will fail to load with got unexpected source type. Use the CLI flags or environment variables below until that is added.

Authentication Methods
1. Username and Password

CLI Usage:

trufflehog elasticsearch --nodes https://localhost:9200 --username elastic --password mypassword

YAML Configuration:

sources:
- connection:
    '@type': type.googleapis.com/sources.Elasticsearch
    nodes:
    - "https://localhost:9200"
    username: elastic
    password: mypassword
  name: es-scan
  type: SOURCE_TYPE_ELASTICSEARCH
  verify: true

2. API Key

CLI Usage:

trufflehog elasticsearch --nodes https://localhost:9200 --api-key xxxxxxxxxxxxxxxxxxxxxxxx

YAML Configuration:

sources:
- connection:
    '@type': type.googleapis.com/sources.Elasticsearch
    nodes:
    - "https://localhost:9200"
    api_key: "xxxxxxxxxxxxxxxxxxxxxxxx"
  name: es-scan
  type: SOURCE_TYPE_ELASTICSEARCH
  verify: true

3. Service Token

CLI Usage:

trufflehog elasticsearch --nodes https://localhost:9200 --service-token AAEAAWVsYXN0aWMv...

YAML Configuration:

sources:
- connection:
    '@type': type.googleapis.com/sources.Elasticsearch
    nodes:
    - "https://localhost:9200"
    service_token: "AAEAAWVsYXN0aWMv..."
  name: es-scan
  type: SOURCE_TYPE_ELASTICSEARCH
  verify: true

Environment variables are also supported for every value above: ELASTICSEARCH_NODES, ELASTICSEARCH_USERNAME, ELASTICSEARCH_PASSWORD, ELASTICSEARCH_CLOUD_ID, ELASTICSEARCH_API_KEY, and ELASTICSEARCH_SERVICE_TOKEN.

Filtering What Gets Scanned

Index Pattern

Limit scanning to indices matching a pattern. The default is *, which matches every index.

trufflehog elasticsearch --nodes https://localhost:9200 --index-pattern "logs-*"

Query JSON

Provide a raw Elasticsearch query to filter which documents are scanned.

trufflehog elasticsearch --nodes https://localhost:9200 --query-json '{"match": {"level": "error"}}'

Since Timestamp

Only scan documents with a @timestamp field on or after the given time. This is ignored once TruffleHog has already recorded a newer timestamp from a previous scan of that index.

trufflehog elasticsearch --nodes https://localhost:9200 --since-timestamp "2024-01-01T00:00:00Z"
Best Effort Scan

By default, TruffleHog scans the matching documents once and stops. Pass --best-effort-scan to keep the source running and repeatedly check the cluster for new documents.

trufflehog elasticsearch --nodes https://localhost:9200 --best-effort-scan

How Scanning Works

Scanning Process
  1. Index Discovery: Lists indices matching the configured index pattern
  2. Document Counting: Counts matching documents in each index using the configured query and timestamp filters
  3. Work Distribution: Splits the documents to scan into roughly equal groups across the configured number of workers
  4. Point in Time Search: Each worker opens a point in time on its assigned index and pages through matching documents in order
  5. Document Scanning: Each document's message field is sent to the detection engine as a chunk, tagged with the index name, document id, and timestamp
  6. Progress Tracking: The newest document timestamp seen in each index is recorded, so a later run or a best effort scan loop can skip documents already scanned
What Gets Scanned
  • The message field of documents in indices matching the index pattern and query filters
What Doesn't Get Scanned
  • Documents outside the configured index pattern, query, or since timestamp filter
  • Fields other than @timestamp and message

Usage Examples

Scanning a Local Cluster
trufflehog elasticsearch --nodes https://localhost:9200 --username elastic --password mypassword
Scanning an Elastic Cloud Deployment
trufflehog elasticsearch --cloud-id "my-deployment:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==" --api-key xxxxxxxxxxxxxxxxxxxxxxxx
Scanning Only Log Indices Created Since a Given Date
trufflehog elasticsearch --nodes https://localhost:9200 --index-pattern "logs-*" --since-timestamp "2024-06-01T00:00:00Z"
Continuously Scanning a Live Cluster
trufflehog elasticsearch --nodes https://localhost:9200 --best-effort-scan

Debug Logs

With debug on for this source, the scan logs:

  • V(1): Elasticsearch source initialized with the config choices (the auth method only, never the credentials, and only whether a query is set, not the query), Listed indices with index and document counts for each pass, and the scan coverage line when best effort scan skips documents to catch up.
  • V(2): Counted documents in index, and Scanning index and Finished scanning index for each index a worker scans.
  • V(3): Scanning document for each document, with its id.
  • V(4): Skipping document with a reason (already scanned, no message field, no @timestamp field, no source, no document id), Elasticsearch request failed for every failed request, a response of 400 or above or a network error like connection refused, including the ones the client retries, Scanned document count differs from expected, and No new documents, waiting in best effort scan.
  • V(5): Elasticsearch request for every request, and Fetched search page for each page of search results.

Lines carry worker and index where they apply. A failure to scan documents is logged as an error at the default level.

Troubleshooting

Common Issues

Issue: Authentication failures when connecting to the cluster Solution: Check that the username, password, API key, or service token is correct and has permission to read the target indices.


Issue: No documents are scanned even though the cluster has data Solution: Check the index pattern matches real index names, and that the query JSON or since timestamp filter is not excluding every document.


Issue: The same documents keep getting scanned again on every run Solution: This is expected the first time a since timestamp is used. On later runs TruffleHog uses the newest timestamp already seen, so the same documents are skipped unless best effort scan restarts from the same state.

Documentation

Index

Constants

View Source
const PAGE_SIZE = 10

Variables

This section is empty.

Functions

This section is empty.

Types

type Document

type Document struct {
	// contains filtered or unexported fields
}

type DocumentSearch

type DocumentSearch struct {
	// contains filtered or unexported fields
}

func (*DocumentSearch) String

func (ds *DocumentSearch) String() string

type FilterParams

type FilterParams struct {
	// contains filtered or unexported fields
}

func (*FilterParams) Query

func (fp *FilterParams) Query(latestTimestamp time.Time) (map[string]any, error)

type Index

type Index struct {
	// contains filtered or unexported fields
}

func NewIndex

func NewIndex() *Index

func (*Index) DocumentAlreadySeen

func (i *Index) DocumentAlreadySeen(document *Document) bool

func (*Index) UpdateLatestTimestampLastRun

func (i *Index) UpdateLatestTimestampLastRun()

type IndexStatus

type IndexStatus int

type Indices

type Indices struct {
	// contains filtered or unexported fields
}

func (*Indices) GetProcessedDocumentCount

func (indices *Indices) GetProcessedDocumentCount() int

Returns the number of documents processed within these indices

func (*Indices) Update

func (indices *Indices) Update(
	ctx context.Context,
	client *es.TypedClient,
) error

Updates a set of indices from an Elasticsearch cluster. If an index has been deleted it will be removed; if it's been added it'll be added; if its document count has changed (based on filterParams and latestTimestamp) it'll be updated.

func (*Indices) UpdateProcessedDocumentCount

func (indices *Indices) UpdateProcessedDocumentCount(additionalDocumentsProcessed int)

Adds documents processed to the count, used for progress

type PointInTime

type PointInTime struct {
	ID        string `json:"id"`
	KeepAlive string `json:"keep_alive"`
}

type SearchRequestBody

type SearchRequestBody struct {
	PIT         PointInTime    `json:"pit"`
	Sort        []string       `json:"sort"`
	SearchAfter []int          `json:"search_after,omitempty"`
	Query       map[string]any `json:"query,omitempty"`
}

type Source

type Source struct {
	sources.Progress
	// contains filtered or unexported fields
}

func (*Source) Chunks

func (s *Source) Chunks(
	ctx context.Context,
	chunksChan chan *sources.Chunk,
	targets ...sources.ChunkingTarget,
) error

Chunks emits chunks of bytes over a channel.

func (*Source) Init

func (s *Source) Init(
	aCtx context.Context,
	name string,
	jobId sources.JobID,
	sourceId sources.SourceID,
	verify bool,
	connection *anypb.Any,
	concurrency int,
) error

Init returns an initialized Elasticsearch source

func (*Source) JobID

func (s *Source) JobID() sources.JobID

func (*Source) SourceID

func (s *Source) SourceID() sources.SourceID

func (*Source) Type

func (s *Source) Type() sourcespb.SourceType

type UnitOfWork

type UnitOfWork struct {
	// contains filtered or unexported fields
}

func NewUnitOfWork

func NewUnitOfWork(maxDocumentCount int) UnitOfWork

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址