s3usage

package
v0.0.0-...-6be7acf Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: Apache-2.0 Imports: 6 Imported by: 0

Documentation

Index

Constants

View Source
const (
	LogTypeTask   = "task_log"
	LogTypeAgent  = "agent_log"
	LogTypeSystem = "system_log"
	LogTypeTest   = "test_log"
)

S3 log types for storage cost tracking.

View Source
const (
	S3PutRequestCost = 0.000005
	S3PartSize       = 5 * 1024 * 1024 // 5 MB in bytes - S3 multipart upload threshold

	S3BucketTypeSmall S3BucketType = "small"
	S3BucketTypeLarge S3BucketType = "large"

	S3UploadMethodWriter S3UploadMethod = "writer"
	S3UploadMethodPut    S3UploadMethod = "put"
	S3UploadMethodCopy   S3UploadMethod = "copy"

	// S3 Intelligent Tiering pricing constants and tier transition thresholds.
	// Transition days (30, 90) are defined by AWS S3 Intelligent Tiering:
	// https://aws.amazon.com/s3/storage-classes/intelligent-tiering/
	S3StandardPricePerGBMonth = 0.023
	S3IAPricePerGBMonth       = 0.0125
	S3ArchivePricePerGBMonth  = 0.004
	S3TransitionToIADays      = 30
	S3TransitionToArchiveDays = 90
	S3BytesPerGB              = 1024 * 1024 * 1024
	S3DaysPerMonth            = 30.0
)

Variables

This section is empty.

Functions

func CalculateS3PutCostWithConfig

func CalculateS3PutCostWithConfig(ctx context.Context, putRequests int, costConfig *evergreen.CostConfig) (standard, adjusted float64)

CalculateS3PutCostWithConfig calculates the S3 PUT request cost, returning both the standard (non-discounted) and adjusted (discounted) values. If config is nil or the discount is invalid, adjusted is returned as 0.

func CalculateS3StorageCostWithConfig

func CalculateS3StorageCostWithConfig(ctx context.Context, uploadBytes int64, expirationDays int, costConfig *evergreen.CostConfig) (standard, adjusted float64)

CalculateS3StorageCostWithConfig calculates the S3 storage cost for uploadBytes over their retention period using the bucket's Intelligent Tiering schedule. expirationDays must be positive; buckets without a lifecycle expiration policy have no defined retention period and cannot have their cost calculated, so this function returns 0 for them. Returns both the standard (non-discounted) and adjusted (discounted) values. If config is nil, standard is still computed but adjusted is returned as 0.

func ComputePerFileExtremes

func ComputePerFileExtremes(files []FileMetrics) (maxPuts, minPuts int)

ComputePerFileExtremes returns the max and min PutRequests across all uploaded files.

Types

type ArtifactIncrementOptions

type ArtifactIncrementOptions struct {
	PutRequests int
	UploadBytes int64
	FileCount   int
	MaxPuts     int
	MinPuts     int
	Bucket      string
	AWSRoleARN  string
	// AWSAccountID is the resolved account ID when AWSRoleARN is empty (key+secret auth).
	AWSAccountID string
	Files        []FileMetrics
	// DevprodOwnedAWSAccountIDs, when non-empty, limits recording to uploads in these accounts.
	DevprodOwnedAWSAccountIDs []string
}

ArtifactIncrementOptions holds the parameters for incrementing artifact upload metrics.

type ArtifactMetrics

type ArtifactMetrics struct {
	S3UploadMetrics `bson:",inline"`
	// Count is the total number of artifacts uploaded per task.
	Count int `bson:"count,omitempty" json:"count,omitempty"`
	// ArtifactWithMaxPutRequests is the highest PUT request count for a single artifact across all s3.put invocations per task.
	ArtifactWithMaxPutRequests int `bson:"max_put_requests_per_file,omitempty" json:"max_put_requests_per_file,omitempty"`
	// ArtifactWithMinPutRequests is the lowest PUT request count for a single artifact across all s3.put invocations per task.
	ArtifactWithMinPutRequests int `bson:"min_put_requests_per_file,omitempty" json:"min_put_requests_per_file,omitempty"`
	// BytesByBucketAndKey groups per-file byte metrics by S3 bucket.
	BytesByBucketAndKey []BucketFileMetrics `bson:"bytes_by_bucket_and_key,omitempty" json:"bytes_by_bucket_and_key,omitempty"`
}

ArtifactMetrics tracks artifact upload metrics with an additional file count.

type BucketFileMetrics

type BucketFileMetrics struct {
	Bucket string `bson:"bucket" json:"bucket"`
	// AWSRoleARN is the IAM role ARN used for artifact uploads (when assume-role is configured).
	AWSRoleARN string `bson:"aws_role_arn,omitempty" json:"aws_role_arn,omitempty"`
	// AWSAccountID is the owning account, resolved by the agent when AWSRoleARN is empty.
	AWSAccountID string      `bson:"aws_account_id,omitempty" json:"aws_account_id,omitempty"`
	Files        []FileBytes `bson:"files" json:"files" extensions:"x-nullable"`
}

BucketFileMetrics groups per-file byte metrics for a single S3 bucket.

type FileBytes

type FileBytes struct {
	FileKey     string `bson:"file_key" json:"file_key"`
	Bytes       int64  `bson:"bytes" json:"bytes"`
	PutRequests int    `bson:"put_requests,omitempty" json:"put_requests,omitempty"`
}

FileBytes tracks bytes uploaded for a single S3 file key.

type FileMetrics

type FileMetrics struct {
	LocalPath     string
	RemotePath    string
	FileSizeBytes int64
	PutRequests   int
}

FileMetrics contains metrics for a single uploaded file.

func BuildFileMetrics

func BuildFileMetrics(ctx context.Context, logger grip.Journaler, localPath, remotePath string, putRequests int) (FileMetrics, int64)

BuildFileMetrics constructs a FileMetrics entry for a successfully uploaded file, statting the file for size. If the stat fails, logs a warning and uses zero size.

type LogMetrics

type LogMetrics struct {
	S3UploadMetrics `bson:",inline"`
	Task            LogTypeMetrics `bson:"task_log,omitempty" json:"task_log,omitempty"`
	Agent           LogTypeMetrics `bson:"agent_log,omitempty" json:"agent_log,omitempty"`
	System          LogTypeMetrics `bson:"system_log,omitempty" json:"system_log,omitempty"`
	Test            LogTypeMetrics `bson:"test_log,omitempty" json:"test_log,omitempty"`
}

LogMetrics tracks log upload metrics broken down by log type.

type LogTypeMetrics

type LogTypeMetrics struct {
	LogKey      string `bson:"log_key,omitempty" json:"log_key,omitempty"`
	Bytes       int64  `bson:"bytes,omitempty" json:"bytes,omitempty"`
	PutRequests int    `bson:"put_requests,omitempty" json:"put_requests,omitempty"`
}

LogTypeMetrics holds the S3 key, byte count, and PUT count for a single log type.

type S3BucketType

type S3BucketType string

type S3UploadMethod

type S3UploadMethod string

type S3UploadMetrics

type S3UploadMetrics struct {
	PutRequests int   `bson:"put_requests,omitempty" json:"put_requests,omitempty"`
	UploadBytes int64 `bson:"upload_bytes,omitempty" json:"upload_bytes,omitempty"`
}

S3UploadMetrics tracks common S3 upload metrics shared across upload types.

type S3Usage

type S3Usage struct {
	Artifacts ArtifactMetrics `bson:"artifacts,omitempty" json:"artifacts,omitempty"`
	Logs      LogMetrics      `bson:"logs,omitempty" json:"logs,omitempty"`
	// contains filtered or unexported fields
}

S3Usage tracks S3 API usage for cost calculation. Call Init before sharing across goroutines.

func (*S3Usage) IncrementArtifacts

func (s *S3Usage) IncrementArtifacts(opts ArtifactIncrementOptions)

IncrementArtifacts updates aggregate artifact upload metrics after an s3.put command.

func (*S3Usage) IncrementLogs

func (s *S3Usage) IncrementLogs(putRequests int, uploadBytes int64, logType, logKey string)

IncrementLogs increments aggregate and per-type log upload metrics for cost tracking. Test logs share a bucket/prefix, so LogKey stores only the most recently written key.

func (*S3Usage) Init

func (s *S3Usage) Init()

Init enables concurrent-access protection. Call once at agent setup before sharing.

func (*S3Usage) IsZero

func (s *S3Usage) IsZero() bool

IsZero implements bsoncodec.Zeroer for BSON marshalling.

func (*S3Usage) Snapshot

func (s *S3Usage) Snapshot(ctx context.Context) S3Usage

Snapshot returns a locked copy for reporting. Logs a critical alert if Init was never called, since concurrent senders may have written without protection throughout the task.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址