Documentation
¶
Index ¶
- Constants
- func CalculateS3PutCostWithConfig(ctx context.Context, putRequests int, costConfig *evergreen.CostConfig) (standard, adjusted float64)
- func CalculateS3StorageCostWithConfig(ctx context.Context, uploadBytes int64, expirationDays int, ...) (standard, adjusted float64)
- func ComputePerFileExtremes(files []FileMetrics) (maxPuts, minPuts int)
- type ArtifactIncrementOptions
- type ArtifactMetrics
- type BucketFileMetrics
- type FileBytes
- type FileMetrics
- type LogMetrics
- type LogTypeMetrics
- type S3BucketType
- type S3UploadMethod
- type S3UploadMetrics
- type S3Usage
Constants ¶
const ( LogTypeTask = "task_log" LogTypeAgent = "agent_log" LogTypeSystem = "system_log" LogTypeTest = "test_log" )
S3 log types for storage cost tracking.
const ( S3PutRequestCost = 0.000005 S3PartSize = 5 * 1024 * 1024 // 5 MB in bytes - S3 multipart upload threshold S3BucketTypeSmall S3BucketType = "small" S3BucketTypeLarge S3BucketType = "large" S3UploadMethodWriter S3UploadMethod = "writer" S3UploadMethodPut S3UploadMethod = "put" S3UploadMethodCopy S3UploadMethod = "copy" // S3 Intelligent Tiering pricing constants and tier transition thresholds. // Transition days (30, 90) are defined by AWS S3 Intelligent Tiering: // https://aws.amazon.com/s3/storage-classes/intelligent-tiering/ S3StandardPricePerGBMonth = 0.023 S3IAPricePerGBMonth = 0.0125 S3ArchivePricePerGBMonth = 0.004 S3TransitionToIADays = 30 S3TransitionToArchiveDays = 90 S3BytesPerGB = 1024 * 1024 * 1024 S3DaysPerMonth = 30.0 )
Variables ¶
This section is empty.
Functions ¶
func CalculateS3PutCostWithConfig ¶
func CalculateS3PutCostWithConfig(ctx context.Context, putRequests int, costConfig *evergreen.CostConfig) (standard, adjusted float64)
CalculateS3PutCostWithConfig calculates the S3 PUT request cost, returning both the standard (non-discounted) and adjusted (discounted) values. If config is nil or the discount is invalid, adjusted is returned as 0.
func CalculateS3StorageCostWithConfig ¶
func CalculateS3StorageCostWithConfig(ctx context.Context, uploadBytes int64, expirationDays int, costConfig *evergreen.CostConfig) (standard, adjusted float64)
CalculateS3StorageCostWithConfig calculates the S3 storage cost for uploadBytes over their retention period using the bucket's Intelligent Tiering schedule. expirationDays must be positive; buckets without a lifecycle expiration policy have no defined retention period and cannot have their cost calculated, so this function returns 0 for them. Returns both the standard (non-discounted) and adjusted (discounted) values. If config is nil, standard is still computed but adjusted is returned as 0.
func ComputePerFileExtremes ¶
func ComputePerFileExtremes(files []FileMetrics) (maxPuts, minPuts int)
ComputePerFileExtremes returns the max and min PutRequests across all uploaded files.
Types ¶
type ArtifactIncrementOptions ¶
type ArtifactIncrementOptions struct {
PutRequests int
UploadBytes int64
FileCount int
MaxPuts int
MinPuts int
Bucket string
AWSRoleARN string
// AWSAccountID is the resolved account ID when AWSRoleARN is empty (key+secret auth).
AWSAccountID string
Files []FileMetrics
// DevprodOwnedAWSAccountIDs, when non-empty, limits recording to uploads in these accounts.
DevprodOwnedAWSAccountIDs []string
}
ArtifactIncrementOptions holds the parameters for incrementing artifact upload metrics.
type ArtifactMetrics ¶
type ArtifactMetrics struct {
S3UploadMetrics `bson:",inline"`
// Count is the total number of artifacts uploaded per task.
Count int `bson:"count,omitempty" json:"count,omitempty"`
// ArtifactWithMaxPutRequests is the highest PUT request count for a single artifact across all s3.put invocations per task.
ArtifactWithMaxPutRequests int `bson:"max_put_requests_per_file,omitempty" json:"max_put_requests_per_file,omitempty"`
// ArtifactWithMinPutRequests is the lowest PUT request count for a single artifact across all s3.put invocations per task.
ArtifactWithMinPutRequests int `bson:"min_put_requests_per_file,omitempty" json:"min_put_requests_per_file,omitempty"`
// BytesByBucketAndKey groups per-file byte metrics by S3 bucket.
BytesByBucketAndKey []BucketFileMetrics `bson:"bytes_by_bucket_and_key,omitempty" json:"bytes_by_bucket_and_key,omitempty"`
}
ArtifactMetrics tracks artifact upload metrics with an additional file count.
type BucketFileMetrics ¶
type BucketFileMetrics struct {
Bucket string `bson:"bucket" json:"bucket"`
// AWSRoleARN is the IAM role ARN used for artifact uploads (when assume-role is configured).
AWSRoleARN string `bson:"aws_role_arn,omitempty" json:"aws_role_arn,omitempty"`
// AWSAccountID is the owning account, resolved by the agent when AWSRoleARN is empty.
AWSAccountID string `bson:"aws_account_id,omitempty" json:"aws_account_id,omitempty"`
Files []FileBytes `bson:"files" json:"files" extensions:"x-nullable"`
}
BucketFileMetrics groups per-file byte metrics for a single S3 bucket.
type FileBytes ¶
type FileBytes struct {
FileKey string `bson:"file_key" json:"file_key"`
Bytes int64 `bson:"bytes" json:"bytes"`
PutRequests int `bson:"put_requests,omitempty" json:"put_requests,omitempty"`
}
FileBytes tracks bytes uploaded for a single S3 file key.
type FileMetrics ¶
FileMetrics contains metrics for a single uploaded file.
func BuildFileMetrics ¶
func BuildFileMetrics(ctx context.Context, logger grip.Journaler, localPath, remotePath string, putRequests int) (FileMetrics, int64)
BuildFileMetrics constructs a FileMetrics entry for a successfully uploaded file, statting the file for size. If the stat fails, logs a warning and uses zero size.
type LogMetrics ¶
type LogMetrics struct {
S3UploadMetrics `bson:",inline"`
Task LogTypeMetrics `bson:"task_log,omitempty" json:"task_log,omitempty"`
Agent LogTypeMetrics `bson:"agent_log,omitempty" json:"agent_log,omitempty"`
System LogTypeMetrics `bson:"system_log,omitempty" json:"system_log,omitempty"`
Test LogTypeMetrics `bson:"test_log,omitempty" json:"test_log,omitempty"`
}
LogMetrics tracks log upload metrics broken down by log type.
type LogTypeMetrics ¶
type LogTypeMetrics struct {
LogKey string `bson:"log_key,omitempty" json:"log_key,omitempty"`
Bytes int64 `bson:"bytes,omitempty" json:"bytes,omitempty"`
PutRequests int `bson:"put_requests,omitempty" json:"put_requests,omitempty"`
}
LogTypeMetrics holds the S3 key, byte count, and PUT count for a single log type.
type S3BucketType ¶
type S3BucketType string
type S3UploadMethod ¶
type S3UploadMethod string
type S3UploadMetrics ¶
type S3UploadMetrics struct {
PutRequests int `bson:"put_requests,omitempty" json:"put_requests,omitempty"`
UploadBytes int64 `bson:"upload_bytes,omitempty" json:"upload_bytes,omitempty"`
}
S3UploadMetrics tracks common S3 upload metrics shared across upload types.
type S3Usage ¶
type S3Usage struct {
Artifacts ArtifactMetrics `bson:"artifacts,omitempty" json:"artifacts,omitempty"`
Logs LogMetrics `bson:"logs,omitempty" json:"logs,omitempty"`
// contains filtered or unexported fields
}
S3Usage tracks S3 API usage for cost calculation. Call Init before sharing across goroutines.
func (*S3Usage) IncrementArtifacts ¶
func (s *S3Usage) IncrementArtifacts(opts ArtifactIncrementOptions)
IncrementArtifacts updates aggregate artifact upload metrics after an s3.put command.
func (*S3Usage) IncrementLogs ¶
IncrementLogs increments aggregate and per-type log upload metrics for cost tracking. Test logs share a bucket/prefix, so LogKey stores only the most recently written key.
func (*S3Usage) Init ¶
func (s *S3Usage) Init()
Init enables concurrent-access protection. Call once at agent setup before sharing.