s3store

package
v1.4.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 22, 2026 License: Apache-2.0 Imports: 29 Imported by: 0

README

Amazon S3 blobstore

Configuration

See https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials on how to set up authentication against s3

Enabling archival is done by using the configuration below. Region and bucket URI are required

archival:
  history:
    status: "enabled"
    enableRead: true
    provider:
      s3store:
        region: "us-east-1"
  visibility:
    status: "enabled"
    enableRead: true
    provider:
      s3store:
        region: "us-east-1"

domainDefaults:
  archival:
    history:
      status: "enabled"
      URI: "s3://<bucket-name>"
    visibility:
      status: "enabled"
      URI: "s3://<bucket-name>"

URI schemes

Two URI schemes are supported:

  • s3://<bucket-name>[/<path>] — plain S3 bucket. The hostname is the bucket name and is passed directly to the AWS SDK. This is the original scheme.
  • s3-ap://<account-id>/<access-point-name>[/<path>] — S3 access point. The region is taken from the archiver's configured region, so the same URI works in every region. This is useful for global domains where the archival URI must be identical across clusters.

At request time, an s3-ap URI is reconstructed into an access point ARN of the form arn:aws:s3:<region>:<account-id>:accesspoint/<access-point-name> and passed as the Bucket parameter to S3 SDK calls. Each cluster contributes its own region to the ARN.

Example — registering a global domain with an access point archival URI:

cadence --do my-domain domain register \
  --history_uri    "s3-ap://710914175400/cadence-archival/prod" \
  --visibility_uri "s3-ap://710914175400/cadence-archival/prod" \
  --history_archival_status enabled \
  --visibility_archival_status enabled \
  --gd true --cl primary,secondary --ac primary --rd 7

In us-east-1 this resolves to the bucket arn:aws:s3:us-east-1:710914175400:accesspoint/cadence-archival; in eu-west-1 the same URI resolves to arn:aws:s3:eu-west-1:710914175400:accesspoint/cadence-archival.

Access point IAM and HeadBucket

At domain-registration time the frontend calls HeadBucket against the constructed ARN to verify the URI. HeadBucket on an access point is evaluated against the underlying bucket's policy, not the access point policy — so granting s3:* on the access point alone is not enough. The role used by the Cadence services needs s3:ListBucket on the underlying bucket (arn:aws:s3:::<underlying-bucket-name>) for the validation check to pass.

The ARN partition is inferred from the region prefix, so the same URI works across aws, aws-cn (China), aws-us-gov (GovCloud), and aws-iso/ aws-iso-b (Secret/Top-Secret) partitions.

Limitations
  • The s3-ap scheme requires the archiver's region config to be set. Plain s3:// URIs work without it (the SDK falls back to environment / instance metadata).
  • s3ForcePathStyle: true is incompatible with access points. Leave it unset (the default) when using s3-ap://.
  • Multi-Region Access Points (MRAP) are not currently supported. MRAP ARNs have no region component and require SigV4A signing, which is a separate scheme.

Visibility query syntax

You can query the visibility store by using the cadence workflow listarchived command

The syntax for the query is based on SQL

Supported column names are

  • WorkflowID String
  • WorkflowTypeName String
  • StartTime Date
  • CloseTime Date
  • SearchPrecision String - Day, Hour, Minute, Second

WorkflowID or WorkflowTypeName is required. If filtering on date use StartTime or CloseTime in combination with SearchPrecision.

Searching for a record will be done in times in the UTC timezone

SearchPrecision specifies what range you want to search for records. If you use SearchPrecision = 'Day' it will search all records starting from 2020-01-21T00:00:00Z to 2020-01-21T59:59:59Z

Limitations
  • The only operator supported is = due to how records are stored in s3.
Example

Searches for all records done in day 2020-01-21 with the specified workflow id

./cadence --do samples-domain workflow listarchived -q "StartTime = '2020-01-21T00:00:00Z' AND WorkflowID='workflow-id' AND SearchPrecision='Day'"

Storage in S3

Workflow runs are stored in s3 using the following structure

s3://<bucket-name>/<domain-id>/
	history/<workflow-id>/<run-id>
	visibility/
            workflowTypeName/<workflow-type-name>/
                startTimeout/2020-01-21T16:16:11Z/<run-id>
                closeTimeout/2020-01-21T16:16:11Z/<run-id>
            workflowID/<workflow-id>/
                startTimeout/2020-01-21T16:16:11Z/<run-id>
                closeTimeout/2020-01-21T16:16:11Z/<run-id>

For s3-ap:// URIs, the path component after the access point name plays the same role as the path under a bucket name. For example, s3-ap://710914175400/cadence-archival/prod produces objects under prod/<domain-id>/history/... inside the underlying bucket.

Using localstack for local development

  1. Install awscli from here
  2. Install localstack from here
  3. Launch localstack with SERVICES=s3 localstack start
  4. Create a bucket using aws --endpoint-url=http://localhost:4572 s3 mb s3://cadence-development
  5. Configure archival and domainDefaults with the following configuration
archival:
  history:
    status: "enabled"
    enableRead: true
    provider:
      s3store:
        region: "us-east-1"
        endpoint: "http://127.0.0.1:4572"
        s3ForcePathStyle: true
  visibility:
    status: "enabled"
    enableRead: true
    provider:
      s3store:
        region: "us-east-1"
        endpoint: "http://127.0.0.1:4572"
        s3ForcePathStyle: true

domainDefaults:
  archival:
    history:
      status: "enabled"
      URI: "s3://cadence-development"
    visibility:
      status: "enabled"
      URI: "s3://cadence-development"

Documentation

Overview

Package s3store is a generated GoMock package.

Index

Constants

View Source
const (
	// URIScheme is the scheme for the s3 implementation using a plain bucket name.
	URIScheme = "s3"
	// URISchemeAccessPoint is the scheme for S3 access point URIs.
	// Format: s3-ap://<account-id>/<access-point-name>[/<path>]
	// The region is injected from the archiver's configured AWS region,
	// so the same URI works across all regions (useful for global domains).
	URISchemeAccessPoint = "s3-ap"
)
View Source
const (
	WorkflowTypeName = "WorkflowTypeName"
	WorkflowID       = "WorkflowID"
	StartTime        = "StartTime"
	CloseTime        = "CloseTime"
	SearchPrecision  = "SearchPrecision"
)

All allowed fields for filtering

View Source
const (
	PrecisionDay    = "Day"
	PrecisionHour   = "Hour"
	PrecisionMinute = "Minute"
	PrecisionSecond = "Second"
)

Precision specific values

Variables

This section is empty.

Functions

func NewHistoryArchiver

func NewHistoryArchiver(
	container *archiver.HistoryBootstrapContainer,
	config *Config,
) (archiver.HistoryArchiver, error)

NewHistoryArchiver creates a new archiver.HistoryArchiver based on s3

func NewVisibilityArchiver

func NewVisibilityArchiver(
	container *archiver.VisibilityBootstrapContainer,
	config *Config,
) (archiver.VisibilityArchiver, error)

NewVisibilityArchiver creates a new archiver.VisibilityArchiver based on s3

Types

type Config

type Config struct {
	Region           string  `yaml:"region"`
	Endpoint         *string `yaml:"endpoint"`
	S3ForcePathStyle bool    `yaml:"s3ForcePathStyle"`
}

Config contains the config for S3 archiver

type MockQueryParser

type MockQueryParser struct {
	// contains filtered or unexported fields
}

MockQueryParser is a mock of QueryParser interface.

func NewMockQueryParser

func NewMockQueryParser(ctrl *gomock.Controller) *MockQueryParser

NewMockQueryParser creates a new mock instance.

func (*MockQueryParser) EXPECT

EXPECT returns an object that allows the caller to indicate expected use.

func (*MockQueryParser) Parse

func (m *MockQueryParser) Parse(query string) (*parsedQuery, error)

Parse mocks base method.

type MockQueryParserMockRecorder

type MockQueryParserMockRecorder struct {
	// contains filtered or unexported fields
}

MockQueryParserMockRecorder is the mock recorder for MockQueryParser.

func (*MockQueryParserMockRecorder) Parse

func (mr *MockQueryParserMockRecorder) Parse(query any) *gomock.Call

Parse indicates an expected call of Parse.

type QueryParser

type QueryParser interface {
	Parse(query string) (*parsedQuery, error)
}

QueryParser parses a limited SQL where clause into a struct

func NewQueryParser

func NewQueryParser() QueryParser

NewQueryParser creates a new query parser for filestore

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址