AWS Big Data Blog

Category: Database

Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables

Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables

Amazon Redshift keeps only 7 days of system table history. This post pairs Datadog Data Observability with the new Amazon Redshift system table integration with Amazon S3 Tables, so you can retain query history for the long term and analyze lineage, query performance, and data quality without adding load to your warehouses.

Multi-statement transactions for Apache Iceberg tables in Amazon Redshift

Multi-statement transactions for Apache Iceberg tables in Amazon Redshift

Amazon Redshift now supports multi-statement transactions for Apache Iceberg tables, so you can run several DML statements against one Iceberg table inside a single BEGIN…COMMIT and publish them as one atomic Iceberg snapshot. This post explains why one commit per statement makes multi-step pipelines fragile, what you can run in a transaction, and how it performs.

Define metrics once and keep them correct with the dbt Semantic Layer on Amazon Redshift

Define metrics once and keep them correct with the dbt Semantic Layer on Amazon Redshift

Amazon Redshift has no native semantic layer, yet teams still need consistent, governed metric definitions across BI tools. This post shows how to define a metric once with the open-source dbt Semantic Layer (MetricFlow) on Amazon Redshift so it stays correct at any grain, and proves a non-additive weekly average comes out right ($280, not $550).

Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

Learn how to migrate Google BigQuery workloads to an open lakehouse on AWS built on Apache Iceberg and Amazon S3. This post explains why an open lakehouse future-proofs your data, presents the target architecture on AWS, and walks through a step-by-step migration: assess, design, convert, migrate, cut over, and optimize.

Resolving concurrent write conflicts for Apache Iceberg tables in Amazon Redshift

Resolving concurrent write conflicts for Apache Iceberg tables in Amazon Redshift

Amazon Redshift now automatically resolves many concurrent writes to Apache Iceberg tables that don’t actually overlap, committing changes that previously failed and had to retry. Learn why these writes conflict, how the engine rebases non-overlapping ones across operation, partition, and file checks, and the throughput gains it delivers.

Materialize once, query anywhere: Introducing Iceberg materialized views in Amazon Redshift

Materialize once, query anywhere: Introducing Iceberg materialized views in Amazon Redshift

Amazon Redshift now supports Iceberg materialized views. Compute an aggregation once in Amazon Redshift and store the result as a standard Apache Iceberg table in Amazon S3, queryable by Amazon Athena, Apache Spark, Amazon SageMaker, and AWS Glue. This post covers use cases, incremental refresh, and a step-by-step getting-started guide.

Run an automated operational review with the Amazon Redshift MCP server

Run an automated operational review with the Amazon Redshift MCP server

Amazon Redshift automates much of its own tuning, but periodic operational reviews still pay off. Learn how the review_cluster tool in the open-source Amazon Redshift MCP server runs a full cluster diagnostic from one natural-language request, evaluating 12 diagnostic areas and returning prioritized, documentation-linked recommendations in minutes.

Aurora PostgreSQL zero-ETL integration with Amazon SageMaker

Aurora PostgreSQL zero-ETL integration with Amazon SageMaker

Amazon Aurora PostgreSQL zero-ETL integration with Amazon SageMaker replicates your operational data to a lakehouse in near real time, without building custom ETL pipelines. Learn the architecture and change data capture mechanics, then set up the integration and query your data in Amazon SageMaker.