AWS Big Data Blog

Today we are announcing that Amazon has signed a definitive agreement to acquire DuckLabs, the Amsterdam-based company behind the open-source analytical database DuckDB. We expect the transaction to close shortly, subject to customary closing conditions. Hannes Mühleisen and Mark Raasveldt, who created DuckDB and co-founded DuckLabs, will continue leading the team and the open-source project’s technical direction as part of AWS. The DuckDB open-source project will also continue to be driven by the DuckLabs team, remain open source under the independent Foundation (the non-profit that oversees DuckDB), and available under the MIT license as it does today.

How Confluent modernized its Amazon OpenSearch Service observability hot tier, ingesting hundreds of terabytes of logs per day, by migrating from r6gd to the purpose-built oi2 instance family. A measure-first approach cut the fleet 46%, roughly halved indexing latency, and cleared write backpressure, lowering cost even at a higher per-node price.

Validating a streaming pipeline beyond current peak takes production-scale test data. This post shows how the stream multiplier pattern on Amazon Managed Service for Apache Flink turns a small pre-recorded sample into sustained, production-scale load while preserving the statistical properties of real traffic.

Agentic Data Ingestion is an open-source solution that uses AI agents on Amazon Bedrock to automate data pipeline creation, from file upload to queryable tables. Point it at a file or database table, and foundation models propose the field mapping, generate and test the AWS Glue transform, and run the job behind human approval gates.

Workday Data Cloud now integrates with AWS, giving developers bi-directional, zero-copy access to Workday’s governed people and money data from the AWS analytics and AI services they already use, without brittle pipelines or data duplication.

Amazon Redshift has no native semantic layer, yet teams still need consistent, governed metric definitions across BI tools. This post shows how to define a metric once with the open-source dbt Semantic Layer (MetricFlow) on Amazon Redshift so it stays correct at any grain, and proves a non-additive weekly average comes out right ($280, not $550).

The CUDOS 5.10 dashboard adds an Amazon Redshift RA3-to-RG section that helps you assess the cost-efficiency opportunity of moving to Graviton-based RG instances, track adoption as your instance-generation mix shifts, and quantify the gains already achieved on RG.

Amazon Athena now offers system tables: SQL-queryable, per-query execution metadata stored as Apache Iceberg in your own account. This post walks through what system tables are, how to turn them on, and how to use them for usage analysis, cost attribution, performance trends, audit, and capacity optimization without custom pipelines.

Storing large semi-structured data such as JSON and arrays in a data warehouse often means splitting or truncating your data. With PartiQL and the SUPER data type, Amazon Redshift is now equipped to handle these issues. In this post, we walk you through three key SUPER data type enhancements that simplify semi-structured data workflows and reduce common migration challenges.

Governed, multi-tenant data access is a key requirement for SaaS and B2C products that embed AI-powered analytics. This post shows how to combine Amazon Cognito and AWS Lake Formation on Amazon Bedrock AgentCore to enforce row- and column-level data access for external users, with a minimal-privilege Lambda acting as a credential broker.

Learn how to migrate Google BigQuery workloads to an open lakehouse on AWS built on Apache Iceberg and Amazon S3. This post explains why an open lakehouse future-proofs your data, presents the target architecture on AWS, and walks through a step-by-step migration: assess, design, convert, migrate, cut over, and optimize.