View a markdown version of this page

Streaming Airline Ticket Shopping Insights - Streaming Airline Ticket Shopping Insights
Streaming airline ticket shopping insights diagramFurther readingDiagram history

Streaming Airline Ticket Shopping Insights

Publication date: March 10, 2021 (Diagram history)

3Victors implemented an AWS architecture to capture and durably store over 10 TB of daily streamed air shopping data into a data lake. Dozens of extract, transform, and load (ETL) jobs run at regular intervals.

The implementation provides an extensible, real-time predictive analytics pipeline for demand forecasting and deal classification.

Streaming airline ticket shopping insights diagram

Architecture for streaming airline ticket shopping insights on AWS.

The following steps describe the architecture:

  1. Route 53 directs per-stream vendor content to source-specific AWS Elastic Beanstalk application orchestration. Handle 20,000 per second HTTPS POST transactions with autoscaling on Amazon EC2 Spot Instances. Monitor with CloudWatch. Log statistics to Amazon Redshift through Amazon Data Firehose.

  2. Transform captured pricing data into a common format. Place data on a durable 24-hour Kinesis streaming buffer across 100 or more shards.

  3. Two concurrent pipelines fan out from the streaming buffer.

  4. A data lake pipeline persists data in Amazon S3 Intelligent-Tiering. Directory paths support partitioning and cross-account access. Update schema-on-read mappings in AWS Glue metastore for Athena analytics. Lifecycle policies archive to Amazon S3 Glacier Deep Archive.

  5. Periodic ETL jobs deposit results into cross-account Data Ponds by using Amazon EMR transient compute with Spot Instance fleets. AWS Glue metastore provides Amazon Redshift Spectrum access integrated with Amazon Redshift for business intelligence tool access.

  6. A real-time analytics pipeline fans out to Kinesis stream buffers. Populate Amazon Aurora (row) and Amazon Neptune (graph) databases for customer API access through serverless API Gateway and Lambda. Optimize database I/O with Amazon ElastiCache.

Further reading

For additional information, see the following resources:

Diagram history

To receive updates about this reference architecture diagram, subscribe to the RSS feed.

ChangeDescriptionDate

Initial publication

Reference architecture diagram first published.

March 10, 2021

RSS subscription requirement

To subscribe to RSS updates, you must have an RSS plugin enabled for the browser you are using.

⌂ 换网址