Autoscaling Asynchronous Job Queues
Publication date: April 15, 2021 (Diagram history)
This architecture shows how to perform background processing jobs such as document generation, extract, transform, and load (ETL) tasks, and inference with AWS Fargate. You use Amazon Simple Queue Service, Amazon DynamoDB, and AWS Lambda to manage the job lifecycle.
Autoscaling Asynchronous Job Queues
The following steps describe the architecture:
-
Each job type requires an Amazon Elastic Container Service instance (desiredCount = 0), an Amazon SQS queue, and associated Amazon CloudWatch alarms.
-
Amazon API Gateway provides an interface to add, query, and cancel jobs.
-
The add job Lambda function creates the job state item in DynamoDB and enqueues the job ID on Amazon SQS.
-
The Amazon SQS job queue contains messages with the IDs of the jobs to be processed.
-
CloudWatch monitors the Amazon SQS queue depth. Alarms are triggered when the depth exceeds a configurable value and when the queue is empty.
-
Amazon ECS responds to the CloudWatch alarms, changing the desired count and scaling the number of parallel jobs.
-
AWS Fargate runs job processing containers from Amazon Elastic Container Registry (Amazon ECR). The containers poll Amazon SQS for job IDs, read the job item from DynamoDB, and perform processing, updating the job status in DynamoDB and publishing to an Amazon Simple Notification Service topic on a status change.
-
A DynamoDB table stores the state, parameters, and results of the jobs by job ID.
-
Amazon ECR stores the job processing container images. An Amazon SNS topic publishes job items on state change notifications to subscribers.
-
Two control Lambda functions allow jobs to be queried and canceled from API Gateway by reading or updating the job item in DynamoDB.
Further reading
For additional information, refer to the following resources:
Diagram history
To be notified about updates to this reference architecture diagram, subscribe to the RSS feed.
| Change | Description | Date |
|---|---|---|
Initial publication | Reference architecture diagram first published. | April 15, 2021 |
Note
To subscribe to RSS updates, you must have an RSS plugin enabled for the browser you are using.
