[2025.03 and earlier] EFS File System Onboard Snapshot Rate Exceeded Issue

Users who have more than 10 EFS file systems within their account/region may experience operational limitations. These limitations can include difficulties in onboard file systems and unsuccessful snapshot operations.

Onboarding File System Symptom

When attempting to onboard a file system through the File Systems page and clicking Onboard File System button will result in the drop down selection being frozen. During this failure, by checking the web browser’s dev tools, you will see EFS DescribeMountTarget API calls failing with a response {"message":"Rate exceeded"}.

Snapshot Failing to Apply Symptom

Attempting to apply a snapshot will result in a failure and will enter the status ROLLBACK_COMPLETE. During this failure, the following log will be emitted within the /{env-name}/cluster-manager CloudWatch Log Group [ERROR] [apply-snapshot] Rate exceeded.

Both of these issues are produced by the same root cause which is DescribeMountTarget API calls being throttled by EFS. Users who have a higher number of EFS file systems within their account/region may run into these issues.

This issue affects RES versions 2025.03 and earlier.

1. Download patch.py and efs_filesystem_fix.patch by replacing <output-directory> with the directory to download the patch script and patch file and <environment-name> with the name of your RES environment in the command below:
   1. The patch has been tested on RES version 2025.03 however earlier versions of RES such as 2023.11 or 2024.01 may not be compatible.
   2. The patch script requires AWS CLI v2, Python 3.9.16 or above, and Boto3.
   3. Configure the AWS CLI for the account / region where RES is deployed, and make sure that you have S3 permissions to write to the bucket created by RES.

OUTPUT_DIRECTORY=<output-directory>
ENVIRONMENT_NAME=<environment-name>
RES_VERSION=<res-version> # e.g. 2025.03

mkdir -p ${OUTPUT_DIRECTORY}
curl https://research-engineering-studio-us-east-1.s3.amazonaws.com/releases/${RES_VERSION}/patch_scripts/patch.py --output ${OUTPUT_DIRECTORY}/patch.py
curl https://research-engineering-studio-us-east-1.s3.amazonaws.com/releases/${RES_VERSION}/patch_scripts/patches/efs_filesystem_fix.patch --output ${OUTPUT_DIRECTORY}/efs_filesystem_fix.patch

2. Navigate to the directory where the patch script and patch file are downloaded. Run the following patch command:

python3 ${OUTPUT_DIRECTORY}/patch.py --environment-name ${ENVIRONMENT_NAME} --res-version ${RES_VERSION} --module cluster-manager --patch ${OUTPUT_DIRECTORY}/efs_filesystem_fix.patch

3. Restart the Cluster Manager instance for your environment. You may also terminate the instance from the Amazon EC2 Management Console.

INSTANCE_ID=$(aws ec2 describe-instances \
    --filters \
    Name=tag:Name,Values=${ENVIRONMENT_NAME}-cluster-manager \
    Name=tag:res:EnvironmentName,Values=${ENVIRONMENT_NAME}\
    --query "Reservations[0].Instances[0].InstanceId" \
    --output text)
 
aws ec2 terminate-instances --instance-ids ${INSTANCE_ID}

4. Verify the Cluster Manager instance status by checking the activity of the auto scaling group starting with the name -cluster-manager-asg . Wait until the new instance is launched successfully.
5. After the patch is applied, RES will invoke the EFS DescribeMountTarget API with a delay and retry mechanism. Please note that with this delay and retry mechanism, if you have a higher number of EFS file systems within your account/region it may take several seconds or possibly up to a minute for the drop down to load when attempting to onboard a file system.