Skip to content

(2025.09 and earlier) Fail to mount EFS file systems on RHEL, Rocky and Ubuntu VDIs

Jump to bottom
Junbo Liang edited this page May 7, 2026 · 3 revisions

Bug description

RHEL, Rocky and Ubuntu VDIs fail to mount EFS systems during the bootstrap process.

The following errors can be found in the VDI installation logs ( /root/bootstrap/logs/install.log.{timestamp}) which complain about the missing dependencies:

warning: aws-lc-fips-sys@0.13.9: Building with: CMake
warning: aws-lc-fips-sys@0.13.9: Symbol Prefix: Some("aws_lc_fips_0_13_9")
error: failed to run custom build command for `aws-lc-fips-sys v0.13.9`

Caused by:
  process didn't exit successfully: `/root/bootstrap/latest/efs-utils/build/rpmbuild/BUILD/amazon-efs-utils/src/proxy/target/release/build/aws-lc-fips-sys-941fd8b2176d27ee/build-script-main` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=AWS_LC_FIPS_SYS_NO_PREFIX
  cargo:rerun-if-env-changed=AWS_LC_FIPS_SYS_PREGENERATING_BINDINGS
  cargo:rerun-if-env-changed=AWS_LC_FIPS_SYS_EXTERNAL_BINDGEN
  cargo:rerun-if-env-changed=AWS_LC_FIPS_SYS_NO_ASM
  cargo:rerun-if-env-changed=AWS_LC_FIPS_SYS_CPU_JITTER_ENTROPY
  cargo:rustc-cfg=x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=AWS_LC_FIPS_SYS_STATIC
  cargo:warning=Building with: CMake
  cargo:warning=Symbol Prefix: Some("aws_lc_fips_0_13_9")
  cargo:rerun-if-env-changed=CMAKE

  --- stderr
  Missing dependency: Go is required for FIPS.
  Missing dependency: cmake
  thread 'main' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-fips-sys-0.13.9/builder/main.rs:518:34:
  called `Result::unwrap()` on an `Err` value: "Required build dependency is missing. Halting build."
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: Bad exit status from /var/tmp/rpm-tmp.3KdWsc (%build)

The VDI app logs (/opt/idea/app/logs/application.log) also include the following errors which indicate that the EFS file systems failed to be mounted:

[2025-11-17 21:56:14,098] [INFO] [bootstrap] Using Provider efs for /home using options efs _netdev,noresvport,tls,iam 0 0
[2025-11-17 21:56:14,098] [INFO] [bootstrap] Using Provider efs for /internal using options efs _netdev,noresvport,tls,iam 0 0
[2025-11-17 21:56:14,114] [INFO] [bootstrap] (0 of 5) Failed to mount FS, retrying in 40 seconds ...
[2025-11-17 21:56:54,170] [INFO] [bootstrap] (1 of 5) Failed to mount FS, retrying in 24 seconds ...
[2025-11-17 21:57:18,208] [INFO] [bootstrap] (2 of 5) Failed to mount FS, retrying in 33 seconds ...
[2025-11-17 21:57:51,228] [INFO] [bootstrap] (3 of 5) Failed to mount FS, retrying in 13 seconds ...
[2025-11-17 21:58:04,247] [INFO] [bootstrap] (4 of 5) Failed to mount FS, retrying in 11 seconds ...
[2025-11-17 21:58:15,271] [INFO] [bootstrap] (5 of 5) Failed to mount FS, retrying in 17 seconds ...

This issue is caused by the latest release of efs-utils (2.4.0) on 10/31/2025 which includes FIPS related changes and require to build aws-lc-fips-sys as a new dependency.

Affected versions

2025.09 and earlier

Mitigation

RES environment with public internet access

  1. Download the patch script and upload it to your own S3 bucket with proper tags by running the following commands:
    1. The commands below require AWS CLI v2.
    2. Configure the AWS CLI for the account / region where RES is deployed, and make sure that you have S3 permissions to write to the bucket for storing the patch script and add tags to S3 objects.
    3. Replace <res_version>, <environment-name>, <output-directory> and <bucket-name> with your RES version, RES environment name, local directory for storing the patch script and your S3 bucket for storing the patch script.
RES_VERSION=<res_version>
ENVIRONMENT_NAME=<environment-name>
OUTPUT_DIRECTORY=<output-directory>
BUCKET_NAME=<bucket-name>

mkdir -p ${OUTPUT_DIRECTORY}
curl https://research-engineering-studio-us-east-1.s3.amazonaws.com/releases/${RES_VERSION}/patch_scripts/patches/install_missing_packages.sh --output ${OUTPUT_DIRECTORY}/install_missing_packages.sh
aws s3 cp ${OUTPUT_DIRECTORY}/install_missing_packages.sh s3://${BUCKET_NAME}/patches/install_missing_packages.sh
aws s3api put-object-tagging \
    --bucket ${BUCKET_NAME} \
    --key "patches/install_missing_packages.sh" \
    --tagging "TagSet=[{Key=res:EnvironmentName,Value=${ENVIRONMENT_NAME}}]"
  1. In the RES web portal, create a new RES project or update your exiting RES project as an administrator by adding s3://<bucket-name>/patches/install_missing_packages.sh as a custom launch script that execute at the start of a VDI.
    1. Replace <bucket-name> without your S3 bucket for storing the patch script.
  2. (Only for 2025.06.01 and earlier) Create an IAM policy with read access to your S3 bucket for storing the patch script, and tag it with the proper tag by running the commands below. Then add this policy to your RES project.
POLICY_NAME="vdi-host-s3-read-policy"
POLICY_DESCRIPTION="IAM policy granting read access to S3 bucket for VDI hosts"
TAG_KEY="res:Resource"
TAG_VALUE="vdi-host-policy"

cat > /tmp/s3-read-policy.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET_NAME}",
                "arn:aws:s3:::${BUCKET_NAME}/*"
            ]
        }
    ]
}
EOF

POLICY_ARN=$(aws iam create-policy \
    --policy-name "$POLICY_NAME" \
    --policy-document file:///tmp/s3-read-policy.json \
    --description "$POLICY_DESCRIPTION" \
    --query 'Policy.Arn' \
    --output text)

aws iam tag-policy \
    --policy-arn "$POLICY_ARN" \
    --tags Key="$TAG_KEY",Value="$TAG_VALUE"
  1. Launch a new VDI from the RES project to verify the fix.

RES environment without public internet access

  1. Bake new RES ready VDI AMIs following https://docs.aws.amazon.com/res/latest/ug/res-ready-ami.html and add the first two build steps (DownloadRESMissingPackagesInstallScrpt and RunRESMissingPackagesInstallScript) below to the image builder component content:
    1. Replace <res_version> with your RES version
#  Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
#  Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
#  with the License. A copy of the License is located at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES
#  OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions
#  and limitations under the License.
name: research-and-engineering-studio-vdi-linux
description: An RES EC2 Image Builder component to install required RES software dependencies for Linux VDI.
schemaVersion: 1.0
parameters:
  - GPUFamily:
      type: string
      description: GPU family (NONE, NVIDIA, or AMD)
      default: NONE
phases:
  - name: build
    steps:
      - name: DownloadRESMissingPackagesInstallScrpt
        action: S3Download
        onFailure: Abort
        maxAttempts: 3
        inputs:
          - source: "s3://research-engineering-studio-us-east-1/releases/<res_version>/patch_scripts/patches/install_missing_packages.sh"
            destination: "/root/bootstrap/res-installation-scripts/install_missing_packages.sh"
      - name: RunRESMissingPackagesInstallScript
        action: ExecuteBash
        onFailure: Abort
        maxAttempts: 3
        inputs:
          commands:
            - "cd /root/bootstrap/res-installation-scripts"
            - "/bin/bash install_missing_packages.sh"
      - name: PrepareRESBootstrap
        action: ExecuteBash
        onFailure: Abort
        maxAttempts: 3
        inputs:
          commands:
            - "mkdir -p /root/bootstrap/logs"
            - "mkdir -p /root/bootstrap/latest"
      - name: DownloadRESLinuxInstallPackage
        action: S3Download
        onFailure: Abort
        maxAttempts: 3
        inputs:
          - source: "s3://research-engineering-studio-us-east-1/releases/latest/res-installation-scripts.tar.gz"
            destination: "/root/bootstrap/res-installation-scripts/res-installation-scripts.tar.gz"
      - name: RunInstallScript
        action: ExecuteBash
        onFailure: Abort
        maxAttempts: 3
        inputs:
          commands:
            - "cd /root/bootstrap/res-installation-scripts"
            - "tar -xf res-installation-scripts.tar.gz"
            - "cd scripts/virtual-desktop-host/linux"
            - "/bin/bash install.sh -g {{ GPUFamily }}"
      - name: RunInstallPostRebootScript
        action: ExecuteBash
        onFailure: Abort
        maxAttempts: 3
        inputs:
          commands:
            - "cd /root/bootstrap/res-installation-scripts/scripts/virtual-desktop-host/linux"
            - 'sed -i ''/^export AWS_DEFAULT_PROFILE="bootstrap_profile"$/d'' install_post_reboot.sh'
            - "/bin/bash install_post_reboot.sh -g {{ GPUFamily }}"
      - name: PreventAL2023FromUninstallingCronie
        action: ExecuteBash
        onFailure: Abort
        maxAttempts: 3
        inputs:
          commands:
            - "rm -f /tmp/imagebuilder_service/crontab_installed"
  1. Launch a new VDI using the new RES ready AMI to verify the fix.

Clone this wiki locally
⌂ 换网址