Skip to content

Releases

Gcsfuse v3.8.0

Choose a tag to compare

@gcsfuse-github-machine-user-bot gcsfuse-github-machine-user-bot released this 19 Mar 04:21
bc3c09b

Asynchronous Metadata Prefetching: Accelerate Metadata-Heavy Workloads

In this release, we are introducing an intelligent directory-aware asynchronous metadata prefetch mechanism designed to significantly reduce latency for file lookups by proactively populating the metadata cache. This feature also addresses performance degradation caused by cache evictions in large buckets by automatically "self-healing" the metadata cache when entries are missing or expired.

By fetching directory metadata in batches in the background, this feature can save approximately 20ms per file access and improve overall performance by up to 48% for metadata-intensive workloads like small file reads. Additionally, it offers massive cost efficiency by reducing GCS API calls (e.g., from 10,000 individual lookups to just 2 List calls for a 5,000-file directory), potentially lowering metadata access costs by over 90%.

  • Feature Enablement: This feature is disabled by default in this release and can be enabled with the --enable-metadata-prefetch flag or metadata-cache:enable-metadata-prefetch config.
  • Use Cases: Metadata-heavy workloads such as large-scale data processing with many small files and any scenarios where high directory locality is expected. Sample use cases include vision model building with many small image files or language models that work on crawled web data or Electronic Design Automation(EDA) simulation.
  • Memory Usage: Memory consumption is bounded by the existing stat-cache-max-size-mb configuration (default: 34 MiB). Prefetch batches are limited to 5,000 entries (configurable) per directory to minimize memory/CPU pressure.
  • CPU Usage: CPU consumption is typically lower when enabled due to the significant reduction in metadata network calls. Concurrent prefetch activity is strictly throttled by a global concurrency limit (10 by default; configurable) to prevent resource exhaustion.
  • Large Directory Optimization: For directories with more than 5000 files, the system automatically switches to a Targeted Mode. Instead of fetching the entire directory, it only caches the immediate lexicographical siblings(up to 5K) of the accessed file, ensuring efficient scaling without excessive resource usage.

New Configuration Flags

The following flags can be used to configure this feature within the metadata-cache section of your configuration or as command-line arguments:

  • enable-metadata-prefetch (bool)
    Enables background prefetching of metadata for objects within a directory during a cache miss. This feature performs a batch update to the metadata cache, significantly reducing latency for subsequent lookups of files in the directory.
    Default: False
  • metadata-prefetch-entries-limit (Int)
    The maximum number of metadata entries (files and directories) to prefetch into the cache upon a prefetch trigger. Note that values higher than 5000 will trigger multiple sequential GCS List calls per directory, as a single call is capped at 5000 results.
    Default: 5000
  • Metadata-prefetch-max-workers (Int)
    The maximum number of concurrent background workers allowed to perform metadata prefetching across all directories. Each directory is limited to one active worker at a time.
    Default: 10

Type cache deprecation

Starting with version 3.8.0, type cache is deprecated. Metadata cache will take care of serving type information (i.e, where it's a file or directory) now. All type-cache related flags such as enable-nonexistent-type-cache and type-cache-max-size-mb, are now no-op.

Reduced GCS Egress for Random Reads

Optimized initial read handling to reduce the initial egress. Previously, all initial requests were treated as sequential, triggering 200MB read requests. Now, initial requests with non-zero offsets are treated as random, utilizing a 1MB read requests.

Bug Fixes and Improvements

  • A4X-Max has been added as a high-performance machine type.
  • Histogram representations for metrics have been adjusted to focus on high latency cases by adjusting histogram boundaries. For eg, fs-op latencies more than 100ms would get adjusted in histogram to have more details than previous representation that truncates them.

Gcsfuse v3.2.5

Choose a tag to compare

Dependency Upgrades

  • Upgraded Golang from 1.24.11 to 1.26.1 to include latest security patches and performance improvements.

Gcsfuse v3.7.3

Choose a tag to compare

Dependency Upgrades

  • Upgraded Golang from 1.24.11 to 1.26.1 to include latest security patches and performance improvements.

Gcsfuse v3.7.2

Choose a tag to compare

Bug fixes & Improvements

Premature EOF on kernels with large page-sizes

Aggressive kernel read-ahead on ARM64 architectures with large page-sizes previously caused premature-EOFs on reads. This release provides a critical stability fix for GCSFuse on ARM64 architectures configured with large page sizes (e.g. Google Cloud A4X and A4X Max instances with 64 KiB memory pages).

Updated handling of appends on Rapid Buckets

We fixed append failures in Rapid storage by eliminating the unnecessary Metadata call that caused issues due to stale server response for same generation objects.

Gcsfuse v3.7.1

Choose a tag to compare

Features:

  • Rapid Buckets Read Performance Improvement: Introduced a new read path for Rapid buckets that optimizes performance by leveraging the read ahead algorithm implemented by kernel which is optimised over the years.

    • Configuration Requirements: To use the new reader, we require a few configuration changes on the machines. These configuration changes are fully managed in the GKE environment. On non-GKE environments, GCSFuse requires sudo permissions or users must manually configure read_ahead_kb, max_background, and congestion_threshold. Default config values:
      • read_ahead_kb: 16384
      • max_background: min(max(12, 2*numCPU), 192)
      • congestion_threshold: (3 * max_background)/4
      • Manual configuration of these settings is shown below using a c4-standard-192 instance:
    export MOUNT_POINT=<mount path>
    
    echo 16384 | sudo tee /sys/class/bdi/0:$(stat -c "%d" $MOUNT_POINT)/read_ahead_kb
    echo 192 | sudo tee /sys/fs/fuse/connections/$(stat -c "%d" $MOUNT_POINT)/max_background
    echo 144 | sudo tee /sys/fs/fuse/connections/$(stat -c "%d" $MOUNT_POINT)/congestion_threshold
    
    • Limitations :
      1. File Cache & Buffered Reader compatibility: This read path is incompatible with File Cache & Buffered Reader. To use these features, users must explicitly set enable-kernel-reader to false during mounting; otherwise, these features will remain inactive.
      2. Dynamic Mounts: The new read path is not supported when using dynamic mounting.

Bug fixes & Improvements:

  • Improved the retry logic in new read path for Rapid Buckets

  • #3974

    Improved handling of errors encountered while deleting objects in non-atomic rename flow(Applicable on non-HNS buckets). GCS Fuse handles the metadata eviction and file cache updation automatically corresponding to deleted objects and there is no action required from the customer.

Dependency Upgrades / CVE fixes

  • Go SDK upgrade from v1.59.1 to 1.59.2

Full Changelog: v3.6.0...v3.7.1

Gcsfuse v3.7.0

Gcsfuse v3.7.0 Pre-release
Pre-release

Choose a tag to compare

@gcsfuse-github-machine-user-bot gcsfuse-github-machine-user-bot released this 03 Feb 07:17
51c09fd

Features:

  • Rapid Buckets Read Performance Improvement: Introduced a new read path for Rapid buckets that optimizes performance by leveraging the read ahead algorithm implemented by kernel which is optimised over the years.

    • Configuration Requirements: To use the new reader, we require a few configuration changes on the machines. These configuration changes are fully managed in the GKE environment. On non-GKE environments, GCSFuse requires sudo permissions or users must manually configure read_ahead_kb, max_background, and congestion_threshold. Default config values:
      • read_ahead_kb: 16384
      • max_background: min(max(12, 2*numCPU), 192)
      • congestion_threshold: (3 * max_background)/4
      • Manual configuration of these settings is shown below using a c4-standard-192 instance:
    export MOUNT_POINT=<mount path>
    
    echo 16384 | sudo tee /sys/class/bdi/0:$(stat -c "%d" $MOUNT_POINT)/read_ahead_kb
    echo 192 | sudo tee /sys/fs/fuse/connections/$(stat -c "%d" $MOUNT_POINT)/max_background
    echo 144 | sudo tee /sys/fs/fuse/connections/$(stat -c "%d" $MOUNT_POINT)/congestion_threshold
    
    • Limitations :
      1. File Cache & Buffered Reader compatibility: This read path is incompatible with File Cache & Buffered Reader. To use these features, users must explicitly set enable-kernel-reader to false during mounting; otherwise, these features will remain inactive.
      2. Dynamic Mounts: The new read path is not supported when using dynamic mounting.

Bug fixes & Improvements:

  • #3974

    Improved handling of errors encountered while deleting objects in non-atomic rename flow(Applicable on non-HNS buckets). GCS Fuse handles the metadata eviction and file cache updation automatically corresponding to deleted objects and there is no action required from the customer.

Dependency Upgrades / CVE fixes

  • Go SDK upgrade from v1.59.1 to 1.59.2

Gcsfuse v3.6.0

Choose a tag to compare

  • Regex-based File Cache Filtering

    • GCSFuse now provides granular control over the set of files cached when file cache mode is enabled through regular expressions, enabling users to prioritize critical datasets. For e.g., you can now specify only parquet files to be cached over the entire dataset. This feature is helpful in AI training where you want to expose raw data via fuse but want to control specific datasets to be cached like parquet files or Web Datasets.
    • This feature introduces two optional configuration parameters: file-cache:include-regex and file-cache:exclude-regex. These are accessible via the command-line flags --file-cache-include-regex and --file-cache-exclude-regex, respectively.
    • Regex patterns are evaluated against the complete object path. Each of these options can be used independently. When configured, files matching the exclude-regex are ineligible for caching. Similarly, when configured, only files matching the include-regex will be cached.
  • Improved handling of paths with special characters

    • GCSFuse gracefully handles objects with names like //, . or .., ensuring the file operations do not throw errors.
      • Listing: Objects with special characters like //, . or .. that conflict with traditional file system hierarchy are hidden from file listings.
      • Renaming or deleting a directory affects all contained objects, including objects with special characters. This prevents these special objects from being accidentally left behind.
  • Sparse file download behind an experimental flag

    • Added support for downloading requested chunks instead of complete files when filecache is enabled. This is helpful for scenarios where entire file can’t be fit in the cache due to its size or to reduce egress from bucket where you only need a part of the file but not the entire file.Currently this feature is in experimental mode and behind a flag --experimental-enable-chunk-cache or file-cache:experimental-enable-chunk-cache config.

Bug Fixes & Improvements

  • Improves HNS folder listing by ~28x. In our benchmarking, this improvement reduced the time to list a folder containing 1000 folders from 1.7 seconds to 60ms. (PR#4099) (Issue#4075)

  • Buffered Read

    • Zero Copy - 6x throughput improvement over default reader
      • Removed a memory copy from the GCSFuse layer for the buffered reader, resulting in over 20% throughput improvement compared to the previous buffered reader implementation and a 6x throughput increase compared to the default reader. This optimization also reduces the CPU overhead previously required for data copy.
    • Support for Sequential-after-Random Workloads
      • GCSFuse now automatically switches back to the buffered reader if the workload's read pattern becomes sequential after a period of random access
  • Redundant ListObject check on mount initialization is removed.

Dependency Upgrades / CVE fixes

  • Go SDK patch release upgrade to v1.58.0

Full Changelog: v3.5.0...v3.6.0

Gcsfuse v3.5.6

Choose a tag to compare

@alleaditya alleaditya released this 13 Jan 06:33
  • Improved mount time by removing redundant ListObject call during the mount process. Bucket existence and access checks will be handled by the existing getStorageLayout call.

Gcsfuse v3.2.4

Choose a tag to compare

  • Improved mount time by removing redundant ListObject call during the mount process. Bucket existence and access checks will be handled by the existing getStorageLayout call.

Gcsfuse v3.5.5

Choose a tag to compare

Bug Fixes

  • Upgraded the go-dns library to v1.3.0 which contains a fix for DNS lookups for machines where send is restricted but sendto is not (#4151) .

  • Removed the reader attribute from the gcs/read_bytes_count metric to bring the metric's cardinality back within recommended limits, preventing potential issues with metric collection and querying (#4142).

  • Fixed a regression introduced in v3.5.1 where the enable-google-lib-auth flag defaulted to true, causing unnecessary cred.UniverseDomain queries even when --anonymous-access was enabled. This causes the authentication to fail. This behavior is now corrected in v3.5.5 to properly bypass all authentication checks..(#4093).

⌂ 换网址