Skip to content

feat(metrics): Implement and capture the metadata cache read count metric. - #5093

Merged
thrivikram-karur-g merged 25 commits into
masterfrom
vikram-metadata-read-cache-simple-implementation
Oct 13, 2026
Merged

thrivikram-karur-g merged 25 commits into
masterfrom
vikram-metadata-read-cache-simple-implementation

Conversation

@thrivikram-karur-g

@thrivikram-karur-g thrivikram-karur-g commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Description

Implementing the metadata_cache/read_count metric to capture the hits and misses with detailed reason for miss or hit as a counter metric to understand the metadata_cache performance.
metadata_cache/read_count was already defined in metrics/metrics.yaml but nothing ever
recorded it. This PR wires up emission and export.

Summary of the code changes:

Each FUSE op that reads the stat cache now records exactly one metadata_cache/read_count event, however many reads, retries or concurrent lookups it makes.

  • Per-op collector: metadata.CacheReads (new, cache_reads.go) travels in the op's context. invokeWrapped in wrappers/monitoring.go creates it when an op starts and emits one event when the op ends.
  • Reads are recorded in the stat-cache layer: fastStatBucket.lookUp / lookUpFolder record every read's outcome. They use the new StatCache.LookUpDetail / LookUpFolderDetail (stat_cache.go), which report positive/negative and found/ttl_expired/not_found.
  • GCS calls are recorded where they happen: the StatObject / GetFolder fallbacks (fast_stat_bucket.go) and the implicit-dir ListObjects in findDirInode (dir.go).
  • How attributes are decided (once per op):
    • cache_hit=false if the op called GCS for metadata, true otherwise.
    • Hit: entry_status=positive if any read found a positive entry, else negative; lookup_detail=found.
    • Miss: lookup_detail=ttl_expired if any read found an expired entry (positive takes priority over negative), else not_found.
    • Ops that never read the stat cache (e.g. ReadDir, or the cache disabled) record nothing.
  • Context propagation: getInterruptlessContext (fs.go) switches to context.Background() when ignore-interrupts is on (the default); it now carries the collector across, otherwise nothing would be recorded.
  • Export: otelexporters.go allowlists the metadata_cache/ prefix.
  • Tests: unit tests for the collector, the stat-cache detail lookups and the caching bucket, plus end-to-end FUSE scenarios (metadata_cache_metrics_test.go, metadata_cache_read_count_test.go). In metrics_test_utils.go, require.Fail became require.Failf, because the old calls panicked instead of failing when a data point was missing.
  • Overhead: no extra allocation per stat-cache read; about 10–17 ns added to a cache hit (benchmarked); one 24 B collector per FUSE op, plus a 48 B context when ignore-interrupts is on (the default).

Compared with the previous revision: the event is now emitted per op instead of from LookUpChild. Every op that reads the stat cache is counted (e.g. GetInodeAttributes, CreateFile), not just lookups, and dir.go no longer needs a recording call in each branch.

Benchmark results for the changes using testing.B to understand cpu cycles, allocations and time consumed to access impact of changes to Fuse Ops: LINK

Conclusion is changes are performant enough to proceed ahead.

Function benchmark results summary:

The implementation is efficient and safe to merge.


Question Answer
Cost per FUSE op ~30 ns and 24 B (1 alloc) for every op. The 16 ops that detach their context (ignore-interrupts is on by default) add ~50–90 ns and 48 B (1 alloc).
Cost per stat-cache read ~7 ns more time holding the bucket mutex; no allocations
Warm LookUpInode, end to end +183 ns on ~1.8 µs (+10%), in-process, without the kernel round trip
Extra GCS calls or cache reads None. Counted per op, identical to master
Locks, goroutines, global state None added; recording uses only atomics
Race detector Clean (see Safety)

Benchmark What it measures
BenchmarkStatCache_LookUpOutcomes LookUpDetail vs LookUp, for a positive hit, a negative hit and an absent key
BenchmarkStatCache_ReadPath A stat-cache read made the way fastStatBucket.lookUp makes it: mutex, LookUpDetail, RecordCacheRead. Run serially and with 8 goroutines
BenchmarkCacheReads_RecordCacheRead Recording one read, with the collector 0, 4 or 16 contexts up, or with no collector
BenchmarkCacheReads_PerOp What one op pays: the collector, its reads, and Emit into the real OTel handle
BenchmarkCacheReads_WithCacheReadsFrom Carrying the collector into the detached context of getInterruptlessContext

Perf test results on the PR:

Screenshot 2026-10-09 at 1 14 47 PM

we do not see any regression and all bandwidth fluctuations seems normal and within healthy range.

Link to the issue in case of a bug fix.

b/512002923

Testing details

  1. Manual

Executed the below manual test plan for checking the metrics count updates for the metadata_cache/read_count metric.

Ran GCSFuse with the below command.

Start GCSFuse with Prometheus metric reporting enabled on port 8080:

go run . --foreground \
  --enable-buffered-read \
  --implicit-dirs \
  --client-protocol http1 \
  --prometheus-port=8080 \
  --enable-metadata-prefetch=false \
  --metadata-cache-negative-ttl-secs=120 \
  --metadata-cache-ttl-secs=120 \
  --log-severity=trace \
  <BUCKET_NAME> <MOUNT_DIR>

Query the exposed Prometheus metrics endpoint to inspect the metadata_cache/read_count counter and its attributes:

bash
curl -s http://localhost:8080/metrics | grep "metadata_cache/read_count"

Verified manually the below test cases for corresponding change in metric counts queried from the prometheus end point.

Scenario Test Case Executed Command(s) Expected Metric Output Delta
1 Cold File Miss stat <MOUNT_DIR>/nonexistent_file.txt metadata_cache_read_count_total{cache_hit="false",lookup_detail="not_found"} +1
2 Cold Directory Miss stat <MOUNT_DIR>/nonexistent_dir/ metadata_cache_read_count_total{cache_hit="false",lookup_detail="not_found"} +1
3 Warm File Hit (Positive) 1. stat <MOUNT_DIR>/existing_file.txt
2. stat <MOUNT_DIR>/existing_file.txt
1st stat: {cache_hit="false",lookup_detail="not_found"} +1
2nd stat: {cache_hit="true",entry_status="positive",lookup_detail="found"} +1
4 Warm Directory Hit (Positive) 1. stat <MOUNT_DIR>/existing_dir/
2. stat <MOUNT_DIR>/existing_dir/
1st stat: {cache_hit="false",lookup_detail="not_found"} +1
2nd stat: {cache_hit="true",entry_status="positive",lookup_detail="found"} +1
5 Warm File Hit (Negative) 1. stat <MOUNT_DIR>/nonexistent_file.txt
2. stat <MOUNT_DIR>/nonexistent_file.txt
1st stat: {cache_hit="false",lookup_detail="not_found"} +1
2nd stat: {cache_hit="true",entry_status="negative",lookup_detail="found"} +1
6 Warm Directory Hit (Negative) 1. stat <MOUNT_DIR>/nonexistent_dir/
2. stat <MOUNT_DIR>/nonexistent_dir/
1st stat: {cache_hit="false",lookup_detail="not_found"} +1
2nd stat: {cache_hit="true",entry_status="negative",lookup_detail="found"} +1
7 Positive File TTL Expired 1. stat <MOUNT_DIR>/existing_file.txt
2. sleep 125 && stat <MOUNT_DIR>/existing_file.txt
1st stat: {cache_hit="false",lookup_detail="not_found"} +1
2nd stat: {cache_hit="false",entry_status="positive",lookup_detail="ttl_expired"} +1
8 Positive Directory TTL Expired 1. stat <MOUNT_DIR>/existing_dir/
2. sleep 125 && stat <MOUNT_DIR>/existing_dir/
1st stat: {cache_hit="false",lookup_detail="not_found"} +1
2nd stat: {cache_hit="false",entry_status="positive",lookup_detail="ttl_expired"} +1
9 Negative Entry TTL Expired 1. stat <MOUNT_DIR>/nonexistent_file.txt
2. sleep 125 && stat <MOUNT_DIR>/nonexistent_file.txt
1st stat: {cache_hit="false",lookup_detail="not_found"} +1
2nd stat: {cache_hit="false",entry_status="negative",lookup_detail="ttl_expired"} +1
10 Directory Listing (No Stat Inflation) ls --color=never <MOUNT_DIR>/ No metric delta for metadata_cache_read_count (ReadDir does not issue per-entry stat calls)
  1. Unit tests - Added
  2. Integration tests - NA

Any backward incompatible change? If so, please explain.

N/A

@thrivikram-karur-g
thrivikram-karur-g requested a review from a team as a code owner September 15, 2026 05:50
@github-actions github-actions Bot added the remind-reviewers Auto remind reviewers in attention set for review post 24hrs of inactivity on PR. label Sep 15, 2026
@thrivikram-karur-g
thrivikram-karur-g removed the request for review from kislaykishore September 15, 2026 05:50

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces detailed metadata cache metrics tracking (such as cache hits, misses, TTL expirations, and positive/negative entry statuses) across gcsfuse. It updates the stat cache, directory inodes, and file inodes to record these metrics, and adds comprehensive end-to-end tests. A critical correctness issue was identified in internal/fs/inode/dir.go where using || instead of && in the negative entry check could lead to false ENOENT errors and data invisibility.

Comment thread internal/fs/inode/dir.go Resolved
Comment thread metrics/helper.go Resolved
Comment thread internal/fs/inode/file.go Resolved
Comment thread internal/fs/inode/file.go Resolved
Comment thread internal/fs/inode/dir.go Resolved
@codecov

codecov Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.82716% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.00%. Comparing base (14985ea) to head (efb8aa5).
⚠️ Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
metrics/metrics_test_utils.go 0.00% 4 Missing ⚠️
internal/cache/metadata/cache_reads.go 97.14% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #5093       +/-   ##
===========================================
+ Coverage        0   85.00%   +85.00%     
===========================================
  Files           0      179      +179     
  Lines           0    19702    +19702     
===========================================
+ Hits            0    16748    +16748     
- Misses          0     2953     +2953     
- Partials        0        1        +1     
Flag Coverage Δ
unittests 85.00% <93.82%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@thrivikram-karur-g

Copy link
Copy Markdown
Contributor Author

Passed Integration tests.

@thrivikram-karur-g thrivikram-karur-g added execute-perf-test Execute performance test in PR and removed execute-integration-tests Run only integration tests labels Sep 21, 2026
@github-actions

Copy link
Copy Markdown

Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you!

@github-actions

Copy link
Copy Markdown

Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you!

6 similar comments
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you!

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown

Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you!

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you!

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you!

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown

Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you!

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown

Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you!

@thrivikram-karur-g thrivikram-karur-g added execute-perf-test Execute performance test in PR and removed execute-integration-tests Run only integration tests labels Oct 8, 2026
@thrivikram-karur-g

Copy link
Copy Markdown
Contributor Author

updated perf test results on the PR

@thrivikram-karur-g thrivikram-karur-g added execute-integration-tests Run only integration tests and removed execute-perf-test Execute performance test in PR labels Oct 9, 2026
@thrivikram-karur-g

Copy link
Copy Markdown
Contributor Author

Re-triggered kokoro build to run the integration_tests

@github-actions

Copy link
Copy Markdown

Hi @vadlakondaswetha, @kislaykishore, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you!

Comment thread internal/cache/metadata/cache_reads.go Resolved
Comment thread internal/cache/metadata/stat_cache.go Resolved
@thrivikram-karur-g
thrivikram-karur-g merged commit 2524017 into master Oct 13, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

execute-integration-tests Run only integration tests remind-reviewers Auto remind reviewers in attention set for review post 24hrs of inactivity on PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ 换网址