Repository navigation
feat(metrics): Implement and capture the metadata cache read count metric. - #5093
thrivikram-karur-g merged 25 commits into
Conversation
…vikram-metadata-read-cache-simple-implementation
…vikram-metadata-read-cache-simple-implementation
There was a problem hiding this comment.
Code Review
This pull request introduces detailed metadata cache metrics tracking (such as cache hits, misses, TTL expirations, and positive/negative entry statuses) across gcsfuse. It updates the stat cache, directory inodes, and file inodes to record these metrics, and adds comprehensive end-to-end tests. A critical correctness issue was identified in internal/fs/inode/dir.go where using || instead of && in the negative entry check could lead to false ENOENT errors and data invisibility.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5093 +/- ##
===========================================
+ Coverage 0 85.00% +85.00%
===========================================
Files 0 179 +179
Lines 0 19702 +19702
===========================================
+ Hits 0 16748 +16748
- Misses 0 2953 +2953
- Partials 0 1 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…vikram-metadata-read-cache-simple-implementation
|
Passed Integration tests. |
|
Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
…vikram-metadata-read-cache-simple-implementation
|
Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
6 similar comments
|
Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
|
Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
|
Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
|
Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
|
Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
|
Hi @vadlakondaswetha, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
…vikram-metadata-read-cache-simple-implementation
|
updated perf test results on the PR |
|
Re-triggered kokoro build to run the integration_tests |
|
Hi @vadlakondaswetha, @kislaykishore, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
…vikram-metadata-read-cache-simple-implementation
Description
Implementing the metadata_cache/read_count metric to capture the hits and misses with detailed reason for miss or hit as a counter metric to understand the metadata_cache performance.
metadata_cache/read_countwas already defined inmetrics/metrics.yamlbut nothing everrecorded it. This PR wires up emission and export.
Summary of the code changes:
Each FUSE op that reads the stat cache now records exactly one
metadata_cache/read_countevent, however many reads, retries or concurrent lookups it makes.metadata.CacheReads(new,cache_reads.go) travels in the op's context.invokeWrappedinwrappers/monitoring.gocreates it when an op starts and emits one event when the op ends.fastStatBucket.lookUp/lookUpFolderrecord every read's outcome. They use the newStatCache.LookUpDetail/LookUpFolderDetail(stat_cache.go), which report positive/negative and found/ttl_expired/not_found.StatObject/GetFolderfallbacks (fast_stat_bucket.go) and the implicit-dirListObjectsinfindDirInode(dir.go).cache_hit=falseif the op called GCS for metadata,trueotherwise.entry_status=positiveif any read found a positive entry, elsenegative;lookup_detail=found.lookup_detail=ttl_expiredif any read found an expired entry (positive takes priority over negative), elsenot_found.getInterruptlessContext(fs.go) switches tocontext.Background()when ignore-interrupts is on (the default); it now carries the collector across, otherwise nothing would be recorded.otelexporters.goallowlists themetadata_cache/prefix.metadata_cache_metrics_test.go,metadata_cache_read_count_test.go). Inmetrics_test_utils.go,require.Failbecamerequire.Failf, because the old calls panicked instead of failing when a data point was missing.Compared with the previous revision: the event is now emitted per op instead of from
LookUpChild. Every op that reads the stat cache is counted (e.g. GetInodeAttributes, CreateFile), not just lookups, anddir.gono longer needs a recording call in each branch.Benchmark results for the changes using testing.B to understand cpu cycles, allocations and time consumed to access impact of changes to Fuse Ops: LINK
Conclusion is changes are performant enough to proceed ahead.
Function benchmark results summary:
The implementation is efficient and safe to merge.
Perf test results on the PR:
we do not see any regression and all bandwidth fluctuations seems normal and within healthy range.
Link to the issue in case of a bug fix.
b/512002923
Testing details
Executed the below manual test plan for checking the metrics count updates for the metadata_cache/read_count metric.
Ran GCSFuse with the below command.
Start GCSFuse with Prometheus metric reporting enabled on port
8080:Query the exposed Prometheus metrics endpoint to inspect the metadata_cache/read_count counter and its attributes:
bash
curl -s http://localhost:8080/metrics | grep "metadata_cache/read_count"
Verified manually the below test cases for corresponding change in metric counts queried from the prometheus end point.
stat <MOUNT_DIR>/nonexistent_file.txtmetadata_cache_read_count_total{cache_hit="false",lookup_detail="not_found"} +1stat <MOUNT_DIR>/nonexistent_dir/metadata_cache_read_count_total{cache_hit="false",lookup_detail="not_found"} +1stat <MOUNT_DIR>/existing_file.txt2.
stat <MOUNT_DIR>/existing_file.txt{cache_hit="false",lookup_detail="not_found"} +12nd stat:
{cache_hit="true",entry_status="positive",lookup_detail="found"} +1stat <MOUNT_DIR>/existing_dir/2.
stat <MOUNT_DIR>/existing_dir/{cache_hit="false",lookup_detail="not_found"} +12nd stat:
{cache_hit="true",entry_status="positive",lookup_detail="found"} +1stat <MOUNT_DIR>/nonexistent_file.txt2.
stat <MOUNT_DIR>/nonexistent_file.txt{cache_hit="false",lookup_detail="not_found"} +12nd stat:
{cache_hit="true",entry_status="negative",lookup_detail="found"} +1stat <MOUNT_DIR>/nonexistent_dir/2.
stat <MOUNT_DIR>/nonexistent_dir/{cache_hit="false",lookup_detail="not_found"} +12nd stat:
{cache_hit="true",entry_status="negative",lookup_detail="found"} +1stat <MOUNT_DIR>/existing_file.txt2.
sleep 125 && stat <MOUNT_DIR>/existing_file.txt{cache_hit="false",lookup_detail="not_found"} +12nd stat:
{cache_hit="false",entry_status="positive",lookup_detail="ttl_expired"} +1stat <MOUNT_DIR>/existing_dir/2.
sleep 125 && stat <MOUNT_DIR>/existing_dir/{cache_hit="false",lookup_detail="not_found"} +12nd stat:
{cache_hit="false",entry_status="positive",lookup_detail="ttl_expired"} +1stat <MOUNT_DIR>/nonexistent_file.txt2.
sleep 125 && stat <MOUNT_DIR>/nonexistent_file.txt{cache_hit="false",lookup_detail="not_found"} +12nd stat:
{cache_hit="false",entry_status="negative",lookup_detail="ttl_expired"} +1ls --color=never <MOUNT_DIR>/metadata_cache_read_count(ReadDir does not issue per-entry stat calls)Any backward incompatible change? If so, please explain.
N/A