Repository navigation
"umoci gc" is slow after commit 4faf753956f69243f72eab667cc5ee0fd74c1d56 #394
Description
4faf753
Noticed that after the above commit, gc is now noticeably slower.
Also, the log messages are possibly misleading?
Copied a vanilla docker busybox:latest to a local filesystem as a oci layout.
Here is the manifest:
{
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:7fd4b8503a536fcfa6eba277028f558032f6d58ac0c77f3882a3f6689d0cbfa9",
"size": 575
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:874f6c6ce6172f05f10703e849cfc880120bd370f64ef27aea0803090d001d4e",
"size": 779781
}
]
}
and yet, the entire (valid) layer is being read/verified and "discarded". For images with larger and many layers, gc is noticeably slower.
$ umoci --verbose ls --layout oci
busybox:latest
$ umoci stat --image oci:busybox:latest
LAYER CREATED CREATED BY SIZE COMMENT
sha256:874f6c6ce6172f05f10703e849cfc880120bd370f64ef27aea0803090d001d4e 2021-05-04T00:19:49.383853296Z /bin/sh -c #(nop) ADD file:dc794c2febce9ec5b68ca4f55027eb4e7d42fc7941e15ba578585d8a166d2d13 in / 779.8kB
<none> 2021-05-04T00:19:49.531663542Z /bin/sh -c #(nop) CMD ["sh"] <none>
$ time umoci --verbose gc --layout oci
• verified reader: 779781 bytes of trailing data discarded from vrdr[b/blobs/sha256/874f6c6ce6172f05f10703e849cfc880120bd370f64ef27aea0803090d001d4e]
real 0m0.015s
user 0m0.012s
sys 0m0.005s
Activity
cyphar commentedon Sep 23, 2021
This appears to be a duplicate of #373 and should've been fixed by #375. Have you tried with the latest HEAD? I'm planning on doing an new umoci release soon.
The "X bytes of trailing data discarded" is not misleading, though maybe it could be worded slightly better -- it means that when calling
Closethat many bytes remained in the buffer and were not read -- in the context of extracting an actual layer those bytes would be discarded bytes because they weren't used in the layer extraction. When writing the patch I forgot that we open blobs but don't read from them in GC, meaning all the data is "discarded" (but still verified by VerifiedReadCloser). In any case this should've been fixed already.cyphar commentedon Oct 21, 2021
Closing since I'm fairly sure this was fixed, and reporter is not responding.
rchincha commentedon Oct 21, 2021
Yes, this looks fixed on latest HEAD.