stargz-snapshotter
stargz-snapshotter copied to clipboard
containerd
Snapshotter runs out of memory when resolving a large layer with prefetch enabled
Hello,
I've recently run into an issue where the snapshotter runs out of memory when resolving a large layer. The image in question is the pgbench 14 image. Unfortunately, I'm unable to provide the exact image as it is in a private repository.
/etc/containerd-stargz-grpc/config.toml
noprefetch = false
max_concurrency = 10
Available System Memory: ~16GB
The content of the stargz image is:
{
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:7986847b2cf6aa99385f53ac46e6ce24a7d105b4700ceeaec822d5d1e4a33beb",
"size": 10967
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:17e5262e6adea25ecd965bfadab9e4735db1b9161f6f0fdedaba95dba6efdb09",
"size": 29205376,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:61da3fc30f957a30b0924c67519e4dafe43c03271f9af747dd522d09d6178bc4"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:f8de078082becc7c11422956ab091ce2bebc45e3af8194609f89bcdc56bf49e3",
"size": 4289511,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:2b2ff5fc711663e115defe39b8a63414ea736f8c31f9b7ecb6f3a3597063819a"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:15269d890b80b9750fc80a11351ee383986e41131c9cb9789f085642474d4e69",
"size": 4587,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:721b0fa2a89238e4c85776cbf8b81335b5cfaa2e16b23645c912a7224e4479c1"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:ce293a065f68d58a8f89d8290b6353c649ec252b841be6798ede920dda17916b",
"size": 1359793,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:187f8b203f65405d6cbefc957943c4bc3b7711c41ae742d7e805198c09b76a5e"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:71e9c401f1c33156dd006a8d22ec75151b547af8115c2a185cd9b3bd98685c53",
"size": 8291211,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:b8ef402b13b7f305891f935ead8bbfeca2fedb25f2e3619c593ad6da8bd98e18"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:bda9a0ad26a39ee7d0e40070c7d323599a6432d9723a4bd5b780c2117943d744",
"size": 303693,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:89997b8d71ddc7d6aca396698d2387b351e1eb632fd111e8c4bf5e35cafeca2b"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:a529b1a9e4f4f489559154b573032b3664e634ec22b8fb365bfdd0c44ed32611",
"size": 608,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:e6902c5f6a98615674fa19380a3c5f80d6ed152bdd0beb44e71633029ddae6da"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:1878d783adddfcfee7cb9d3e8a3658223b35f192e8943086301a05f4fc132097",
"size": 3673,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:7a06728d288635b43365489d1dff679105028042d9189eb90c07b8c37aa9ee68"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:cc14c639217217b4eeb780a814097e351f98bc62ba4502c39916ef7bd21db509",
"size": 94964217,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:95f9e517724962c5dfeb84e429ffc47a1a017f4ab465af590b7c7e8a68b3c6bf"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:1a67a26b3dd177e51ed96eea8e9f5e49da81ad180a93b7b396930aed05df6271",
"size": 17968,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:414580f03d043ef486de8dab5a336016b31c7cef9e8d650413eed0a01bff216a"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:99ca52bd3e74d60aa212a33474f9b2e83de408542e635362e4f83c1d4749759a",
"size": 699,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:166e6697dcc9b97be9cb9cf04003fa3f980e53690b90906de70adb1c7597f6b9"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:75f05b17f7429b7c6f7668faeb8b8e1393e0e5fb110cee54e6a31f0868451554",
"size": 920,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:9c48dc0c6e4b523f82eb463b2e90821056d3e3d3f03bc87ce351543c638f7dc7"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:2b60598ac3476ce9240a7acc753a12efa2d053e80f03bdaea9a88ee8b785480e",
"size": 4437,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:73ca3c282d4811f60504b834398e3288260ca33b3b5c44a1ac12e3633cb27a1f"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:f403d2e1f998ff05e6ec992f1025c7adff229240201099cef49fd7014ae72a2f",
"size": 633,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:ab181951ded947e6d5f5a41d362676b2f17b6ecbb8e677413b500eb9cf4541ce"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:09c35e8efcc4ec49fad18bf1cdf237fad3a683de32e612094b4d896ea3bf98b1",
"size": 873803975,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:d77826b8a4faa831ee407ad228aee000913d30c7aacb89e6dbcb8bd8087cdbc8"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:f7b84947ee25f10c9cc5329f5dfe379956eb18fb77e0ffeefe14cc0c78ae26b2",
"size": 908,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:e0e6a43dfde918f1791ad896f8eafa868ea0a3d902fcc0a994c81271cabe93eb"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:33a7916abd4f28d17331c8101b37865c8ba1f2aa68475081f428a8d66a7fbb92",
"size": 923,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:b199705c89dff961241797310c65cae490512c810a0ee6f5e58f7eeab5a1377e"
}
}
]
}
The issue only arises when prefetch is enabled (noprefetch = false).
The culprit seems to be the 833M layer (digest: sha256:09c35e8efcc4ec49fad18bf1cdf237fad3a683de32e612094b4d896ea3bf98b1).
The working theory is that this is caused by buffers backing the memory cached chunks being put inside a buffer pool after the cache is written to the filesystem
Ref: https://github.com/containerd/stargz-snapshotter/blob/481bc84894a3bffe04f2e6e9f157dd62e81b3d9a/fs/layer/layer.go#L191) .
We found that a large number of 4M chunks were being created (~2900) that would be placed inside the buffer pool after being written out to the filesystem.
Our theory seems to be validated when we pass in cache.Direct() to the Cache function in:
https://github.com/containerd/stargz-snapshotter/blob/a6e2491c17afe1e2835d22e9ed3d8a238816ab5b/fs/layer/layer.go#L437
Using cache.Direct(), memory usage doesn't spike (remains consistent) and cumulative memory usage is reduced by half.
Graph showing memory usage with and without memcaching (note: w/ memcaching is only shown until the snapshotter crashes):
I would like to ask the containerd/snapshotter community, what the ideal way to go around fixing this would be? Thanks.
Thank you for reporting this. This seems a bug. We might need to limit the size of lrucache and buf pool.
As a quick workaround, let's add a configuration (of config.toml) to forcefully enable Direct mode for the cache. Then I'll work on improving the memory consumption of the cache (maybe we need something like size-aware sync.Pool implementation).
@rdpsin Please try v0.7.0 https://github.com/containerd/stargz-snapshotter/releases/tag/v0.7.0 You can enable direct mode of the cache to avoid this issue.
[directory_cache]
direct = true
We'll work on the improvement of the memory consumption of the cache.
FYI: Specifying metadata_store = "db" in config.toml enables storing filesystem metadata on disk. (since v0.10.0)
This can reduce the memory consumption of containerd-stargz-grpc (https://github.com/containerd/stargz-snapshotter/pull/415#issuecomment-948366623).