Cannot run an OCI image in Sarus

[haampie]

$ sarus pull stabbles/sarus-breaks
[153440.815513349] [nid003192-256744] [main] [ERROR] Error trace (most nested error last):
#0   parseJSON at "Utility.cpp":1027 Error parsing JSON string:
'{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"digest":"sha256:342a65410489fb8ec0e87f1f9af0b7da91ba922f689345b0007380e19cc2703f",
"size":18007253
}
],
"annotations":{
"org.opencontainers.image.description":"mvapich@=3.0b%gcc@=12.1.0~alloca~cuda~debug+regcache+wrapperrpath build_system=autotools ch3_rank_bits=32 file_systems=auto netmod=ofi pmi_version=simple process_managers=auto threads=multiple arch=linux-ubuntu22.04-x86_64_v2"
}
}'
Input data is not valid JSON
Error(offset 163): The document root must not be followed by other values.

This image runs fine elsewhere.

[haampie]

The manifest is valid json

{
"mediaType":"application/vnd.oci.image.manifest.v1+json",
"schemaVersion":2,
"config":{
"mediaType":"application/vnd.oci.image.config.v1+json",
"digest":"sha256:83458c6e7916894b0563f961363b192d8fb80b44c44d1a1659b0287fd024a95e",
"size":17518
},
"layers":[
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"size":29533050,
"digest":"sha256:6b851dcae6ca1461dde247915abc5048061f34332929ca8fb37d9dc18f2e2f44"
},
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"digest":"sha256:8865abe986b561226615e9907c07c4c6de1808a4f9cededc17d8a32c77c08ccc",
"size":1056788
},
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"digest":"sha256:b7fe3dfae7cf12915a1976d124772059bd5ad99fb283c3186ff2f59138bc5aad",
"size":223312
},
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"digest":"sha256:bc87881b92ddff6193c8178c0b42abe43851bf3035d276cd5106506dff1f7e5d",
"size":2119229
},
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"digest":"sha256:fa330cbdcfa81c99b77bc1627f27e1c0ccbe824036aadc99159834a5c8427ee5",
"size":1722529
},
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"digest":"sha256:d37eb909a5a6592edf820d56bc693495bb0dda0360bb888d05bc3a90bd1039df",
"size":154244
},
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"digest":"sha256:172834eda86ed531132a7e0a8ea69a9930b865001f7909477e0ecae53954de69",
"size":2864548
},
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"digest":"sha256:342a65410489fb8ec0e87f1f9af0b7da91ba922f689345b0007380e19cc2703f",
"size":18007253
}
],
"annotations":{
"org.opencontainers.image.description":"mvapich@=3.0b%gcc@=12.1.0~alloca~cuda~debug+regcache+wrapperrpath build_system=autotools ch3_rank_bits=32 file_systems=auto netmod=ofi pmi_version=simple process_managers=auto threads=multiple arch=linux-ubuntu22.04-x86_64_v2"
}
}

[haampie]

Probably it's

https://github.com/eth-cscs/sarus/blob/9c01d76736940feb360175c515e5778e408e631e/src/image_manager/SkopeoDriver.cpp#L194

The "raw" manifest json is not indented...

[haampie]

Yeah, looks like it's an issue processing skopeo output, cause after I've minified the json fully (removing newlines) it runs fine :confused:

[Madeeks]

Hi @haampie, which Sarus version are you using? I was able to pull the image successfully with Sarus 1.6.0, 1.5.1 and 1.5.0. Version 1.5.1 in particular introduced fixes to make the parsing of Skopeo output more robust, but I have not been able to reproduce the error even with 1.5.0.

[haampie]

I ran this on CSCS's Hohgant, I guess that was 1.6.0. Will check, I re-uploaded it from ghcr to dockerhub, maybe dockerhub minifies/modifies the manifest on upload.

[haampie]

Yeah, docker push or DockerHub reformats the file, not minifies, but prettifies, cause who cares about reducing metadata download overhead 😆

[haampie]

Uploaded another example:

$ ssh hohgant sarus --version
1.6.0

$ ssh hohgant sarus pull ghcr.io/haampie/sarus-test:zlib-1.2.13-cojigpatbtimllckdvzklgzekk6kqfwy
# image            : ghcr.io/haampie/sarus-test:zlib-1.2.13-cojigpatbtimllckdvzklgzekk6kqfwy
# cache directory  : "/scratch/e1000/hstoppel/.sarus/cache"
# temp directory   : "/tmp"
# images directory : "/scratch/e1000/hstoppel/.sarus/images"
[770142.55442019] [nid003192-119296] [main] [ERROR] Error trace (most nested error last):
#0   parseJSON at "Utility.cpp":1027 Error parsing JSON string:
'{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"digest":"sha256:527bca282263ba1baafa344a90d2b07664ec9e241503f678b80182e3cda239e7",
"size":153466
}
],
"annotations":{
"org.opencontainers.image.description":"zlib@=1.2.13%gcc@=12.2.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu22.10-zen2"
}
}'
Input data is not valid JSON
Error(offset 161): The document root must not be followed by other values.

The manifest file looks like this:

$ export TOKEN="$(curl -LfsS "https://ghcr.io/token?service=ghcr.io&scope=repository:haampie/sarus-test:pull" | jq -r .token)"
$ curl -LfsS https://ghcr.io/v2/haampie/sarus-test/manifests/zlib-1.2.13-cojigpatbtimllckdvzklgzekk6kqfwy -H "Authorization: Bearer $TOKEN" -H "Accept: application/vnd.oci.image.manifest.v1+json"

output

{
"mediaType":"application/vnd.oci.image.manifest.v1+json",
"schemaVersion":2,
"config":{
"mediaType":"application/vnd.oci.image.config.v1+json",
"digest":"sha256:89210d0e9eb2eb59718480e05952c5e6e2f281e647a455b56d18e3a3add39006",
"size":1663
},
"layers":[
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"size":29533422,
"digest":"sha256:3153aa388d026c26a2235e1ed0163e350e451f41a8a313e1804d7e1afb857ab4"
},
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"digest":"sha256:527bca282263ba1baafa344a90d2b07664ec9e241503f678b80182e3cda239e7",
"size":153466
}
],
"annotations":{
"org.opencontainers.image.description":"zlib@=1.2.13%gcc@=12.2.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu22.10-zen2"
}
}

[Madeeks]

Hi @haampie, I finally was able to look better into this, I can confirm it is an issue, and I have a fix in the works. I would like to ask you if I could re-push the image you have used in this thread (ghcr.io/haampie/sarus-test:zlib-1.2.13-cojigpatbtimllckdvzklgzekk6kqfwy) to the container registry we use for Sarus' CI, so we can test for support of multi-line, non-indented manifests in the integration tests.

[haampie]

Thanks, that's fine with me :)