containers
containers copied to clipboard
ReproNim
script to mirror all versions of containers from docker hub
e.g. immediate usecase is "backup" of all neurodebian containers from https://hub.docker.com//neurodebian . I only wonder if it should all be dumped here or should be a separate dataset (probably separate).
@kyleam - do you some easy wrapper around docker to given a docker hub "repository" (such as https://hub.docker.com//neurodebian) to mint all needed calls to datalad containers-add?
@kyleam - do you some easy wrapper around docker to given a docker hub "repository" (such as https://hub.docker.com/_/neurodebian) to mint all needed calls to
datalad containers-add?
Hmm, does this come down to wanting a containers-add call for each tag in the repository? If so, perhaps something like this:
import requests
endpoint = "https://hub.docker.com/v2"
repo = "neurodebian"
next_page = f"{endpoint}/repositories/library/{repo}/tags"
while next_page:
print(next_page)
response = requests.get(next_page)
response.raise_for_status()
data = response.json()
next_page = data.get("next")
for result in data.get("results"):
print(result["name"])
# containers-add call if container isn't already present
I've pushed a rough prototype, which may or may not be viable start, to kyleam/datalad-container-dhub-tags (obviously not its final home if it ends up being useful). python containers_add_dhub_tags.py --help will give a few more details, but basically
echo neurodebian | python containers_add_dhub_tags.py
should lead to a containers-add --url dhub:// ... call for each tag under the neurodebian repository.
echo repronim/ | python containers_add_dhub_tags.py
should get a containers-add --url dhub:// ... call for each tag in each repository under repronim.
awesome -- I will give it a shot now. I am just curious -- why stdin? ;-)
It works! Some observations I would like your input on -- I might be not seeing some possible issues
- since there is only one version for a tag at a given point in time when we run it, and otherwise information about the tag would be lost, why not to create those container names following tags, e.g. instead of
neurodebian-f5098f2for dhub://library/neurodebian:xenial-non-free it to be justneurodebian--xenial-non-free(or havelibrary-prefix for consistency) or if we want really to include the image hexsha (they aren't sortable -- humans would get confused which one to use) could beneurodebian--xenial-non-free--f5098f2(we cannot use.I believe in container names, otherwise it could have been a dot). - even though not immediately usable, it would imho still be useful then to place those images not under flat
.datalad/imagesbut to establish uniform hierarchy of directories reflecting those docker hub repositories etc., so smth like
datalad containers-add neurodebian--xenial-non-free -u dhub://library/neurodebian:xenial-non-free -i dhub/library/neurodebian/xenial-non-free--f5098f2
then it would be easy to navigate/see how many we have without requiring datalad containers-list (or knowing to look/grep .datalad/config).
NB it could even be -i dhub://library/neurodebian/xenial-non-free/f5098f2 thus really reflecting original URL ;)
- since there is only one version for a tag at a given point in time when we run it, and otherwise information about the tag would be lost, why not to create those container names following tags, e.g. instead of
neurodebian-f5098f2for dhub://library/neurodebian:xenial-non-free it to be justneurodebian--xenial-non-free[...]
For the initial pass, the digest gave me an easy thing to use without worrying about cleaning the names. There is (or was, after the latest push) a todo comment about this. I've switched it to cleaning the names now.
- even though not immediately usable, it would imho still be useful then to place those images not under flat
.datalad/imagesbut to establish uniform hierarchy of directories reflecting those docker hub repositories etc., so smth like
I've added namespace/repo subdirectories under the directories for images and manifests.
Please feel free to tweak and to move the script to wherever you'd like it.
@kyleam do you know if there is some associated with manifest date (I do not see anything in manifest)?
do you know if there is some associated with manifest date (I do not see anything in
manifest)?
I'm not sure I understand your question, but either way I'm confident that I don't know.
FWIW, I see that response headers for the tag contain only current Date, not some Last-modified
*(Pdb) p resp_man.headers
{'Content-Length': '2189', 'Content-Type': 'application/vnd.docker.distribution.manifest.v2+json', 'Docker-Conten
t-Digest': 'sha256:9e131ac6f30d682d71cbdbcd98e0c40b0b730e179172535dce4c5a82a2283c26', 'Docker-Distribution-Api-Ve
rsion': 'registry/2.0', 'Etag': '"sha256:9e131ac6f30d682d71cbdbcd98e0c40b0b730e179172535dce4c5a82a2283c26"', 'Dat
e': 'Fri, 30 Oct 2020 16:47:35 GMT', 'Strict-Transport-Security': 'max-age=31536000'}
why am I asking -- I thought to add the datetime to the image directory/name so it would provide ordering among images
FWIW, found the dates in the listing of tags:
*(Pdb) pprint(data)
{'count': 72,
'next': 'https://hub.docker.com/v2/repositories/library/neurodebian/tags?page=2',
'previous': None,
'results': [{'creator': 2215,
'full_size': 46057490,
'id': 13429244,
'image_id': None,
'images': [{'architecture': 'amd64',
'digest': 'sha256:9e131ac6f30d682d71cbdbcd98e0c40b0b730e179172535dce4c5a82a2283c26',
'features': '',
'last_pulled': '2020-10-30T14:15:58.27871Z',
'last_pushed': '2020-10-23T18:40:17.764065Z',
'os': 'linux',
'os_features': '',
'os_version': None,
'size': 46057490,
'status': 'active',
'variant': None}],
'last_updated': '2020-10-23T18:40:49.940676Z',
'last_updater': 1156886,
'last_updater_username': 'doijanky',
'name': 'xenial-non-free',
'repository': 42825,
'tag_last_pulled': '2020-10-30T14:15:58.27871Z',
'tag_last_pushed': '2020-10-23T18:40:49.940676Z',
'tag_status': 'active',
'v2': True},
so will take last_pushed for the image which might (unless the same image is re-pushed periodically for some reason?). also it suggested that we might want to encode architecture as well generally speaking, and may be even dump that image information record along side.
eh, confused between those "images" (multiple per tag) and then "manifest"'s digest being just one and do correspond to the image ID (but not really to digest record in the list of images), e.g. if I look at "busybox" (with bunch of architectures):
dump of exploration
*(Pdb) pprint([i for i in images if i['architecture'] == 'amd64'])
[{'architecture': 'amd64',
'digest': 'sha256:c9249fdf56138f0d929e2080ae98ee9cb2946f71498fc1484288e6a935b5e5bc',
'features': '',
'last_pulled': '2020-10-30T15:49:02.178905Z',
'last_pushed': '2020-10-14T10:25:29.856917Z',
'os': 'linux',
'os_features': '',
'os_version': None,
'size': 764619,
'status': 'active',
'variant': None}]
(Pdb) pprint(resp_man.json())
{'config': {'digest': 'sha256:f0b02e9d092d905d0d87a8455a1ae3e9bb47b4aa3dc125125ca5cd10d6441c9f',
'mediaType': 'application/vnd.docker.container.image.v1+json',
'size': 1493},
'layers': [{'digest': 'sha256:9758c28807f21c13d05c704821fdd56c0b9574912f9b916c65e1df3e6b8bc572',
'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
'size': 764619}],
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'schemaVersion': 2}
(Pdb)
[3] + 3845902 suspended python ./containers_add_dhub_tags.py <(echo busybox)
(dev3) 2 12894 ->148 [3].....................................:Fri 30 Oct 2020 01:26:39 PM EDT:.
(git)smaug:/mnt/btrfs/datasets/datalad/crawl/repronim/containers-backup[master]git
$> docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
9758c28807f2: Pull complete
Digest: sha256:a9286defaba7b3a519d585ba0e37d0b2cbee74ebfe590960b0b1d6a5e97d1e1d
Status: Downloaded newer image for busybox:latest
(dev3) 2 12895 [3].....................................:Fri 30 Oct 2020 01:26:48 PM EDT:.
(git)smaug:/mnt/btrfs/datasets/datalad/crawl/repronim/containers-backup[master]git
$> docker images --digests --all | grep busybox
busybox latest sha256:a9286defaba7b3a519d585ba0e37d0b2cbee74ebfe590960b0b1d6a5e97d1e1d f0b02e9d092d 2 weeks ago 1.23MB
with code diff (probably not paste/patchable as is)
diff --git a/containers_add_dhub_tags.py b/containers_add_dhub_tags.py
index 0755fbe..7f70bfa 100644
--- a/containers_add_dhub_tags.py
+++ b/containers_add_dhub_tags.py
@@ -16,6 +16,7 @@ import fileinput
import json
import logging
from pathlib import Path
+from pprint import pprint
import re
import requests
@@ -42,7 +43,10 @@ def clean_container_name(name):
def add_container(repo, tag, digest):
from datalad.api import containers_add
- target = Path("images", repo, digest)
+ # add suffix .dockersave so later we might save some other types
+ # of serialization or singularity converted images
+
+ target = Path(repo, "%s-%s.dockersave" % (tag, digest[:8]))
if target.exists():
lgr.info("Skipping %s:%s. Already exists: %s",
repo, tag, target)
@@ -73,21 +77,24 @@ def write_manifest(repo, digest, manifest):
target.write_text(json.dumps(manifest))
-def get_manifests(repo, tags):
+def get_manifest_images(repo, tag_images):
resp_auth = requests.get(REGISTRY_AUTH_URL.format(repo=repo))
resp_auth.raise_for_status()
headers = {
"Authorization": "Bearer " + resp_auth.json()["token"],
"Accept": "application/vnd.docker.distribution.manifest.v2+json"}
- for tag in tags:
+ for tag, images in tag_images:
lgr.debug("Getting manifest for %s:%s", repo, tag)
# TODO: Can we check with HEAD first to see if the digest
# matches what we have locally?
resp_man = requests.get(f"{REGISTRY_ENDPOINT}/{repo}/manifests/{tag}",
headers=headers)
resp_man.raise_for_status()
- yield tag, resp_man.json()
+ if len(images) != 1:
+ import pdb; pdb.set_trace()
+ raise NotImplementedError("ATM supporting only 1 image per tag. Got %s" % str(images))
+ yield tag, resp_man.json(), images[0]
def walk_pages(url):
@@ -101,10 +108,10 @@ def walk_pages(url):
yield from data.get("results", [])
-def get_repo_tags(repo):
+def get_repo_tag_images(repo):
url = f"{DHUB_ENDPOINT}/repositories/{repo}/tags"
for result in walk_pages(url):
- yield result["name"]
+ yield result["name"], result["images"]
def get_namespace_repos(name):
@@ -150,7 +157,8 @@ def process_files(files):
for repo in repos:
try:
- for tag, manifest in get_manifests(repo, get_repo_tags(repo)):
+ for tag, manifest, image in get_manifest_images(repo, get_repo_tag_images(repo)):
+ import pdb; pdb.set_trace()
digest = manifest["config"]["digest"]
assert digest.startswith("sha256:")
digest = digest[7:]
So is "image id" (which is in the manifest) the same across architectures?
yeap -- manifest is for all archs. Then it is possible to pull a specific image by giving that digest from images:
$> docker pull busybox@sha256:b8946184ce3ad6b4a09ebad2d85e81cfcaadc6897bfae2e9c6e2a4fe6afa6ee0
sha256:b8946184ce3ad6b4a09ebad2d85e81cfcaadc6897bfae2e9c6e2a4fe6afa6ee0: Pulling from library/busybox
5dce72bf4214: Pull complete
Digest: sha256:b8946184ce3ad6b4a09ebad2d85e81cfcaadc6897bfae2e9c6e2a4fe6afa6ee0
Status: Downloaded newer image for busybox@sha256:b8946184ce3ad6b4a09ebad2d85e81cfcaadc6897bfae2e9c6e2a4fe6afa6ee0
$> docker images --digests --all | grep busybox
busybox <none> sha256:b8946184ce3ad6b4a09ebad2d85e81cfcaadc6897bfae2e9c6e2a4fe6afa6ee0 65a89d0f0344 2 weeks ago 1.4MB
busybox latest sha256:a9286defaba7b3a519d585ba0e37d0b2cbee74ebfe590960b0b1d6a5e97d1e1d f0b02e9d092d 2 weeks ago 1.23MB
busybox latest sha256:c9249fdf56138f0d929e2080ae98ee9cb2946f71498fc1484288e6a935b5e5bc f0b02e9d092d 2 weeks ago 1.23MB
which is I guess what docker pull does -- chooses the one for matching architecture, but could be hand-twisted to run for the one desired (arm64 in this example)
$> docker run -it --rm busybox@sha256:b8946184ce3ad6b4a09ebad2d85e81cfcaadc6897bfae2e9c6e2a4fe6afa6ee0
standard_init_linux.go:178: exec user process caused "exec format error"
so, to support dumping images for various architectures there should also be a layer with <arch>-digest/.
another note (sorry for abusing this issue for that) summary: so far found no "datetime" for tag manifest, and tag manifest could change while referenced images would stay the same (and the same dates) (or change), so there in principle no "datetime" for tag (manifest), but for the images... but the next confusion point -- manifest record lists layers (a single one) -- how come that only 1 (I bet different architectures images have different ones... uff)
edit: if I do not request specific type of record for manifest, returned record does match described in API doc and returns record for a specific architecture
details
(Pdb) pprint(requests.get(f"{REGISTRY_ENDPOINT}/{repo}/manifests/{tag}", headers=dict(Accept="application/vnd.docker.distribution.manifest.v2+json", **headers)).json())
{'config': {'digest': 'sha256:f0b02e9d092d905d0d87a8455a1ae3e9bb47b4aa3dc125125ca5cd10d6441c9f',
'mediaType': 'application/vnd.docker.container.image.v1+json',
'size': 1493},
'layers': [{'digest': 'sha256:9758c28807f21c13d05c704821fdd56c0b9574912f9b916c65e1df3e6b8bc572',
'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
'size': 764619}],
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'schemaVersion': 2}
*(Pdb) pprint(requests.get(f"{REGISTRY_ENDPOINT}/{repo}/manifests/{tag}", headers=dict(**headers)).json())
{'architecture': 'amd64',
'fsLayers': [{'blobSum': 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},
{'blobSum': 'sha256:9758c28807f21c13d05c704821fdd56c0b9574912f9b916c65e1df3e6b8bc572'}],
'history': [{'v1Compatibility': '{"architecture":"amd64","config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["P
ATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["sh"],"ArgsEscaped":true,"Image":"sha256:11565868e68267a053372359046e1e70ce095538e95ff8398defd49bb66ddfce","Volumes":null,"WorkingDir":"","Entrypoint":nu
ll,"OnBuild":null,"Labels":null},"container":"6f1f5d35fed541933daae185eac73e333818ccec0b0760eb4cc8e30ce8d69de6","container_config":{"Hostname":"6f1f5d35fed5","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"A
ttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/sh","-c","#(nop) '
'","CMD '
'[\\"sh\\"]"],"ArgsEscaped":true,"Image":"sha256:11565868e68267a053372359046e1e70ce095538e95ff8398defd49bb66ddfce","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":{}},"c
reated":"2020-10-14T10:07:34.124876277Z","docker_version":"18.09.7","id":"6ed978c75173f577f023843ea61461568332f466c963e1b088d81fe676e8816c","os":"linux","parent":"bf938fec00b8d83c6d28a66dd6aa1cf76384aec8e63c7771648007b0dfce6fd8
","throwaway":true}'},
{'v1Compatibility': '{"id":"bf938fec00b8d83c6d28a66dd6aa1cf76384aec8e63c7771648007b0dfce6fd8","created":"2020-10-14T10:07:33.97009658Z","container_config":{"Cmd":["/bin/sh '
'-c #(nop) ADD '
'file:6098f054f12a3651c41038294c56d4a8c5c5d477259386e75ae2af763e84e683 '
'in / "]}}'}],
'name': 'library/busybox',
'schemaVersion': 1,
'signatures': [{'header': {'alg': 'ES256',
'jwk': {'crv': 'P-256',
'kid': '4XJH:HFRU:YB6H:MXPT:C4BI:KMEY:LEEZ:C4XI:6V73:X2JX:MPND:UCMK',
'kty': 'EC',
'x': '0LayaLTMdYehsVNmUXsLFLC35jCGdXtHNcudkUUlf70',
'y': '2B0uAwUNSkbn2DhNtw-pX5Shigm0Lpm-YtcGm1Rzj-w'}},
'protected': 'eyJmb3JtYXRMZW5ndGgiOjIxMjgsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAyMC0xMC0zMFQxNzo1NTozMFoifQ',
'signature': 'mxFa64066f0LyZ-Xw5expe2CylulXFBFgZgcyqor6e6Xt7KI2WSrY4nNRWDY3IpjaVUWFbAa2Gv_wPQfV-8JXA'}],
'tag': 'latest'}
edit2: https://docs.docker.com/registry/spec/manifest-v2-2/ came to rescue. I can request "fat" list of manifests and then it lists for all archs!
(Pdb) pprint(requests.get(f"{REGISTRY_ENDPOINT}/{repo}/manifests/{tag}", headers=dict(Accept="application/vnd.docker.distribution.manifest.list.v2+json", **headers)).json())
{'manifests': [{'digest': 'sha256:c9249fdf56138f0d929e2080ae98ee9cb2946f71498fc1484288e6a935b5e5bc',
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'platform': {'architecture': 'amd64', 'os': 'linux'},
'size': 527},
{'digest': 'sha256:a7c572c26ca470b3148d6c1e48ad3db90708a2769fdf836aa44d74b83190496d',
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'platform': {'architecture': 'arm',
'os': 'linux',
'variant': 'v5'},
'size': 527},
{'digest': 'sha256:ce800872092c37c5f20ef111a5a69c5c8e94d0c5e055f76f530cb5e78a26ec03',
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'platform': {'architecture': 'arm',
'os': 'linux',
'variant': 'v6'},
'size': 527},
{'digest': 'sha256:6655df04a3df853b029a5fac8836035ac4fab117800c9a6c4b69341bb5306c3d',
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'platform': {'architecture': 'arm',
'os': 'linux',
'variant': 'v7'},
'size': 527},
{'digest': 'sha256:b8946184ce3ad6b4a09ebad2d85e81cfcaadc6897bfae2e9c6e2a4fe6afa6ee0',
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'platform': {'architecture': 'arm64',
'os': 'linux',
'variant': 'v8'},
'size': 527},
{'digest': 'sha256:ba65e8d39e89b5c16f036c88c85952756777bf5385bce148bc44be48fac37d94',
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'platform': {'architecture': '386', 'os': 'linux'},
'size': 527},
{'digest': 'sha256:d7e83316d74e150866d82c45de342e78f662fe0aefbdb822d7d10c8b8e39cc4b',
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'platform': {'architecture': 'mips64le', 'os': 'linux'},
'size': 527},
{'digest': 'sha256:0a11a95568b680dce6906a015bed88381e28ad17b31a63f7fec057b35573235a',
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'platform': {'architecture': 'ppc64le', 'os': 'linux'},
'size': 528},
{'digest': 'sha256:426c855775f026d3fe76988b71938f4c9dc6840f09c0f29d8d4c75cc4238503b',
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'platform': {'architecture': 's390x', 'os': 'linux'},
'size': 528}],
'mediaType': 'application/vnd.docker.distribution.manifest.list.v2+json',
'schemaVersion': 2}
and then request specific manifest for specific architecture (digest of layer would differ) and the skinny one for the tag just corresponds to the first in the list (in this case arch amd64).
*(Pdb) pprint(requests.get(f"{REGISTRY_ENDPOINT}/{repo}/manifests/{tag}", headers=dict(Accept="application/vnd.docker.distribution.manifest.v2+json", **headers)).json())
{'config': {'digest': 'sha256:f0b02e9d092d905d0d87a8455a1ae3e9bb47b4aa3dc125125ca5cd10d6441c9f',
'mediaType': 'application/vnd.docker.container.image.v1+json',
'size': 1493},
'layers': [{'digest': 'sha256:9758c28807f21c13d05c704821fdd56c0b9574912f9b916c65e1df3e6b8bc572',
'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
'size': 764619}],
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'schemaVersion': 2}
*(Pdb) pprint(requests.get(f"{REGISTRY_ENDPOINT}/{repo}/manifests/sha256:c9249fdf56138f0d929e2080ae98ee9cb2946f71498fc1484288e6a935b5e5bc", headers=dict(Accept="application/vnd.docker.distribution.manifest.v2+json", **headers)).json())
{'config': {'digest': 'sha256:f0b02e9d092d905d0d87a8455a1ae3e9bb47b4aa3dc125125ca5cd10d6441c9f',
'mediaType': 'application/vnd.docker.container.image.v1+json',
'size': 1493},
'layers': [{'digest': 'sha256:9758c28807f21c13d05c704821fdd56c0b9574912f9b916c65e1df3e6b8bc572',
'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
'size': 764619}],
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
'schemaVersion': 2}
*(Pdb) pprint(requests.get(f"{REGISTRY_ENDPOINT}/{repo}/manifests/sha256:d7e83316d74e150866d82c45de342e78f662fe0aefbdb822d7d10c8b8e39cc4b", headers=dict(Accept="application/vnd.docker.distribution.manifest.v2+json", **headers)).json())
{'config': {'digest': 'sha256:1e4c9e707e11df98b555522738e99a0ece7f06cd157b53f4a0823de59a9b9478',
'mediaType': 'application/vnd.docker.container.image.v1+json',
'size': 1460},
'layers': [{'digest': 'sha256:3aef78b622fab981bf1fab26571f5bdc024afe8a9a8e2557b20d19243798e620',
'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
'size': 948824}],
'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
Ok, I think mystery is somewhat solved. We can get a list of images, use those records digests to requests manifests per specific arch digest (instead of requesting "default" one for the first architecture), save that manifest along with image manifest which has dates information.
now there is https://github.com/datalad/datalad-container/pull/135 , which should close this issue if merged