Feedback after migrating a Docker Trusted Registry

[matias2681]

I've recently use this tool to migrate our DTR on-prem to Artifactory.

Here are a few things that I found lacking in the documentation:

1. In the generic method it's confusing the repo concept as a parameter. It should be mentioned somewhere that all the DTR repositories will be move to that Artifactory repo.
2. There should be disclaimer about the need to have enough disk available depending on the layer size. I found that we have some ridiculous big images (11 GB) with layers going up to 3 GB. If you run that with 16 threads it will use 48 GB that's quite a number depending on which filesystem you run the tool.
3. It would be nice to have a more secure way to run the tool without having to pass the passwords as a parameters. The code doesn't seem to support an environment variable to set those details. The main problem is that the passwords are leaked when you do ps aux | grep python

That covers the improvements given my use case. Beside those things that I mentioned the tool is working just fine.

[matias2681]

After 16 hours of migration and 3 TB of data I have some more feedback:

1. Retries in failures. It'd be nice to have a default of 3 or so retries when there's a failure to upload the layer, not sure if that's already in place but it doesn't seem to be in the logs. I've found some errors that maybe it could be fixed with a retry.

2020-08-28 18:04:42,535 [Thread-8] [ERROR] (migrator.ArtifactoryDockerAccess:204) - Error uploading artifact:
2020-08-28 18:04:42,539 [Thread-8] [ERROR] (migrator.Migrator:78) - Unable to upload layer sha256:35e8c4ca28705b5b731c3d457172acf5e1aa3e2137763a969b51c22a0c8fsad5 for ....

2. Metrics.

• number of images migrated
• [nice to have] avg network transfer speed
• time that took the migration

3. logs. I had to run the tool in verbose mode to know that the tool wasn't stuck as it wasn't showing anything till the very end. Maybe you can add information for each image that is migrated in the console.

Thanks for taking the feedback. The tool works quite well for us.

[arturo-aparicio]

Thanks for the feedback. We have added some of the items as feature requests to potentially be implemented in the future.

I would like to mention that we recently discovered a bug in Docker Trusted Registry. This bug affects some DTR versions and it may affect you. The bug causes DTR to not report a full list of all the images via the catalog REST API. This is what the tool relies on to build the list of images to import. This means some images may not have been imported. We have created a new release in the master branch that includes a 'dtr' specific function that uses the DTR proprietary API instead of the catalog, this should get the full list of all images. Give this a try, you can just run it against the same DTR and Artifactory repository and it will only import any images that were missed the first time around (if any).