docker-registry Failed push breaks repo

I believe we had a failed push, which broke a repo in our registry. The logs from the push look like it hung, and was eventually killed. This seem to put a layer in the index which doesn't exist, and we can no longer pull that image.

When pulling, the docker server shows Server error: 404 trying to fetch remote history for 0554e7f95c4a74fcf9600d76ef62f0c4d32

The pull continues after the 404, but ends with Could not find repository on any of the indexed registries.

This is a prod registry running with S3 as a backend.

May 16 '14 14:05 jbardin

I have this problem too.

May 26 '14 11:05 panjunyong

Hi guys,

I would like to reproduce this, and possibly add it to the tests.

Do you have any info for me to get started on that? Do you believe the push broke when sending a layer?

Thanks!

May 30 '14 17:05 dmp42

I don't have that registry up any longer, since it needed to be repiared. We think that the push failed while sending a layer, and never completed.

I think we ended up with a layer in _index_images which didn't exist in S3.

May 30 '14 17:05 jbardin

Ok, thanks - I'll try to smoke it out.

May 30 '14 17:05 dmp42

/cc @samalba @shin-

Jun 04 '14 04:06 jpetazzo

@jpetazzo did you get anything new on this?

Jun 04 '14 04:06 samalba

I was drawn to this issue by this message on docker-user: https://groups.google.com/forum/#!topic/docker-user/QVWneEWcMk4

On Tue, Jun 3, 2014 at 9:08 PM, Sam Alba [email protected] wrote:

@jpetazzo https://github.com/jpetazzo did you get anything new on this?

— Reply to this email directly or view it on GitHub https://github.com/dotcloud/docker-registry/issues/366#issuecomment-45049762 .

@jpetazzo https://twitter.com/jpetazzo Latest blog post: http://jpetazzo.github.io/2014/03/23/lxc-attach-nsinit-nsenter-docker-0-9/

Jun 04 '14 04:06 jpetazzo

Has any progress been made on this issue? I'm currently running into it using AWS ECR.

Jul 11 '16 21:07 Sciencegeek123