Getting weird errors when doing a docker push to my private registry

[rchaudha]

Have been using the private docker registries for a while now. Am using registry-0.8.0. This is the first time I ran into the following errors:

I tried "docker push :":

1st time: HTTP code 400 while uploading metadata: {"error": "Image depends on a non existing parent"} 18:44:16 docker push /linux/ucs:1.0-20141121-184657.

2nd time: HTTP code 500 while uploading metadata: invalid character '<' looking for beginning of value 00:03:45.237 docker push /linux/ucs:1.0-20141121-195610.

In both cases, looking at the "images/" folder in the registry, I saw the previous sha that the docker client tried to push did not have the "layer" at all, instead it had "_inprogress" and then returned an error to the client.

Wondering what could be the issue here. Any suggestions on where to look? or what to look for?

[rchaudha]

Guys any help here?

[dmp42]

Hello @rchaudha

I would suggest you use the latest registry version to start (0.9).

Also, please read https://github.com/docker/docker-registry/blob/master/DEBUGGING.md and provide the information listed there ("Basics" and "Your private registry").

After that, might need as well your docker registry logs, and docker daemon logs.

I know it's boring... but really we can't help without at least these.

[dmp42]

Any update @rchaudha ?

[rchaudha]

Have been looking thru the registry code. Haven't upgraded to the latest registry yet. My setup is actually little complex. I have the registries writing to the NFS filer and I think there might be some kind of race condition in the registries being able to see the data on the NFS filer between writing the layer and the metadata. Same thing sometimes between writing json and layer too.

[databus23]

+1 We are are also seeing these errors. We are running multiple instances of a private registry, all using the same nfs share for storage. we are running version 0.9.1 of the registry.

[BugRoger]

Mounting the NFS shares with the option lookupcache=none magically fixes this problem. Quoting the nfs man page:

Note:  lookupcache=none can adversely affect per-
              formance, but may be necessary  if  shared  files
              created or deleted on the server need to be imme-
              diately visible to any  applications  running  on
              NFS clients.

[dmp42]

Ah, consistency and distributed filesystems... Thanks @BugRoger !

[kadishmal]

I had to disable cache via Redis in the registry config.yml for this error to disappear.