download blob timeout in kraken agent

[mmpei]

we use kraken to dispatch docker images. and about 400 nodes pulling a same image concurrently. all work well except two timeout. kraken version v0.1.2 see log from agent-nginx-log: 2019/05/24 03:09:24 [error] 22#22: *6 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /v2/push-perform/harbortestimages/blobs/sha256:e30bae4a4f776956ec66d75e10437e2d39c1624d53b29e5050224238c3d33eb4 HTTP/1.1", upstream: "http://unix:/tmp/kraken-agent-registry.sock/v2/push-perform/harbortestimages/blobs/sha256:e30bae4a4f776956ec66d75e10437e2d39c1624d53b29e5050224238c3d33eb4", host: "localhost:13047" see the attachment, agent47.log is the log from timeout agent. and agent46.log is from the agent worked well. Please do me a faver. i will do some deep investigation too.

[yiranwang52]

Had a discussion with @mmpei on Slack.

One possibility is, their tracker and redis have different latency compared to ours (i.e. their tracker has higher latency than redis; our redis has higher latency than tracker). Thus a peer could start announcing itself and itself discoverable, and instantly got 10 connections before it received response from tracker or finished processing the response, so it unexpectedly passed this check and that prevented it to call queue.Ready later.

[mmpei]

i add queue.Ready in the 'if' statement and it works well until now. i am not sure the cause of problem before. maybe it is clear now. i think it still needs some test. so is it needed for me to raise a PR after that? @yiranwang52

[yiranwang52]

@mmpei you can leave this one to us - I think we need a slightly bigger change.

[mmpei]

OK

[ish-xyz]

@yiranwang52 Hi are you still working on this project? I'm trying to contact any of the core contributor but I received no answer so far. I have a similar error.