Dragonfly icon indicating copy to clipboard operation
Dragonfly copied to clipboard

Published 20 hours ago • verified publisher icon dragonflyoss

Unable to use harbor p2p distribution feature to preheat images with dragonfly

Open bmmadhuprasad opened this issue 4 years ago • 12 comments

I have deployed Harbor v2.0 and dragonfly latest version trying to use harbor p2p image distribution to dragonfly but getting below when applied the preheat policy to preheat image Check preheat progress: Task is failing and distribution is not happening

2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:93]: Preheating image 'dragonfly/test:0.1@sha256:54c9b521ae50156029ea48394041885f33414c1dad2a5deefcaa46ddf22079a3' to the target preheat provider: dragonfly newdragon:http://dragonflynew-supernode.dfn.svc.cluster.local:8002 2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:120]: Get preheat provider driver: dragonfly 2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:134]: Check health of preheat provider instance: Healthy 2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:146]: Sending preheat request is successfully done 2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:170]: Start to loop check the preheating status until it's success or timeout(30m) 2021-03-31T13:19:49Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z 2021-03-31T13:19:59Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z 2021-03-31T13:20:09Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z 2021-03-31T13:20:19Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z 2021-03-31T13:20:29Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z

Below is logs from dragonfly

2021-03-31 14:17:50.496 DEBU sign:9 : GET /_ping remote:192.168.71.183:40028 cost:2.901µs err: 2021-03-31 14:17:50.498 DEBU sign:9 : create preheat: Type[image] URL[https://harbor.madhu.mwpdemos.com/v2/test/mem/manifests/0.1] Filter[] Identifier[] Headers[map[Authorization:Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjdXTUE6S1ZYTzpLRjI1OjM1TkE6R0ozNTpaR0tZOlBHWVQ6STVVVDpUR1dZOlVVSTQ6UFQ3WjpYM0tXIiwidHlwIjoiSldUIn0.eyJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE2MTcyMDIwNjcsImp0aSI6Imc3TFhnUmJidmkzOFV4bFUiLCJpYXQiOjE2MTcyMDAyNjcsImlzcyI6ImhhcmJvci10b2tlbi1pc3N1ZXIiLCJuYmYiOjE2MTcyMDAyNjcsInN1YiI6ImRpc3RyaWJ1dG9yIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L21lbSIsImFjdGlvbnMiOlsicHVsbCJdfV19.DFeugff4V8VplCOGrW1AqAUhUTRo8F4pCwPAKa06zP8A67Euls4GC6MidfXpzAazU5QndtHBMkKdh-CBAUgzKLv73F3Kz1tdhXBAcyGXJHX7r8otQ4J8wNf0TwOHesmoW1sKc0IUBmlsA_a4fXR_7DAspzJmFiBchNin5qGCvhvn6449Tr_kF5HyoknOrw1NoK5_mG0mcBWLSC6E_16P1YIeAWaisfbuPurfnd-Zs2MLfSltWDjgibPXU02csiBmfGgy1mES0icDKkgeuqiLlFz90pgV61cZcQieqAeB5k3quMN20HWC3EaYTqPyspo0ZMjc_w1WItldMFDusncH8kqak7iKfQYSISf2MlD7WSmHO77_jgePWsEYKvdQIuLOcY-PZ-dswISb1DE72hJ6ZvNEnOna7LKe67VfFgfBvar4UEtwCzMCyJt0aAUzepgvFbdA1ajUQagpeAlco5o-oZmvmbfay_xiBRxEsBa0n9KIKrxQ3ceLhx5xCFau3OH9Poa9ch2QeMI7bYe5BAkL2xQKzZdk4LNkG6hpQtucLNlCzMTOZ_HO7E4lFXe2LQlYcNy6XSSv16g1R5qjM2W4UBf_pmGGo1E4wTC0WSVlgIXdlCkVo8P1h6xfVhmOfnZWiZgVmPynR21urj1pW-M0FEbarBefU-JTvQU4Qgt25Q0]] 2021-03-31 14:17:50.498 DEBU sign:9 : POST /preheats remote:192.168.71.183:40028 cost:193.662µs err: 2021-03-31 14:18:00.502 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:33.646µs err: 2021-03-31 14:18:10.499 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:24.602µs err: 2021-03-31 14:18:20.499 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:37.729µs err: 2021-03-31 14:18:30.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:25.405µs err: 2021-03-31 14:18:40.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:23.134µs err: 2021-03-31 14:18:50.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:28.289µs err: 2021-03-31 14:19:00.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:33.542µs err:

Can anybody let me know anything i am missing what may be the cause for the above issue

bmmadhuprasad avatar Apr 01 '21 09:04 bmmadhuprasad

even i have deployed dragonfly supernode in VM directly from source code without using docker as per document and tried to use that as end point in harbor but i see same error when i created p2p preheat policy and apply to initiate image preheat to dragonfly Below is the error i see from harbor 2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:93]: Preheating image 'test/mem:0.1@sha256:9871d56e046e38e9daaef6b592e67282c1e286a85e84797ed18b942a8263eadf' to the target preheat provider: dragonfly vm:http://54.215.243.61:8002 2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:120]: Get preheat provider driver: dragonfly 2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:134]: Check health of preheat provider instance: Healthy 2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:146]: Sending preheat request is successfully done 2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:170]: Start to loop check the preheating status until it's success or timeout(30m) 2021-04-05T13:49:10Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:49:20Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:49:30Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:49:40Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:49:50Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:50:00Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:50:10Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:50:20Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:50:30Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:50:40Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:50:50Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:51:00Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:51:10Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:51:20Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:51:30Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:51:40Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:51:50Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:52:00Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check p below is the logs from dragonfly supernode 1ZYTzpLRjI1OjM1TkE6R0ozNTpaR0tZOlBHWVQ6STVVVDpUR1dZOlVVSTQ6UFQ3WjpYM0tXIiwidHlwIjoiSldUIn0.eyJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE2MTc2MzIzMzksImp0aSI6IkNobzlxcXFhMmFGTWJKeDMiLCJpYXQiOjE2MTc2MzA1MzksImlzcyI6ImhhcmJvci10b2tlbi1pc3N1ZXIiLCJuYmYiOjE2MTc2MzA1MzksInN1YiI6ImRpc3RyaWJ1dG9yIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L21lbSIsImFjdGlvbnMiOlsicHVsbCJdfV19.RXfGEpeMUpNYQA2v2F60IPSnSy4h_X4WKFEIlYAd0H363oVE5omj25faYccMBEU3XFPeFUWx2tGaHCdrbgnSa5SnQ5qJgerJ4FmGYCNgi1Pvr-eFjEjC8hqNAREoiZ25xsF4lg90t7-YQ_UZSxgmwMKQNRfaREseJw1emTXDLAulJSxzJVO6egEt2_jWAWsK58zyTBu40zTFGwD6fUr27vAXn2xf3KSPETf4Hf0HunDpzoZFTqMqAmHSp5w3loPuHvGF9pt9f4_blkT1JULmFFc3_zcSkuF-NDcqT2UAxN7ZQHaYJpG6uhuQmEc0bJ8fR69v1Ag_CKppOzUZbl0fGVMEBn0PDJLn67leEo5UwBn-knydMtPozfYC1fg20PxVAASo9i964ZS1jkbOoDfjpG5QKf-al5DgaCge8O6ULFo2PKPUZaip-6opnpcdG65tRAP64UZvCCYBZ8l1maecZHK2MLgQYiD5lGInfjcUWEYaENtw4FXrRltRteUiv1GYN8jBktuoAKZevLsia0ULMhayenlv3u999yAnLnAgjGaYijNWv6r2IMeO6mFnI6AZSzlKH2ig3N25gPbxh1UT7PC7jd-mJEuxVd_BBMFhS2BttRbLqyJf6Bkr5T4gKOYyVjAm0tBG0zdhulrjakAt97BW6v3IFbP2cI0aL1-vk"},"type":"image","url":"https://harbor.madhu.mwpdemos.com/v2/test/mem/manifests/0.1"}, res:{"body":{"id":"3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735"},"headers":{},"statusCode":"OK","statusCodeValue":200} 2021-04-05 13:49:00.958 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:c1c1ba809094891c97d8ca9f34817f54c72cfab868b6914d85f2a156a6ac26a9 parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 2021-04-05 13:49:00.969 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:ff59de7222ecb2f8dd85ac954ca687d3b63b726d7a9631964f00535e65339584 parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 2021-04-05 13:49:00.970 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:79f39971f65f5d879d2ed9e4402b5fba9a825803c4c76b8a09e4926ea58185c2 parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 2021-04-05 13:49:00.973 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:e72295e4b7adb87d61408b4101c0e4f8ccc15c84ccbfdbfe756e410f83b0fcaf parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 2021-04-05 13:49:00.974 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:e38d1a0197a41b27b23313a6ae0ec7f4219c31fdb55e5f5b9c43ee781237312f parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 2021-04-05 13:49:00.980 INFO [preheat-12] c.d.d.s.s.p.PreheatServiceImpl - command: /usr/local/bin/dfget -u https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:ff59de7222ecb2f8dd85ac954ca687d3b63b726d7a9631964f00535e65339584 -o /home/admin/supernode/repo/preheat/87626317-ef0f-448a-8a6e-78e60f72cc6c --callsystem dragonfly_preheat --totallimit 90M -s 90M --notbs --header Authorization:Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjdXTUE6S1ZYTzpLRjI1OjM1TkE6R0ozNTpaR0tZOlBHWVQ6STVVVDpUR1dZOlVVSTQ6UFQ3WjpYM0tXIiwidHlwIjoiSldUIn0.eyJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE2MTc2MzIzMzksImp0aSI6IkNobzlxcXFhMmFGTWJKeDMiLCJpYXQiOjE2MTc2MzA1MzksImlzcyI6ImhhcmJvci10b2tlbi1pc3N1ZXIiLCJuYmYiOjE2MTc2MzA1MzksInN1YiI6ImRpc3RyaWJ1dG9yIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L21lbSIsImFjdGlvbnMiOlsicHVsbCJdfV19.RXfGEpeMUpNYQA2v2F60IPSnSy4h_X4WKFEIlYAd0H363oVE5omj25faYccMBEU3XFPeFUWx2tGaHCdrbgnSa5SnQ5qJgerJ4FmGYCNgi1Pvr-eFjEjC8hqNAREoiZ25xsF4lg90t7-YQ_UZSxgmwMKQNRfaREseJw1emTXDLAulJSxzJVO6egEt2_jWAWsK58zyTBu40zTFGwD6fUr27vAXn2xf3KSPETf4Hf0HunDpzoZFTqMqAmHSp5w3loPuHvGF9pt9f4_blkT1JULmFFc3_zcSkuF-NDcqT2UAxN7ZQHaYJpG6uhuQmEc0bJ8fR69v1Ag_CKppOzUZbl0fGVMEBn0PDJLn67leEo5UwBn-knydMtPozfYC1fg20PxVAASo9i964ZS1jkbOoDfjpG5QKf-al5DgaCge8O6ULFo2PKPUZaip-6opnpcdG65tRAP64UZvCCYBZ8l1maecZHK2MLgQYiD5lGInfjcUWEYaENtw4FXrRltRteUiv1GYN8jBktuoAKZevLsia0ULMhaye Please help us to fix the issue and let us know anything i am missing

bmmadhuprasad avatar Apr 05 '21 14:04 bmmadhuprasad

Is the image public?

ansinlee avatar Apr 06 '21 02:04 ansinlee

yes project created in harbor where this image pushed is public image

bmmadhuprasad avatar Apr 06 '21 05:04 bmmadhuprasad

dragonfly supernode image i am using is dragonflyoss/supernode:1.0.6,I have also tried with latest image of supernode but same issue.

bmmadhuprasad avatar Apr 06 '21 05:04 bmmadhuprasad

Thanks for report this issue. I will verify it ASAP.

jim3ma avatar Apr 07 '21 02:04 jim3ma

query the detail error message from dragonfly server of the task as the following command curl http://{dragonfly-server-address}/preheat/{task-id}

for example: curl http://192.168.0.180:8002/preheats/51e4eec494ff56f5eeb1e2dfcc96b1a65783b110e54671d45dc4b98f030b6127

{"ID":"51e4eec494ff56f5eeb1e2dfcc96b1a65783b110e54671d45dc4b98f030b6127","finishTime":"2021-04-07T08:42:52.682Z","startTime":"2021-04-07T08:42:49.681Z","status":"FAILED","errorMsg":"Get http://xx.xx.xx.xx/v2/library/nginx/manifests/latest: dial tcp xx.xx.xx.xx:80: i/o timeout"}

ansinlee avatar Apr 07 '21 09:04 ansinlee

@ansinlee curl http://13.56.114.60:8002/preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 {"ID":"3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735","finish Time":"2021-04-08T09:28:33.656Z","startTime":"2021-04-08T09:28:33.400Z","status" :"FAILED","errorMsg":"Get https://harbor.madhu.mwpdemos.com/v2/test/mem/manifest s/0.1: x509: certificate signed by unknown authority"}

bmmadhuprasad avatar Apr 08 '21 09:04 bmmadhuprasad

@bmmadhuprasad It seems your harbor cert is not valid for dragonfly supernode.

jim3ma avatar Apr 08 '21 11:04 jim3ma

I also met the same problem, how to solve it?

liang-junwei avatar May 28 '21 08:05 liang-junwei

I also met the same problem，my harbor version is v2.2.1

huangpxs avatar Aug 27 '21 14:08 huangpxs

Follow https://blog.jim.plus/blog/post/jim/selfsign-ca-and-update-certficates to update ca chains.

jim3ma avatar Aug 29 '21 08:08 jim3ma

@bmmadhuprasad Obviously, this is a bug of Dragonfly 1.x. In the file https://github.com/dragonflyoss/Dragonfly/tree/master/supernode/daemon/mgr/preheat/image_preaheater.go, the getLayers method does not support TLS to obtain image layer information.

Therefore, before fixing the problem, you need to manually set the certificate trust for the supernode container. The operation is as follows: docker exec -it supernode bash cp /certs/ca.crt /usr/local/share/ca-certificates/ update-ca-certificates docker restart supernode

zkking avatar Nov 22 '21 12:11 zkking