LAION-Face
LAION-Face copied to clipboard
fail to download laion face image
hello, when I run
bash download.sh ./laion_face_meta ./laion_face_data
tha download fails
total - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 258 - count: 20000
worker - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 131 - count: 10000
total - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 387 - count: 30000
5it [01:23, 8.00s/it]worker - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 124 - count: 10000
total - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 495 - count: 40000
worker - success: 0.003 - failed to download: 0.997 - failed to resize: 0.000 - images per sec: 123 - count: 10000
total - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 616 - count: 50000
9it [01:28, 2.50s/it]worker - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 116 - count: 10000
total - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 698 - count: 60000
worker - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 116 - count: 10000
total - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 811 - count: 70000
worker - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 117 - count: 10000
total - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 927 - count: 80000
worker - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 117 - count: 10000
total - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1043 - count: 90000
How can I fix it? Thanks.
Hello,did you solve the problem?
I've encountered the same issue, and partially resolved it.
-
laion_face_data/*/*_stats.json
files reported many name resolution failure- something like:
<urlopen error [Errno -3] Temporary failure in name resolution>: 9950
- something like:
- While running
download.sh
,ping google.com
reportedping: google.com: Temporary failure in name resolution
. - a fast dns resolver didn't resolve the issue.
- Lowering process count and thread count made it significantly better. (in
download.sh
, line 11, I changed--processes_count 16 --thread_count 128
to--processes_count 2 --thread_count 16
1it [09:09, 549.96s/it]worker - success: 0.813 - failed to download: 0.179 - failed to resize: 0.008 - images per sec: 18 - count: 10000
total - success: 0.813 - failed to download: 0.179 - failed to resize: 0.008 - images per sec: 18 - count: 10000
2it [09:18, 231.67s/it]worker - success: 0.802 - failed to download: 0.189 - failed to resize: 0.009 - images per sec: 18 - count: 10000
total - success: 0.807 - failed to download: 0.184 - failed to resize: 0.009 - images per sec: 36 - count: 20000
- After the change,
laion_face_data/*/*_stats.json
looked sane. - 80% of URL being up should be normal.
- It may be the issue of my DNS server, but I stopped digging further.
Thank you very much, it helps me a lot
Thank you for your suggestion, it helps me a lot
------------------ 原始邮件 ------------------ 发件人: "Han @.>; 发送时间: 2023年8月27日(星期天) 凌晨0:45 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [FacePerceiver/LAION-Face] fail to download laion face image (Issue #10)
I experience the same issue.
a fast dns resolver didn't resolve the issue
Lowering process count and thread count made it significantly better. (in download.sh, line 11, I changed --processes_count 16 --thread_count 128 to --processes_count 2 --thread_count 16
1it [09:09, 549.96s/it]worker - success: 0.813 - failed to download: 0.179 - failed to resize: 0.008 - images per sec: 18 - count: 10000 total - success: 0.813 - failed to download: 0.179 - failed to resize: 0.008 - images per sec: 18 - count: 10000 2it [09:18, 231.67s/it]worker - success: 0.802 - failed to download: 0.189 - failed to resize: 0.009 - images per sec: 18 - count: 10000 total - success: 0.807 - failed to download: 0.184 - failed to resize: 0.009 - images per sec: 36 - count: 20000
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>
Thanks for the tips. In my case, setting up a fast DNS server speeds up downloading significantly.