uniclust-pipeline icon indicating copy to clipboard operation
uniclust-pipeline copied to clipboard

Published 20 hours ago • verified publisher icon soedinglab

uniclust30 download mirror?

Open clayfos opened this issue 6 years ago • 11 comments

Greetings,

Is there a mirror for the database downloads? I'm attempting to download the uniclust30 for hhsuite using the link from the uniclust.mmseqs.com site, but the download is exceedingly slow and fails after reaching ~16gb.

Thanks!

clayfos avatar Aug 29 '18 14:08 clayfos

Could you please try if the following url works better: http://wwwuser.gwdg.de/~compbiol/uniclust/2017_10/

milot-mirdita avatar Aug 29 '18 14:08 milot-mirdita

I'm sorry, I should have been more specific, that's the directory that uniclust.mmseqs.com directs me to. This is a direct link to what I have been attempting to download that ultimately fails:

http://wwwuser.gwdg.de/~compbiol/uniclust/2017_10/uniclust30_2017_10_hhsuite.tar.gz

clayfos avatar Aug 29 '18 15:08 clayfos

uniclust.mmseqs.com links to a different server (subdomain gwdu111). Does the wwwuser server work better/differently?

milot-mirdita avatar Aug 29 '18 15:08 milot-mirdita

Unfortunately no, not that I can see. But I may be misunderstanding, because uniclust.mmseqs.com appears to already link to the wwwuser server, and that's what I already attempted to download.

clayfos avatar Aug 29 '18 16:08 clayfos

Okay now I see the source of confusion, the link in the text points to: http://wwwuser.gwdg.de/~compbiol/uniclust/2017_10/

The link in the header points towards: http://gwdu111.gwdg.de/~compbiol/uniclust/2017_10/

Try the other one and see if it improves the situation. We don't have other servers. wget/curl support continuing previous downloads, maybe try that?

milot-mirdita avatar Aug 29 '18 18:08 milot-mirdita

The gwdu111 subdomain link worked great! Thanks very much for looking into this for me.

clayfos avatar Aug 30 '18 16:08 clayfos

This is an old issue but I am currently trying to download from the gwdu111 link and I am getting less than 100 KB/s download speed. Have tried on 2 different connections and both are slow. Any plans to host them on high bandwidth servers?

Thanks.

KrollBio avatar Jun 21 '19 20:06 KrollBio

It took a day for me to download the hh-suite archive. My guess is something is wrong with the server.

rayoub avatar Jan 25 '20 15:01 rayoub

I get around 1MB/s even on good networks. Would it make sense to mirror the database to github (as release-attached file), or to zenodo?

tonigi avatar Dec 22 '20 15:12 tonigi

I would recommend to try to download with aria2c. It offers the option to use multiple simultaneous connections to download (-x or --max-connection-per-server). That might speedup the download.

Zenodo might work, but we are already very close to the 50GB limit.

milot-mirdita avatar Jan 06 '21 14:01 milot-mirdita

I would recommend to try to download with aria2c. It offers the option to use multiple simultaneous connections to download (-x or --max-connection-per-server). That might speedup the download.

Zenodo might work, but we are already very close to the 50GB limit.

I am downloading this. It is at 3.7 MB/s with total wait time of 1.5 hours

skr3178 avatar Dec 02 '22 02:12 skr3178