discogs-xml2db icon indicating copy to clipboard operation
discogs-xml2db copied to clipboard

Parallelize get_latest_dumps script with xargs

Open rayrrr opened this issue 6 years ago • 3 comments

To maximize usage of bandwidth and make the download marginally faster, use xargs in combination with wget.

Also, remove USER_AGENT as the parentheses don't play nice with xargs, and also because wget functions just fine without it.

The output is a bit hectic, with the progress bars from the parallel processes constantly overwriting each other...but totally worth it IMHO :)

rayrrr avatar Jul 31 '18 18:07 rayrrr

The discogs team requires User-Agent for their API.

I don't think it's required to download the zip files - is there a way we can make sure of it?

philipmat avatar Jul 31 '18 20:07 philipmat

@philipmat thanks for the heads up! I will put it back. Is it cool if we make the User-Agent something simpler, like DiscogsXml2Db/1.0? That's what the Discogs API documentation recommends.

rayrrr avatar Jul 31 '18 20:07 rayrrr

Of interest: if we remove the USER_AGENT parameter, wget will provide a default user agent, Wget/1.0 or whatever version, which it should probably technically be. https://superuser.com/questions/495855/what-is-the-default-user-agent-when-using-wget-on-linux

rayrrr avatar Aug 23 '18 14:08 rayrrr