discogs-xml2db
discogs-xml2db copied to clipboard
Parallelize get_latest_dumps script with xargs
To maximize usage of bandwidth and make the download marginally faster, use xargs in combination with wget.
Also, remove USER_AGENT as the parentheses don't play nice with xargs, and also because wget functions just fine without it.
The output is a bit hectic, with the progress bars from the parallel processes constantly overwriting each other...but totally worth it IMHO :)
The discogs team requires User-Agent for their API.
I don't think it's required to download the zip files - is there a way we can make sure of it?
@philipmat thanks for the heads up! I will put it back. Is it cool if we make the User-Agent something simpler, like DiscogsXml2Db/1.0? That's what the Discogs API documentation recommends.
Of interest: if we remove the USER_AGENT parameter, wget will provide a default user agent, Wget/1.0 or whatever version, which it should probably technically be. https://superuser.com/questions/495855/what-is-the-default-user-agent-when-using-wget-on-linux