discogs-xml2db
discogs-xml2db copied to clipboard
Parallelize get_latest_dumps script with xargs
To maximize usage of bandwidth and make the download marginally faster, use xargs
in combination with wget
.
Also, remove USER_AGENT
as the parentheses don't play nice with xargs
, and also because wget
functions just fine without it.
The output is a bit hectic, with the progress bars from the parallel processes constantly overwriting each other...but totally worth it IMHO :)
The discogs team requires User-Agent for their API.
I don't think it's required to download the zip files - is there a way we can make sure of it?
@philipmat thanks for the heads up! I will put it back. Is it cool if we make the User-Agent something simpler, like DiscogsXml2Db/1.0
? That's what the Discogs API documentation recommends.
Of interest: if we remove the USER_AGENT
parameter, wget
will provide a default user agent, Wget/1.0
or whatever version, which it should probably technically be. https://superuser.com/questions/495855/what-is-the-default-user-agent-when-using-wget-on-linux