hoad
hoad copied to clipboard
speed up Crossref API calls
To speed up Crossref API fetch, I used the furrr package for parallel processing and was able to obtain metadata records in about 90 minutes. The code ran on a AWS t2.2xlarge instance type (8vCPU).
To be kind, we should avoid running sessions in parallel, but closely monitor the performance of the calls. Maybe we can avoid parallel processing in future.
The code is tracked in the 2020_update
branch
https://github.com/subugoe/hoad/blob/f44634f3faaa3319dfb5ad623b1cb09c3701f8d7/R/cr_fetching.R#L231-L251