hoad icon indicating copy to clipboard operation
hoad copied to clipboard

speed up Crossref API calls

Open njahn82 opened this issue 4 years ago • 1 comments

To speed up Crossref API fetch, I used the furrr package for parallel processing and was able to obtain metadata records in about 90 minutes. The code ran on a AWS t2.2xlarge instance type (8vCPU).

To be kind, we should avoid running sessions in parallel, but closely monitor the performance of the calls. Maybe we can avoid parallel processing in future.

The code is tracked in the 2020_update branch

https://github.com/subugoe/hoad/blob/f44634f3faaa3319dfb5ad623b1cb09c3701f8d7/R/cr_fetching.R#L231-L251

njahn82 avatar Apr 14 '20 10:04 njahn82