yari icon indicating copy to clipboard operation
yari copied to clipboard

use async fs in buildDocument loop

Open LeoMcA opened this issue 3 years ago • 1 comments

this results in a ~10% speedup on my machine

didn't throw the sitemap and search index stuff in Promise.alls yet, as they're only iterating over each locale, so I don't imagine we'll get a huge speed-up there

next up is async-ing the reading of files in this document loop, which is a bit convoluted because the Document.read function isn't async

LeoMcA avatar Jun 22 '22 11:06 LeoMcA

This pull request has merge conflicts that must be resolved before it can be merged.

github-actions[bot] avatar Aug 12 '22 16:08 github-actions[bot]

@LeoMcA Do you recall if the changes in this PR have any negative side-effect, or limitations?

caugner avatar Aug 24 '22 15:08 caugner

One negative side-effect was the duplicated index.html/md files we had racing and producing invalid JSON (where in our current build process, because they're done sequentially, one "wins out" rather than a Frankenstein's Monster of a bit of each).

We'd want to check we have appropriate guards in place against that - Florian suggested checking before the build and failing if there was, as well as our PR checks on content/translated-content.

Otherwise, I sadly didn't see the expected speed gains on Github Actions when testing, so I imagine there's some other bottleneck not present on my machine - but there shouldn't be a downside, and this might help us move towards a build system where we can parallelize one locale across multiple threads, rather than our current limitation of one locale per thread.

I'm happy to loop back to this after the collections work, but feel free to take it on if it interests you!

LeoMcA avatar Aug 25 '22 09:08 LeoMcA