MountainProject icon indicating copy to clipboard operation
MountainProject copied to clipboard

Attempts to make db build faster

Open derekantrican opened this issue 5 years ago • 1 comments

See if I can take some hints from buildxl to only rebuild things that have actually changed.

For instance: if a route has not changed, we may be able to avoid all the HTML parsing and instead skip it

derekantrican avatar Mar 27 '20 18:03 derekantrican

Ideas:

  • Hashing the HTML and if it has not changed, skipping all of the IHtmlDocument/parsing/etc lines (note that we will probably still need to get the IHtmlDocument for subareas/routes so that we can check those as well)
    • Need to check speed of hashing algorithms and see if there would actually be any time savings here to "hash HTML and check hash against previous" or "just load the IHtmlDocument and do all the parsing anyway"
  • Maybe there's a faster way to download HTML from a URL? Possibly: https://stackoverflow.com/a/54502464/2246411 (may need to investigate a couple different options)
  • See what happens if I take off the awaits before ParseRouteAsync and ParseAreaAsync (since I don't actually need to wait before those things complete. Last time I tried this I think this meant that the Task.WaitAll actually finished too quickly because all the DestAreas finished before any of the children

derekantrican avatar Mar 28 '20 18:03 derekantrican