DOCFX Build is extremely slow - times out 1 hour limit for AppVeyor builds

[smaillet]

Operating System: Windows

DocFX Version Used: 2.42.2

Template used: custom (based on statictoc)

Steps to Reproduce:

1. clone the repo from github https://github.com/UbiquityDotNET/Llvm.NET/tree/f717300813a11f6bf9621dc740e7772cd783b08d (This commit point shows the problem and won't include any workarounds I end up with for dealing with this)
2. Open powershell command prompt to the newly cloned source location
3. run .\Buildall.ps1

Expected Behavior: All code and docs build in a reasonable amount of time.

Actual Behavior: Total build time exceeds 1 hour on AppVeyor builds causing a failure due to time limits of 1 hour per job. Overwhelming majority of the total time is all in the DOCFX stage. (see: AppVeyor logs of failing build for more details)

[dave-sampson13]

Any thoughts on this one? I believe this is affecting us too (using just the standard statictoc template).

If I use the default template we can build our docs in just under 2 minutes, using the statictoc template instead takes in the vicinity of an hour.

Why does it take so long, what is the statictoc template doing that the default template isn't? And is there a reason why the default template (& any other templates based on it, for that matter) couldn't support filesystem HTML viewing (including a table of contents & navigation) rather than needing an entirely separate template just to support that?

We'd like to be able to distribute our documentation so that it can be viewed from the filesystem OR hosted by a web server, & not have to execute a different build using a different template dependent on that.

Thanks for your help!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[smaillet]

This is not stale. Ignoring the issue should not auto close it. This is an ongoing issue that blocks use of docfx as described

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[shmuelie]

Not stale

[dave-sampson13]

@smaillet @SamuelEnglard I ended up working around this issue by replacing the statictoc template with my own "jstoc" template. Instead of the approach taken by the statictoc template that seems to build & add the toc to every single page (I guess hence the static part of its name) - which I'd assume is why it's so slow & the resulting output is so large - I build a toc once & reference it in js. This works for both web & filesystem deployment, the only (minor) downside is that it doesn't work in IE as it relies on JS template literals, but works in modern browsers. If you have interest (given there seems to be little interest here) I'm happy to share with you directly. In a perfect world I'd contribute & link from the docfx templates page, but in the interests of expediency thought it might be easier to share directly.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[thetestgame]

I have this issue as well. Builds take well in excess of 2 hours

[BreakPointSoftware]

@smaillet @SamuelEnglard I ended up working around this issue by replacing the statictoc template with my own "jstoc" template. Instead of the approach taken by the statictoc template that seems to build & add the toc to every single page (I guess hence the static part of its name) - which I'd assume is why it's so slow & the resulting output is so large - I build a toc once & reference it in js. This works for both web & filesystem deployment, the only (minor) downside is that it doesn't work in IE as it relies on JS template literals, but works in modern browsers. If you have interest (given there seems to be little interest here) I'm happy to share with you directly. In a perfect world I'd contribute & link from the docfx templates page, but in the interests of expediency thought it might be easier to share directly.

@dave-sampson13 I am getting its problem also, would you be able to point me in the direction of th work around you use?

[smaillet]

@BreakPointSoftware I'd be interested in any solutions for reducing docs build times.

[dave-sampson13]

@BreakPointSoftware @smaillet

I've attached a ZIP file containing the custom docfx template we wrote, as well as a minimal docfx.json file that references it. There's a fair bit in the template because we just needed to find everywhere that referenced the old HTML-based TOC & replace it with the js-based TOC. I think you should be able to use the template as-is, we did have another template in the pipeline in the docfx.json file after the jstoc template but from memory that's just cosmetic changes to the material theme, & not needed for the jstoc template to work. Do let me know though if you have any issues & I'll try to help, this has "just been working" for us for the last year and a half so haven't really needed to touch it in a while!

docfx-jstoc.zip

[sandrofigo]

Running docfx build on ~100 projects takes several hours to complete. Using default docfx.json from tutorials in documentation.

I haven't had the time to look at the engine behind docfx, but what parts of it might cause this behaviour?

[paulushub]

Running docfx build on ~100 projects takes several hours to complete.

How long does it take to build the ~100 projects without DocFX?

[sandrofigo]

Doing a rebuild of the whole solution takes ~7 - 10 minutes.

After a fresh build (cache regenerated) docfx took only about 30 minutes. Don't know what caused the first run to take so long. Still a little bit too long for my taste for ~100 projects with a total of ~7000 .cs files.

After digging into the code base a little bit. I guess the combination of parsing the Roslyn projects and caching all the metadata to disk with a YAML serializer can take a lot of time for larger solutions.

[paulushub]

@sandrofigo Thanks for the information.

caching all the metadata to disk with a YAML serializer can take a lot of time for larger solutions.

Right, but the "several hours to complete" will make it unusable. Being a tool used by docs.microsoft.com, it cannot be that slow in my imaginations!