Verify performance of Table of Contents on Index Pages

[aj-stein-nist]

User Story

As an OSCAL tool developer, in order to efficiently review all locations of a given OSCAL syntax element in one or more models with the index pages, I want assurance the pages load quickly without hangs or lagging (with the table of contents disabled enabled only if it is efficient).

Goals

• [ ] Analyze table of contents performance on index pages
• [ ] Customize tocbot defaults or disable on relevant index page configuration and generation pipeline

Dependencies

No response

Acceptance Criteria

• [ ] All OSCAL website and readme documentation affected by the changes in this issue have been updated. Changes to the OSCAL website can be made in the docs/content directory of your branch.
• [ ] A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.
• [ ] The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.

[david-waltermire]

In July, @aj-stein-nist and I researched this by profiling some large pages. It is clear that tocbot, which drives the right navigation, doesn't perform well for large pages. The only the model reference pages exhibit this problem due to their large size. This includes the index, reference, and definition pages for XML and JSON/YAML.

We have a plan for addressing the reference pages, where we will disable the right nav (#1372) and use the outline pages as an index providing better linking between the reference page and the outline (#1371).

It's not clear yet how to address navigation for the index documentation pages. Perhaps an alphabetical listing at the top of the page?

[aj-stein-nist]

@nikitawootten-nist and @wendellpiez, do we need to work together on doing the performance analysis on the site? We can discuss what I did casually in pairing sessions or coming up with something a little more rigorous, let me know. Thanks.

[wendellpiez]

I am happy with either a formal or an informal analysis. Indeed I don't wish to be the one making the complaints such as one helping to facilitate solutions.

Can we think of this in stages, starting with removing the 'all' schema pages? I am probably getting ahead of us. But in a later phase we could restore 'all' pages or indeed create new sleek 'all' indexes that would properly point to the model pages, rather than to other heavyweight 'all' pages.

What does our analysis need to show beyond that the 'all' pages are especially egregiously bad? I am in favor of rework and staging it but there are also more expedient mitigations possible, I think.

Not voting against a shared session and spiking on the analysis together btw -- on the contrary! Aware there are also other improvements that can and should be made.

[nikitawootten-nist]

@wendellpiez @aj-stein-nist I sent you both a calendar invite for Wednesday to discuss.

[aj-stein-nist]

Nikita, Wendell, and I met today more thoroughly reviewed Lighthouse processing for OSCAL JSON 1.x reference pages (not, the outline or index). We took notes in this HackMD.

More to follow.

[nikitawootten-nist]

Next steps:

@wendellpiez and I are going to:

• Adjust the doc generation pipeline to exclude the "Switch to ..." button and collapse headings by default (@wendellpiez)
• Set up lighhouse's cli for repeatable tests (@nikitawootten-nist)

On Friday we'll touch base and attempt to run Lighhouse on our 4 scenarios:

• Site as it is today
• Site without buttons
• Site with collapsed headings
• Site with no buttons and collapsed headings

If lighthouse shows a favorable result, we can consider the issue of verifiable performance resolved (with follow on issues to be made to fix specific findings)!

[aj-stein-nist]

Thank you both. Glad to see forward progress over things, let me know if you need help and assistance. Do let the team know if we need to adjust, swap out, or remote related website perf issues included in this sprint.

[aj-stein-nist]

Also for AJ to call out and think about: we need to determine how we track these changes that will end up in the Metaschema repo.

[nikitawootten-nist]

@wendellpiez until I have a more permanent home for this script in the OSCAL repo, refer to the script we'll be using to test here: https://gist.github.com/nikitawootten-nist/76dfdb45ad68f7d49c396780c90dbf63

[aj-stein-nist]

Next steps:

@wendellpiez and I are going to:

* Adjust the doc generation pipeline to exclude the "Switch to ..." button and collapse headings by default (@wendellpiez)

* Set up lighhouse's cli for repeatable tests (@nikitawootten-nist)

On Friday we'll touch base and attempt to run Lighhouse on our 4 scenarios:

* Site as it is today

* Site without buttons

* Site with collapsed headings

* Site with no buttons and collapsed headings

If lighthouse shows a favorable result, we can consider the issue of verifiable performance resolved (with follow on issues to be made to fix specific findings)!

@nikitawootten-nist will continue with Wendell. They will schedule time next week.

[nikitawootten-nist]

As an update, I realize that the work on this issue may have reached far afield of the original intention of the issue, so I'd like to provide a simple conclusion to this issue so that we can more efficiently plan next steps.

• The "Table of Contents" (provided through tocbot) has a significant performance penalty on larger pages (like the complete reference pages)

• Lighthouse scores for the complete json reference jump from 5/100 to 11/100 when toc is set to false.

  • This is also confirmed in my subjective experience.

• Lighthouse testing did not show a huge benefit to the other performance improvements that we floated (collapsing headings + removing buttons), but collapsing headings specifically led to a huge subjective increase in performance. Maybe a good way forward for complete pages is to disable the toc while also collapsing the models (to make the site more navigable in lieu of a table of contents).

• Lastly, none of these suggestions improve the time to interactive metric to an acceptable value. The likely culprit for this is the script that executes in the background adding anchor tags to all headings.

  • As a note, disabling the table of contents improved the time to interactive from 19 to 12 seconds on the JSON reference (good but definitely not acceptable)

[nikitawootten-nist]

Another note, the removal of the "Switch to XML/JSON" buttons did not yield any meaningful performance gains, signalling that #1369 may not be a meaningful next step

[aj-stein-nist]

I met with Nikita and Wendell and they agreed that the analysis is complete. I will move this back to Sprint 63 as it is closeable and work on the actual analysis is not needed in Sprint 64. It's more or less done.

They need to add issues to track static generation of anchors (from XSLT run by generate-model-documentation.sh) and not dynamically in JS and a second more-important spike issue on designing the index pages for crucial performance gains.

[aj-stein-nist]

Thank you for creating the follow-on issues ☝️ as requested. Thanks to both of you for your work, I will mark this issue as closed for now.