Reports with Lots of Data Load Very Slowly

[bbertucc]

Problem

Displaying live data limits filtering available to users with large amounts of URLs, and results in very slow loading of reports with lots of data. Users also don't use the live data, since they only check on issues once in a while.

Solution

We are going to step through optimizations, trying things as we go.

First Optimization

• [ ] Making a smaller dictionary of messages and tags, so that users are sharing data instead of having their own. That should speed up inserts and queries.

We'll bounty that one task @heythisischris for now, then check in after you do that. Please add a budget here.

Future Optimization Ideas

A "Generate Report" feature would queue report generation, then disable results of generated report when user accesses content via API or Dashboard.

Dashboard User Experience Changes

• Add Timestamp of last generated and "Update" button to report page. Regenerate would be conditionally disabled with the value "Updating.." if a report is being regenerated.
• When report is generating, add notice "Report is updating" in area for timestamp and disable "Update" Button. That notice will be replaced with "Updated: dd/mm/yy hh:mm CST" and the "Update" button will become active. This would be displayed after filters are updated, report is created for the first time, or "Generate" button is clicked.

Accessibility Notes

• Use toast notification to announce when a report has been updated
• Use toast notification to announce when a report has started updated
• Use toast notification to announce if a report update has failed.

Architecture Notes

• Generating reports should be processed in a queue based system, like BullMQ (we have a pro license), so we can fairly prioritize generation to clients. IE- If one client pushes two reports, then another pushes one report we should prioritize the other clients first report over the one client's second
• We should build this to easily scale workers doing this work. IE- If we need to generate more reports in parallel we'll be able to do this just by spinning up more workers for the task. This helps us scale.

API Changes

• /get/results?reportId=... returns generated report data
• get/charts?reportId=... returns generated report data
• update/reports triggers generation of report
• add/reports triggers generation of report