Performance degradation with increasing use of Autograder

[Aulud]

[Bug] Performance degradation with increasing use of Autograder

UX issue and potential bug.

Bug description

The front-end Autograder (in the side content tabs) allows students to test the correctness of their programs with a set of provided testcases.

However, starting from PR #833 (which extended the Autograder to allow it to run grader code in an elevated context), increasing use of the Autograder on the same question results in significant performance degradation. Each additional testcase run takes increasingly longer to complete, resulting in the browser slowing to a sluggish pace and eventually locking up.

Performance and memory use was inspected with the inbuilt profiler of the browser, with a testcase run about every 2 seconds for the first 30 seconds. The resultant graphs below shows that both the duration of the browser freezes and heap memory in use grows progressively with each additional testcase run.

At around the 30 second mark, the program in the editor was evaluated again. Whilst this froze the browser for about 1.5 seconds, it (interestingly) reset the length of the browser freezes for each testcase run back to the baseline. This is a temporary workaround for keeping the length of the browser freezes manageable whilst the Autograder is in use.

However, this does not resolve the large memory consumption, and it continues to grow as more testcases are run after the editor is re-evaluated (the final heap memory in use was 664 MB).

https://github.com/source-academy/cadet-frontend/blob/121f817552b598abd9dc5749e789df26664c9d8a/src/commons/sagas/WorkspaceSaga.ts#L300-L322

Brief investigation suggests the issue stems from the privileged context created to allow grader code to always run in Source 4, whilst limiting student programs to run in the Source sublanguage configured for that assessment. This hypothesis seems to be supported by the fact that commenting out the lines above resolves both the performance and memory issue as shown in the graphs below. Note that this effectively disables the privileged context, requiring elevatedContext in L325 to be replaced with context. The peak heap memory in use only reached 35 MB.

This issue happens in both the assessment and grading workspaces, so it will affect both students and staff both. It happens both in Chrome and Firefox.

Replication instructions

1. Navigate to any assessment or submission, then proceed to any programming (not MCQ) question
2. Bring up the side-content Autograder (airplane icon), and evaluate the editor's contents
3. Run multiple testcases individually (by clicking on the testcase card) in succession - you may opt to open the browser's performance profiler via F12 > Performance

Last updated 17 Jul 2020, 2:00AM