Tests do not stop if certain tests fail and continue running indefinitely* - Inefficient testing

[gerteck]

Please confirm that you have searched existing issues in the repo

Yes, I have searched the existing issues

Any related issues?

https://github.com/MarkBind/markbind/issues/1637 - Test Site Improvements https://github.com/MarkBind/markbind/issues/2631 - Modularize Snapshot Tests

Tell us about your environment

Win 11

MarkBind version

5.3.3

Describe the bug and the steps to reproduce it

See action here: https://github.com/MarkBind/markbind/actions/runs/13703858478/job/38324234697

Noticed it on my local computer as well. Need to ctrl-c locally if tests fail to end. Only happens to certain tests failing.

Actions ran were only ultimately cancelled because a new commit was pushed over it. ( Members by default do not have access writes to cancel workflows).

Think it is for packages/cli tests, as they were the only ones that did not run to completion. Unsure if it is due running diff between expected site files and generated files that takes very long, which then it would actually stop, but just take very long.

Expected behavior

No response

Anything else?

No response

[gerteck]

Some investigation on this:

I think I possibly made a wrong diagnosis on this - instead of not stopping, it is taking too long to load the diff between expected files and rendered html in the functional tests for the cli package tests.

This is related to #1637 #2631

Two root causes -

1. each change causes too many unrelated file changes
2. render functions diff also checked

This adds too much overhead such that the test does not finish in time. (even after 360 minutes - then auto aborted)

For example, the render function files are not ignored. An example of the render function file is:

https://github.com/MarkBind/markbind/blob/8c9de42d78a67e4e73d8ed06c39d741ed71ddeb2/packages/cli/test/functional/test_site/expected/index.page-vue-render.js#L2-L20

The snapshot tests also check for differences in the page render function - which is unnecessary and adds considerable overhead.

I will add additional explanation under #1637

[tlylt]

This reminds me of https://github.com/MarkBind/markbind/pull/1645 ...