Add tests for animation functionality

[caleb531]

@dsr20030703 @eliotwrobson The animation functionality added in #252 currently only gets 24% code coverage. This doesn't impact the overall package coverage much (still at 95%), but I would prefer to increase coverage here. What would it take to write some tests to increase that coverage?

----------------------------------------------------------------------
Ran 434 tests in 8.712s

OK
Name                            Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------------------
...
automata/fa/animation.py          110     77     34      1    24%
...
-----------------------------------------------------------------
TOTAL                            2947    132   1310     21    95%

[dsr20030703]

Ahh, I've mentioned it at the beginning of that PR. But seems that all of us forgot that since the coverage test passed for some reason.

Another problem I have encountered is that I don't know how to add the tests. As manim is an animation engine, they have developed a unique test framework based on pytest, which is different from this project. So I haven't added any test. (But at least the examples in the Jupyter Notebook all works, except the last of which)

The problem is that, since what we've used for test is unittest instead of pytest, the Graphical Unit Tests and Video Format Tests seem to be not available for us. Unit Tests can be done with current unittest though, but I have no idea how to let unit tests test out the visually problematic scenes, like the last animation in the fa_animation.ipynb jupyter notebook.

[caleb531]

@dsr20030703 I see, thank you for clarifying the current constraints!

@eliotwrobson What do you think? Is it maybe worth migrating automata's test suite to pytest? Certainly, such an undertaking would be, at minimum, a modest amount of effort. But maybe long-term, it would be worth it, especially since I'm reading that nose2 is effectively in maintenance mode (compared to pytest, which is still actively developed).

[eliotwrobson]

@caleb531 I think this is probably worth it. Pytest will still run tests that are set up using unittest, so the invocation / dependencies can be changed and then test style can gradually be changed later.

[caleb531]

@eliotwrobson Okay, great! I am already converting one of my other projects to use pytest this afternoon, and once that is done and successful, I will try converting automata to use pytest.

@dsr20030703 Does that work for you? Anything I should note for the new pytest setup in order for the Graphical Unit Tests and Video Format Tests to be properly available/integrate-able?

[caleb531]

@dsr20030703 Okay, we've migrated from nose2 to pytest as the test runner for this project (develop branch). Are you able to integrate the Graphical Unit Tests and Video Format Tests?

[dsr20030703]

I've enabled our repository to use Graphical Unit Tests of manim in the animation branch of my repository, now we can use frames comparison provided by manim, and do Graphical Unit Tests in the way manim test themselves like this (suppose has switched to animation branch of my repository):

cd tests
mkdir control_data
pytest test_animation.py --set_test -s # then there would be 2 npz file in control_data/animation directory
# now we can do the Graphical Unit Test by testing the frame!
pytest test_animation.py -s --show_diff # --show_diff option requires matplotlib installed
# the npz files can be seen with following
python extract_frames.py control_data/animation/dfa_graph.npz # only one npz allowed
# then there would be one png file in output directory

But there cames problems:

1. The graphs generated by Graphviz (also pygraphviz) are NOT DETERMINISTIC, i.e. the same DFA could generate different graphs, as shown in following (generated by pytest test_animation.py -s --show_diff above)

Therefore, it may come up that one test failed, but running it again passed. I made the generated graph of DFA and NFA cached in the code submitted this spring to solve this problem. But I have no idea how to solve it in the test -- after all, it is the generated graph that we are testing.

2. For some reason, the texts generated by manim seem to have differerent fonts on Windows and Linux.

The DFA graph generated on Windows looks like following:

The DFA graph generated on Linux (Ubuntu WSL) looks like following:

(I don't have a Mac so I don't know what will it be on macOS)

So it will be bound to fail the test if one is testing on a different OS from the OS npz file generated on.

Since frames are different, videos must be different as well. I have no idea how to do Graphical Unit Tests and Video Format Tests now. 😮‍💨

[caleb531]

@dsr20030703 Interesting, thanks for the detailed insight! I am willing to bet the out-of-order states in the visualizations is due to the unordered nature of sets in Python.

The only true solution I see is to use the stdlib's collections.OrderedDict class to create an OrderedSet class that we can use for constructing DFA that will produce deterministic visualizations, as well (probably the better solution).

As for the font issue, can we force the use of a font family that is available on all platforms? (like Times New Roman or Arial)?

@eliotwrobson Do you have any thoughts on how we could address these?

[dsr20030703]

As for the font issue, can we force the use of a font family that is available on all platforms? (like Times New Roman or Arial)?

Well, we do can set a different font for manim.Text object. However, as the font must be installed in the system and Pango must know about it, I fount that the fonts installed on WSL are mearge. The fonts that are supported both Windows and WSL on my machine are: Serif, System-ui, Sans and Monospace.

All of them are actually different fonts on Windows and WSL. Probably we need to install font when testing, as well as CI.

[eliotwrobson]

@caleb531 I don't have detailed knowledge of how tests in manim work, but I think having a frame-by-frame comparison is too strict. The tests for pygraphviz are not this strict (they just check that the internal representation is arranged as expected).