Improve Regression Test Suite

[martanne]

For some time there exists basic testing infrastructure on a separate branch. Unfortunately it currently lacks proper test cases.

As mentioned elsewhere having a proper test suite would be extremely helpful to guarantee stability and make sure fixed bugs are not reintroduced. Ideally every bugfix / newly introduced feature would have an accompanying test suite entry. The test suite should at all times pass without failure.

Also working on the test suite is a good way to contribute to the project without necessarily having to delve into the C code base.

There currently exists 3 different kind of tests (each with a README in the corresponding sub directory):

1. test/vim these are simple black box tests, comparing vim compatibility. They are completely independent of vis and could possibly be reused by other vim-like editors. The idea is to start the editor with a given file, feed it keyboard input as would normally be done by a human and compare the resulting file content to the one produced by vim.
2. test/vis these are similar to 1) but test vis features which are either not present in vim core (e.g. multiple cursors) or have deliberately different behavior. Instead of comparing the final file content to the one produced by vim it is compared to a fixed expected result file.
3. test/core these are C unit tests for core data structures used in vis. Writing tests for those is a good way to become familiar with the core APIs of vis.

For types 1 and 2 a good way to start would be to go through all operators / motions and make sure they:

• work when given a count
• are repeatable with . as expected
• (apply to all cursors, only for type 2)

This approach would ensure that tests are nicely categorized by functionality.

Another good starting point would be to go through he commit log and past closed issues (like #160 #152 #149 #144 #141 #131 #123 #118 #113 #92 #91 #89 #45 #44) and add test cases for those.

Adding some vimgolf entries (not necessarily their solution) might also be an option, but remember the goal is to test existing editor functionality.

[martanne]

If we have named folders per test, could we just name the tests test.in test.keys?

Only one test per folder doesn't seem that useful. Instead the folders should be used to categorize the test by "topic" e.g.

• operators
  • delete
  • change
  • ...
• motions
  • ...
• text objects
  • ...
• modes
  • ...

Running tests are pretty slow now. Is it possible to speed them up?

The keys utility used to translate the symbolic keys to something the terminal understands has a delay for certain keys <Escape> in particular. It might be possible to lower this delay.

It might be desirable to speed up vim by making sure no plugins and stuff is loaded i.e. use vim -u NONE in the test script.

For development purposes it is possible to run only one test

$ ./test.sh delete/delete

Would it make sense to have a test.ref for vim as well?

I'm not yet sure. This would also speed up the tests (vim would not be needed). However the drawback is that when you change the test you will also have to update the reference. Whereas at the moment this is automatic.

Also i suggest to make one output line per test.

Why? Maybe I do not understand what you suggest ...

Would be nice to be able to run just one test by typing make sometest.

This is already possible see above, give the name of the test to the test.sh script.

[martanne]

I added a new vim/golf directory with a few vimgolf challenges. However remember that the idea is not to be bug-for-bug compatible with vim. Instead a similar user interface / editing paradigm should be provided. For now use only features available in vis.

In general when debugging test cases it is useful to look at the intermediate output that the editor will get. I.e.

cat test.keys | cpp -P

and

cat test.keys | cpp -P 2>/dev/null | keys > log && vis log

[martanne]

A sam sub directory with tests for the structural regular expression based command language, either comparing to expected reference files or output produced by ssam(1) would also be useful ...

[joshaw]

Do you mind if I ask why the tests are on a separate branch? I'm having difficulty getting the existing tests to run -- I think it may be because the build is different (wrt termkey main makefile vs util/keys).

[martanne]

@joshaw they are on a separate branch because:

• the testing infrastructure is still relatively new and somewhat under flux
• some of the test are of dubious quality
• parts of the tests might also be useful for other vim like text editors (unlikely, but still)
• I do not really want to clutter the source tree

However I understand that it would encourage people to actually run the tests if they where included in master. Maybe they should be moved to a separate git repository and then included as a git sub module.

Besides the things mentioned in an earlier comment, one way to speed up the tests is to also store *.ref files for the vim related tests. Or at least to only run vim once and then keep the output as a reference solution.

[martanne]

I moved the tests to a separate repository. If you still have some fundamental issues running them I would encourage you to join the IRC channel, because it is easier to debug those in real time.

[mcepl]

Test suite exists, what remains in this ticket to deserve to be open?