Package Extensions for tests

[schnellecom]

As discussed with @ThomasBreuer on GAPDays 25 Summer: It would be great, if there was an option to also include certain test files in the extensions record. These would then only run, if the package is loaded, and also be respected by e.g. testall.tst. If somebody has a good suggestion how to implement this, I would be happy to try implementing it.

[ThomasBreuer]

As discussed with @schnellecom: For packages using a TestDirectory call in tst/testall.g, one can proceed as follows.

• For each set of suggested packages, put those tests into a separate testfile that make sense only if these packages are loaded.
• Before calling TestDirectory, compute which of the suggested packages are currently loaded, and deduce which testfiles shall currently not be run.
• Use the exclude component of the second argument of TestDirectory to omit the testfiles in question.

The first of these steps is part of the package setup.

The other steps could be done automatically by TestDirectory, provided the information about testfiles to be excluded is available somewhere.

For example, we could support a component testfiles in the declarations of package extensions, whose value is a list of those filenames relative to the tst directory of the package which shall be processed only if the packages listed in the needed component are loaded.

Then it would suffice to exclude exactly those testfiles that are currently listed in the pending extensions of the package in question.

It would be possible to have an extension record for a combinations of packages for which a testfiles component but no filename component is needed.

(Would it make sense to support also the situation that certain tests shall be executed only if certain other packages are not loaded? Then one could check for different behaviour depending on which suggested packages are loaded, but perhaps this would be too exotic.)

[limakzi]

@schnellecom @ThomasBreuer Let me understand.

• You want to load suggested packages for specific test cases only?
• Give the end-user possibility to run each testfile separately?
• ...

Whats our main goal? :)

[ThomasBreuer]

@limakzi No.

The idea is as follows.

A package can have suggested packages, which may or may not be loaded in the current GAP session. Typically, additional code from a package gets read as soon as loading suggested packages has finished --the recommended mechanism for that is "package extensions"-- and then additional functionality becomes available.

It is reasonable to provide tests for the code from such extensions. These tests make sense only if the suggested packages are loaded. The current issue asks for a way to let GAP decide automatically at runtime which tests can be run, depending on the currently loaded suggested packages.

[limakzi]

@ThomasBreuer Understood.

For example, we could support a component testfiles in the declarations of package extensions, whose value is a list of those filenames relative to the tst directory of the package which shall be processed only if the packages listed in the needed component are loaded.

It would be possible to have an extension record for a combinations of packages for which a testfiles component but no filename component is needed.

I like this idea. Do you have any concrete minimum valuable product in mind? Maybe keep testall.g, but change its structure to list of records with included testifies?

[ThomasBreuer]

@limakzi

I had tried to sketch a possible implementation in my first comment above:

• In each package that wants to use the feature, list the testfiles that need a given package extension in the record in PackageInfo,g that defines this extension.
• Change GAP's TestDirectory to use this information.

[limakzi]

@ThomasBreuer I'm just wondering how much backwards compatibility do we want to keep. It feels like a breaking change for packages.

[schnellecom]

@limakzi wouldn't we be able to keep the backwards compatibility by only excluding the tests for the packages in the list of extension packages? Currently for passing the tests the packages should be able to function anyways especially in a minimal configuration, so I would suspect that for the packages using extensions they do not provide tests for this functionality yet. iirc this is how I stumbled onto this problem.

[ThomasBreuer]

@limakzi Why should such a change be breaking? If no testfiles are explicitly regarded as depending on certain package extensions (which is the current behaviour and the default), TestDirectory runs all tests.