plk
Set up a full set of tests
As discussed in #372, a full set of tests are needed. I'll look at this: first making sure we can run things appropriately, then looking for assistance. If necessary I'll adjust l3build to make this possible.
The newest biblatex broke also the biblatex code of tex4ht (http://puszcza.gnu.org.ua/bugs/?273). Imho if would be good to add also some test code for tex4ht to the test suite. Michal could perhaps help to make it.
I'm going to look at the code merge first then sort this: give me a little while (say until Easter)!
No rush. Likely 3.4 will be out before then and there is one large-ish change to prefixnumber handling to address some user issues and limitations (see CHANGES file which I will make much more prominent in the wake of 3.3 …)
PK
On 12 Mar 2016, at 10:49 am, Joseph Wright [email protected] wrote:
I'm going to look at the code merge first then sort this: give me a little while (say until Easter)!
— Reply to this email directly or view it on GitHub.
Dr Philip Kime
Take your time. I only added the comment because I say the tex4ht message and made the connection and didn't want to forget it.
Yes, tex4ht support generally breaks with every release, because it redefines lot of internal macros (of biblatex and of LaTeX itself). It got little bit messy because we need to keep support also for older biblatex versions.
One reason why it is hard for us to hack in the current version, is some stuff which is called in \AtEndPreamble in biblatex_.sty, namely patching of \MakeUppercase and \MakeLowercase commands, which fails because they are patched by tex4ht as well.
I can hopefully help with testing the tex4ht output.
If we can help at all by putting code into biblatex to ease the tex4ht releases, I'm all for it - tex4ht is an important package for a lot of academics who have to submit in other formats and biblatex is not as journal friendly as bibtex.
Just so you know @josephwright - there is a meagre start to this already in the build subdir of the biblatex dist. I had implemented some tests for the range macros as a start. I have a relatively clean set of regression test docs in the doc/latex/biblatex/examples dir which is used as the reference set for the very picky testfull.pl perl script in the obuild directory. obuild is "old build" as I envisaged the build directory as the new build system, based on l3build. However, obuild is currently used to do everything and works quite well. I now review any differences that testfull.pl finds using a low-level PDF diff viewer and this will stop most regression problems going forward until there is a proper functional test-suite.
Just out of curiosity. Is the test suite for biblatex working? Is it accessible or can it be modified to test styles as well?
Well, what happens currently is that I compile all of the test files in the examples/ folder (the obuild/build.sh script does this automatically but to use it you might have to change it to your environment). Then, the testoutput verb of the same build script compares all of the output PDF with a reference set (using the Perl CAMPDF module). If there are differences, I use a visual PDF comparison program to check the details. There is a build directory which is the beginning of a proper test suite using the latex3 test functionality but this hasn't progressed much due to limitations in the l3build module. @josephwright was looking at this but not much is happening in that direction currently. It will be a lot of effort to make a full set of functional tests. The test suite I have now is essentially an output test, not a functional test.
Thank you for the quick reply. I will look into your folders and hope for some progress with l3build.
I'm back looking at this: we've been doing a reasonable amount of work to make l3build work with more complex set ups. In particular, I can now cover the fact that biblatex isn't 'flat' (though the version string set up is a bit tricky). I'll likely start a new branch to fiddle about with stuff, which will probably need a burning-edge l3build to work with (I'll be adjusting l3build in parallel to get the necessary features).
At the moment I'm thinking of an approach similar to the one used by @plk but using log-based comparison (so using the existing examples). Better long-term would be I guess to have a way of redirecting bibmacro output to the log. That will of course mean 'hitting' all of the right places: one for the future!
I should also add that depending on how things go, I'll also see if I can get l3build to produce a CTAN-ready product. I've done that for beamer, so it should be doable. Much of the work has to be done anyway: it's minor stuff that I'll need to think about.
I could do with a feeling about whether tests have to cover all engines and whether both backends are needed. Multiple engines are easy, multiple back-ends less so: probably that would go with using two set-ups. I think it's all doable, but perhaps not for 3.11 (depending on timing).
I got a reminder about this: I really do mean to sort out l3build to support it.I'll see if I can find the time for the Lua adjustments next month.
@josephwright Cool. I actually have l3build running for biblatex-ext already (https://github.com/moewew/biblatex-ext), but I'm not entirely happy with my setup. The main problem is that the only way I could find to test biblatex properly is via boxed content, which can get extremely long for comprehensive tests. (It's a while ago, but I think when I first tried it, PDF comparison would give too many false positives, possibly because of differences in fonts, and the failed test would give me no indication where to look anyway.) Having to run LaTeX, Biber, LaTeX, LaTeX means that even my relatively modest test suite for biblatex-ext takes quite a while to complete (the time on Travis is acceptable, but when I run it locally on my virtual machine it takes about three times as long).
