Biblatex fails due to version mismatch

[MTRNord]

tectonic fails to build with the 2020.0r0 bundle and biblatex:

Running external tool biber ...
error: the external tool exited with an error code; its stdout was:

===============================================================================
INFO - This is Biber 2.17
INFO - Logfile is 'default.blg'
INFO - Reading 'default.bcf'
ERROR - Error: Found biblatex control file version 3.7, expected version 3.8.
This means that your biber (2.17) and biblatex (3.14) versions are incompatible.
See compat matrix in biblatex or biber PDF documentation.
INFO - ERRORS: 1
===============================================================================
error: its stderr was:

===============================================================================
===============================================================================
error: the external tool exited with error code 2

any ideas how I can fix that?

[pkgw]

My guess is that the best fix will be to upgrade to the soon-to-be-released next version of Tectonic that will bump us up to TeXLive 2021.3! I will try to get it done today.

[MTRNord]

Hm I tried now with version 0.9.0 of tectonic and the newer bundle, and it seems that suffers from a different but essentially the same issue:

RROR - Error: Found biblatex control file version 3.7, expected version 3.8.
This means that your biber (2.17) and biblatex (3.16) versions are incompatible.
See compat matrix in biblatex or biber PDF documentation.
INFO - ERRORS: 1
===============================================================================
error: its stderr was:

===============================================================================
===============================================================================
error: the external tool exited with error code 2
[Finished running. Exit status: 0]

biblatex is now slightly newer, but the error still happens. Biber I guess is from my MikTex install, which should be up-to-date.

[KevoSoftworks]

I'm running into the same issue as well. A workaround to this problem is to downgrade biber to version 2.16.

[pkgw]

OK, well, the upgrade to TeXLive 2022.0 should happen much faster since the diffs are comparatively small and lots of good infrastructure is now in place, and maybe that will make biber 2.17 happy ...

[doronbehar]

I'm still experiencing this issue now, with tectonic 0.9.0 :sob:. Is there a --web-bundle url I can give to it that will fix it? Please :pray:.

[charmoniumQ]

@doronbehar My work around is to build the current master branch of tectonic from source. The easiest way to do that is a Nix flake (which also takes care of dependencies). Note that this version of Tectonic works with a different (older?) bundle in Tectonic.toml. If you see error: expl3-code.tex:22489: Invalid code (13), should be in the ranges 1..4, 6..8, 10..12, that is a problem with the bundle. I found the right bundle by using tectonic -X new.

tldr:

$ url=https://www.toptal.com/developers/hastebin/tulajibike.properties
$ curl $url > flake.nix
$ nix develop --command tectonic -X new test
$ diff test/Tectonic.toml Tectonic.toml
$ nix develop --command tectonic -X build

[WtfJoke]

Downgrading biber to 2.16 worked for me (with tectonic 0.9.0)

[Neved4]

As of currently, tectonic 0.11.0 now works with biber 2.17, but gives the same problem with last biber 2.18.

[Neved4]

For people who don't need to use TeX Live alongside tectonic, dropping the first and installing biber separately is easier than juggling the the two installs in the same system.

Under macOS brew's biber version is currently 2.17 one can simply:

brew install tectonic
brew install biber

To get a working tectonic/biblatex setup.

[MTRNord]

As of currently, tectonic 0.11.0 now works with biber 2.17, but gives the same problem with last biber 2.18.

Previously tlextras-2021.3r1.tar as bundle fixed it for me, so I wonder is there a newer bundle that we can try to work around this by any chance? (turns out it is incredibly difficult to find bundles)

Edit oops tlextras-2022.0r0.tar is merged but that has another conflict :) So I mean newer than that

[pkgw]

@MTRNord Do you mean that there is an even newer version of biber that requires a newer version of the biblatex package? I don't anticipate upgrading Tectonic's built-in biblatex until TeXLive 2022.1 comes out, but if there's an incompatibility it will probably be worth taking extra steps. (It would be nice if biber/biblatex didn't keep breaking compatibility, though!)

I'm sorry that the bundle infrastructure has not been well-documented. For what it's worth, with the current setup, this file should start accumulating a helpful list of bundle URLs going forward ...

[Neved4]

@pkgw I confirm: there's an even newer version of biber: 2.18 that requires a newer version of biblatex. We're good on the next to last one, see: https://github.com/tectonic-typesetting/tectonic/issues/893#issuecomment-1270156377. I can reproduce and share a test file if needed.

[pkgw]

Thanks for confirming @mnrvwl . I'm not familiar with biber/biblatex (as is probably quite clear by now). Does anyone know what the maintainers of those tools generally recommend that people do in order to keep things compatible? Unless I'm missing something, the latest release of TeXLive is 2022.0 has the older biblatex that will be incompatible with this new release. It seems like that would be a problem for a potentially large number of users. Do they tell people not to upgrade biber until the new TeXLive release comes out? Do they expect people to upgrade their biblatex package out-of-band with regards to the TeXLive releases?

[Neved4]

@pkgw You make great questions!

TeX Live 2022 has biber 2.17 and biblatex 3.17: the version numbers match each other, so a minor version would be compatible with the equivalent one. When updating with tlmgr, we get a bump to biber 2.18 and biblatex 3.18, working just fine.

Both are developed together, so there shouldn't be issues updating before the next release comes out. This makes me very inclined to the idea of tectonic bundling it in the future.

Below are some ideas better suited for Discussions.

---

Packages and updates

I personally do update TeX Live on a weekly basis, I work with a fixed set of ≈100 packages. I also know some people who just stick to $LASTYEAR or different versions that publishing $PLATFORM mandates, for reproducibility.

If you live on the edge like me, you'll notice a couple things. In the first weeks after a release, one sees that around ≈ 200 packages get updated, a significant amount of the recurring ones. The updates are mostly patches and fixes for incompatibilities that arise when newer versions of said packages are used together. While there's a pre-release for TeX Live and they call for testing help, you still see most get patched after the release when things hit production.

I amusingly call this TeX Live 2022.1: when you update one month after the release and freeze after to benefit both from the latest packages and the fixes, without keeping the bugs for a year longer. I believe is one of the most robust approaches that takes into account how much or little pre-releases are tested. It also goes without saying, there's not such an official version. It'd be nice to have that granularity, yet I understand how it could add complexity and why the team wants to keep it yearly.

As of 2022-10-28, installing TeX Live 2022 and then running the usual tlmgr update --self --all we get a whopping of ≈ 668 packages that get updated (using basictex only would be ≈ 59).

Scenarios

As of today, few are presented to users:

• Run latest TeX Live $YEAR, with frozen versions.

  This is nice when tectonic bundle version is equal to TeX Live's. It fails when $TECVER < $LIVEVER.

• The user runs TeX Live $YEAR, with latest updates.

  The updates can be variable, both in time: daily, weekly, monthly; and scope: update all indiscriminately vs update specific ones systematically. It fails when $TECVER < $BIBERVER.

To deal with the situation, one can either:

• Ditch TeX Live and just run tectonic, install biber with a package manager.

• Manually manage the installation of tectonic and biber separately from TeX Live.

  This means calling tectonic with a different path that points first to the compatible version of biber —installed through a package manager— before TeX Live's one, avoiding running into incompatibilities.

A future possibility comes to mind:

• Bundle biber with tectonic, keeping different versions for each bundle.

  Already mentioned in https://github.com/tectonic-typesetting/tectonic/issues/35, this is the best way to guide, control and define a policy: leading on what works for tectonic and what doesn't. I believe it's an option that maximizes reproducible builds too.

Summary

The problem mostly lies with how we distribute biber, the fact that some people pull it from TeX Live instead of managing it separately, together with some people keeping TeX Live up to date, making it incompatible with tectonic's bundle and with tectonic's not defining which biber version ought to work, bundling its own biber, having a way to update biblatex source from TeX Live to match a newer biber if present in the system, warning/guiding the user more directly about the matter at hand before compiling —detecting what's in the system— and what to do with it... etc.

Whenever the installed biber version, regardless of the means of installation, is above tectonic's bundled (biblatex) one, we'll get this incompatibility. If we can find a way, through package managers, bundling, etc.; to roll a fixed biber version for a particular bundle and make sure that tectonic solely points to that by default we can avoid relying on people updating their TeX Live or having newer versions that tectonic supports.

[Neved4]

@pkg Additional info can be found under the documentation search for the "compatibility matrix" table.

biber releases lag biblatex ones, but if you match both minor versions you're good to go.

The suggestion to have the version numbers be normalised in sync was also made here: https://github.com/plk/biber/issues/427.

[pkgw]

Thanks for the detailed rundown! I guess my initial feeling is that we should at least make sure that people have a straightforward recipe to set themselves up with an updated biblatex.sty if they install a biber newer than the one compatible with the latest bundle. This would be a good topic for the Howto section of the book.

[pkgw]

Some broader commentary: the issue here is a subset of the quip that "all software evolves to eventually include a package manager". While I would like Tectonic to avoid external dependencies as much as possible, there's a limit to how far you can take that — people have real-life workflows that depend on biber, minted, image-processing tools, and so on.

I'd really like to make it so that if people are going to use these tools, we have a nice clean way for them to get a setup that "just works" with tested version combinations that work in any environment. But, I really don't want to implement a half-assed package manager in Tectonic! I've been interested in software packaging for decades at this point and one thing that I've learned is that it's a lot harder than some people think it is. For instance, to package biber, suddenly you have to package Perl and whatever chunk of that ecosystem biber needs to run. Recreating all of that is a road I do not want to go down.

In that context, I think the way to get something reliable would be to build on an existing package manager, perhaps through some kind of specialized "Tectonic Environment" tool/product. Conda comes to mind, since it's (relatively) cross-platform, doesn't require admin permissions, and can demonstrably handle deep Perl and Python stacks. (I'm also a core member of the conda-forge project and maintain many many packages in that system.) There are a lot of things about it that I wish were better, but it meets a lot of the criteria I'd be looking for.

A wrinkle is that I foresee that Tectonic will need to integrate well with at least one other package management system: the NPM/Yarn ecosystem. In my work on tt-weave, it became clear to me that to deliver nice HTML output you need to leverage the whole web development ecosystem, which means using NPM or Yarn to install a deep dependency tree and create an output bundle. We're not going to be providing biber through NPM any time soon, so this ecosystem isn't sufficient to cover everyone's needs for building documents, but I think we are going to need to integrate with it well, somehow.

(I have had the devious idea that maybe we can (ab)use the NPM system as a TeX package manager to fulfill the longstanding need for a way for people to update packages between bundle releases. When HTML output is better supported we will need to distribute code with synchronized TeX, HTML, CSS, and JS content, and it feels easier to add TeX to an NPM bundle than to integrate tlmgr into NPM/Yarn web bundler frameworks.)

[Rafael-Conde]

Hello there Folks, appreciate all the proposed solutions, but I coudn't make it work complete for my scenario, seems like I have to choose in between using tectonic or other compilation methods.

it seems that when I compile with tectonic, it uses version Biblatex 3.17 and when I compile with VScode it uses Biblatex 3.18b, which I believe is the only one I in fact have installed, because I just made a clean texlive installation and when I run tlmgr info biblatex I get cat-version: 3.18b. I tried searching on my disk for another biblatex.sty file but only got the one on TeXLive's installation directory.

I manually overritten the biber executable to the version 2.17 and then tectonic worked, but when I tried to compile with VScode I got the error, but for inverted versions, now biber was "outdated". is there a way to get it working for everything? it would be interesting for me for testing purposes and also because people I work with uses VScode while I try to stick with Neovim.

[pkgw]

@Rafael-Conde This is not at all a scalable solution, but previous source code releases of biblatex seem to be here on GitHub. One could in principle unpack them and copy their files into your document source tree to try to update the TeX side of things. But besides being a pain, it feels more tractable to figure out a way to get VS Code to run the version of biblatex that you want. Unfortunately I don't know anything about how that setup works :-/

[S1M0N38]

I think that the mismatching of biber/biblatex version is still an issue. The simplest solution is to downgrade to a compatible biber version. Here is how can be done if you use brew as package manager.

0. Install tectonic and compile a simple main.tex without biblatex. Ensure everything is working.

1. Add biblatex package and a citation from your .bib file:

   /* bibliography.bib */
   @article{1903.01182v2,
     author = {Chen, Hao-Yun and Wang, Pei-Hsin and Liu, Chun-Hao and Chang,
       Shih-Chieh and Pan, Jia-Yu and Chen, Yu-Ting and Wei, Wei and Juan,
       Da-Cheng},
     eprint = {1903.01182v2},
     month = {Mar},
     title = {Complement Objective Training},
     url = {http://arxiv.org/abs/1903.01182v2},
     year = {2019},
   }
   

   % main.tex
   \documentclass{article}
   
   \usepackage[backend=biber]{biblatex}
   \addbibresource{bibliography.bib}
   
   \begin{document}
   I'm reading paper \cite{1903.01182v2}
   
   \printbibliography
   \end{document}
   

2. Install biber using brew with brew install biber

3. Try compiling main.tex with tectonic -X compile main.tex rise the following error

   note: "version 2" Tectonic command-line interface activated
   Running TeX ...
   Running external tool biber ...
   error: the external tool exited with an error code; its stdout was:
   
   ===============================================================================
   INFO - This is Biber 2.19
   INFO - Logfile is 'main.blg'
   INFO - Reading 'main.bcf'
   ERROR - Error: Found biblatex control file version 3.8, expected version 3.10.
   This means that your biber (2.19) and biblatex (3.17) versions are incompatible.
   See compat matrix in biblatex or biber PDF documentation.
   INFO - ERRORS: 1
   ===============================================================================
   error: its stderr was:
   
   ===============================================================================
   ===============================================================================
   error: the external tool exited with error code 2
   

4. Your biblatex version is 3.17 so you need biber 2.17 and not biber 2.19 (the minor version of biblatex and biber must match). So we have to downgrade biber from 2.19 to 2.17. Following this blog post you have to:

   # 0. uninstall previous biber version
   brew uninstall biber
   # 1. create a new tap
   brew tap-new $USER/local-biber
   # 2. extract into local tap
   brew extract --version=2.17 biber $USER/local-biber
   # 3. run brew install @version as usual
   brew install [email protected]
   

   The installation takes bit of time because brew is building biber 2.17 from source code.

5. Try again to compile main.tex: tectonic -X compile main.tex

   note: "version 2" Tectonic command-line interface activated
   Running TeX ...
   Running external tool biber ...
   Rerunning TeX because biber was run ...
   note: downloading SHA256SUM
   note: downloading lmmono10-regular.otf
   Rerunning TeX because "main.aux" changed ...
   Rerunning TeX because "main.run.xml" changed ...
   Running xdvipdfmx ...
   Writing `main.pdf` (12.19 KiB)
   Skipped writing 5 intermediate files (use --keep-intermediates to keep them)
   

If you found this solution useful, please leave a :+1: so that other people can easily find it.

[hughesjs]

Can I bump this, having the same issue with newer version number.

INFO - This is Biber 2.19
INFO - Logfile is 'qualifying-report.blg'
INFO - Reading 'qualifying-report.bcf'
ERROR - Error: Found biblatex control file version 3.8, expected version 3.10.
This means that your biber (2.19) and biblatex (3.17) versions are incompatible.
See compat matrix in biblatex or biber PDF documentation.
INFO - ERRORS: 1

I'd suggest that my other issue might address this issue?

https://github.com/tectonic-typesetting/tectonic/issues/1010

---

I can work around this by downgrading my system biber to 2.17, but this is far from ideal...

[zzamboni]

I just spent a few hours debugging this error, so I thought I'd post an issue, and since this is already here, I'll add my thoughts and suggestions.

First, the error I started with:

ERROR - Error: Found biblatex control file version 3.8, expected version 3.10.
This means that your biber (2.19) and biblatex (3.17) versions are incompatible.
See compat matrix in biblatex or biber PDF documentation.

For me the first hurdle was figuring out what the problem was at all, and why compiling my doc with tectonic failed, while compiling it with latexmk (I use the TinyTeX distribution) worked fine. It finally dawned on me that the problem is the version mismatch between biber (an external command, coming from my TeX distribution) and the biblatex style (which is embedded in the tectonic binary, so we cannot change it).

Once I figured this out, I had to find the correct version of biber to use. This was not easy! The compatibility matrix indicates compatibility between minor versions of biber/biblatex (at least for recent releases), but as can be seen in the message, the "biblatex control file version" does not match the "biblatex version", so I had to do a bit of binary search downloading old versions of biber to see which one worked. Finally, I found that for the current release of tectonic 0.12.0, we need biber 2.17.

Based on the above, I have a few ideas, at least until (if) biber is bundled with Tectonic so that it just works 😄

Suggestion 1: at the very least, please document which version of biber is needed for each version of tectonic. I could not find this in the documentation. It would be nice to find this in the release notes and/or somewhere in the book.

Suggestion 2: even better - it would be nice if tectonic would catch the above error and interpret it for the user, describing what the problem is and including a link to the correct biber version to use.

Suggestion 3: now that I installed biber 2.17 and it's in my default path, my TinyTeX builds fail :) It would be nice to be able to tell tectonic the path to the biber binary it should use, that way we could have both systems coexisting without interference (maybe this is possible already? I have not found any relevant flags).

Thanks for all your work on Tectonic!

[Vigilans]

Quoted from https://github.com/tectonic-typesetting/tectonic/discussions/988:

Second, there are some logistical questions about where Tectonic should look for "local files"

I am using archlinux and downgrading biber is nearly an impossible task: after downgrading biber from 2.19 to 2.17 from archives, biber 2.17 requires perl 5.34 where system's perl 5.36 will execute biber 2.17 with error.

However, archlinux's texlive-bibtexextra package provides biblatex.sty of version 3.19 that is compatible with biber 2.19. If tectonic can use the system provided biblatex.sty (located in /usr/share/texmf-dist/tex/latex/biblatex/biblatex.sty, while in debian it's /usr/share/texlive/texmf-dist/tex/latex/biblatex/biblatex.sty from texlive-bibtex-extra package), the problem is solved.

First, a major goal that I have for Tectonic is to ensure that documents can be built reproducibly and reliably. if Tectonic just discovers that special file implicitly, it may or may not exist in the future, leading to surprising breakages.

So, isn't the expectation that "By installing both biber and biblatex.sty from system package manager, I can build my tex in a compatible way" equally important to reproducibility? By sticking with a built-in custom version instead of using system provided one, tectonic is already unreliable and leading to surprising breakages that makes user giving up tectonic.

My suggestion: We already have someone who ensures biber and biblatex.sty is compatible: our system package manager. So for biblatex.sty related package, why not try to search and use a system provided one if exists?

[Vigilans]

With the usage of -Z search-path, I've successfully make biber use package manager provided biblatex:

Given that your system package manager provided biblatex is located in /usr/share/texmf-dist/tex/latex/biblatex (archlinux, /usr/share/texlive/texmf-dist/tex/latex/biblatex for debian), run tectonic with

tectonic -Z search-path=/usr/share/texmf-dist/tex/latex/biblatex main.tex

It will successfully compile.

[Neved4]

It's a good one! Can confirm that it works[^1] with macOS:

tectonic -Z search-path=/usr/local/texlive/*/texmf-dist/tex/latex/biblatex test.tex

In many cases this only helps people who have tectonic alongside a TeX Live installation and the future work is to solve https://github.com/tectonic-typesetting/tectonic/issues/1010 and prevent the issue entirely.

We need to reflect these workarounds in the documentation somehow, if only for how many people stumbled with this issue.

[^1]: Make sure that pathname expansion is enabled, see: #893 (comment) Footnotes. If it's not, as of 2023 you can use the following, for TeX Live and MacTeX: sh tectonic -Z search-path=/usr/local/texlive/2023/texmf-dist/tex/latex/biblatex test.tex

For BasicTeX:
```sh
tectonic -Z search-path=/usr/local/texlive/2023basic/texmf-dist/tex/latex/biblatex test.tex
```

[fperez]

For anyone struggling with this problem, I want to update that on macOS if you use bash as your default shell, the above solution by @onmv needs to be slightly updated to

tectonic -Z search-path=/usr/local/texlive/2023/texmf-dist/tex/latex/biblatex test.tex

(or whatever the actual year of your texlive install is). I suspecdt @onmv's solution is for a zsh-based setup, but bash doesn't expand that wildcard correctly.

Don't ask me why I'm still using bash :) But thanks for the tip above, at least it got me through this super annoying biber problem!! I hope the tectonic team can sort it out in the long run, b/c it's a pretty painful one to debug and will stop many in their tracks.

[Neved4]

@fperez Thanks for pointing out! I was not using a zsh setup, yet I was using BasicTeX instead of MacTeX, therefore my path is /texlive/2023basic/ instead of /texlive/2023/. Overall, the globbing approach will be a more long-term solution when it's enabled.

Generally speaking, most shells come with pathname expansion enabled[^1], unless set -f is in effect.[^2] You can check whether it's enabled or disabled with set -o:

$ set -o | grep noglob
noglob         	off

If it's on it's disabled, if it says off it's enabled. You can disable it with either set -f or set -o noglob, and reenable it with set +o noglob.

Updating the original post to better reflect this clarification 🖤

[^1]: See pathname expansion in POSIX 2017.1 - Word Expansions [^2]: See set -e in POSIX 2017.1 - Special Built-In Utilities, set

[fperez]

Thanks so much @onmv for that great explanation/clarification!

I'm actually puzzled b/c I just tested again on another macOS box (bash v 5.2.15(1) from homebrew), and even though it shows me:

(base) (main)longs[ttest]> set -o | grep noglob
noglob          off

it still doesn't work with the * glob, and I need to pass explicitly the .../2023/... path to avoid the dreaded This means that your biber (2.19) and biblatex (3.17) versions are incompatible.

Not a huge deal, and I don't want to derail this issue which is about tectonic. Just reporting my experience in case anyone else runs into similar problems. But thanks again for your patient and detailed input, I'm still learning unix tricks, 30 y later :)

[Neved4]

@fperez Interesting! I was able to do it properly, I wonder what's affecting it in your case:

bash-5.2$ du -hcs /usr/local/texlive/*/texmf-dist/tex/latex/biblatex
2.3M	/usr/local/texlive/2023/texmf-dist/tex/latex/biblatex
2.3M	total

[fperez]

No clue! Weird... Your du example works for me identically, and yet:

$> set -o | grep noglob
noglob          off
$> tectonic -Z search-path=/usr/local/texlive/*/texmf-dist/tex/latex/biblatex paperdemo.tex 
Running TeX ...
warning: lineno.sty:296: Invalid UTF-8 byte or sequence at line 296 replaced by U+FFFD.
warning: opensans.sty:115: 

[... elided ...]

Running external tool biber ...
error: the external tool exited with an error code; its stdout was:

===============================================================================
INFO - This is Biber 2.19
INFO - Logfile is 'paperdemo.blg'
INFO - Reading 'paperdemo.bcf'
ERROR - Error: Found biblatex control file version 3.8, expected version 3.10.
This means that your biber (2.19) and biblatex (3.17) versions are incompatible.
See compat matrix in biblatex or biber PDF documentation.
INFO - ERRORS: 1
===============================================================================

But with the explicit 2023:

$> tectonic -Z search-path=/usr/local/texlive/2023/texmf-dist/tex/latex/biblatex paperdemo.tex 
Running TeX ...
warning: lineno.sty:296: Invalid UTF-8 byte or sequence at line 296 replaced by U+FFFD.
warning: opensans.sty:115: 

[... elided ...]

Writing `paperdemo.pdf` (19.01 KiB)
Skipped writing 7 intermediate files (use --keep-intermediates to keep them)

In any case, no need to resolve this bash mystery here :) I'm loving tectonic and I want it to succeed, thanks to the whole team for getting us this great alternative to the monolithic, root-requiring TeXs (nothing against those, but they aren't always the right solution in some scenarios)!