Full `biblatex` support: somehow add `biber` engine to the backend

[pkgw]

It seems that some biblatex modes require the backend program biber as an alternative to bibtex. (Cf issue #33.) I don't know anything about it, but perhaps we should add it.

To gauge interest, everyone is invited to add a short comment on this issue if they use biber in their day-to-day TeX'ing.

[pkgw]

Here's biber. It's written in Perl. Hmmm.

[flying-sheep]

definitely. bibtex has no unicode support and is generally weird and flaky.

any modern LaTeX toolchain is LuaLaTeX/XeTeX + biber.

[mbollmann]

Just stumbled upon this project (wonderful idea!), but only recently switched to working with LuaLaTeX+biber (mainly because of Unicode support), so would definitely need this.

[jtojnar]

In the worst case, you could use the provided binaries (~20 MB).

[telemachus]

Just chiming in to say, yes please! I installed tectonic and I was very happy with the results overall. But sadly, my citations are not processed. They're simply left in bold after the build process. I would switch from latexmk to tectonic in a heartbeat if this was working.

If you decide not to include support for biblatex or for the time being, I'd suggest that if biblatex is present, the build should fail or output a clear warning that citations via biblatex aren't supported.

Thanks.

[pkgw]

OK, seems that this is a popular request. Anybody want to attempt to rewrite biber as a Rust crate? :-)

In the meantime I will consider pull requests that let Tectonic invoke the biber executable as in other TeX engines. This very much goes against the goal of reproducible, self-contained document compilations, but we should also be practical here.

[joostkremers]

I just found out about Tectonic and am tempted to give it a try, but this issue would really be a deal breaker for me.

But even more so, since you aim for a fully Unicode-enabled environment, you really cannot rely on BibTeX. BibTeX will fail in subtle ways when using Unicode: it may sort things incorrectly and you may end up with things like ä in your list of references. But since BibTeX doesn't produce any errors, you won't know unless you carefully examine the output.

goal of reproducible, self-contained document compilations

Do you have any ideas on how to deal with updates of packages? Updates fix bugs, which is good, but they may also wreak havoc on reproducibility, which is bad...

[pkgw]

@joostkremers Thanks; you make a good point — if we're going to say that we're Unicode-friendly, it matters to support it in .bib files as well as .tex.

As for package updates, Tectonic's "bundle" system tries to provide a framework in which you get both reproducibility and the capability to upgrade packages. This StackExchange comment elaborates a tiny bit more, but I need to write some real documentation that explains the concept in more detail.

[pkgw]

This might well be Google-able, but if anyone has a sample document using biblatex and biber in a nontrivial way, that they don't mind sharing, could you post a link to the files? I want to take a look at this and figure out how hard it would be to get the less-than-ideal "shell escape" solution working.

[jtojnar]

You can try one of the examples provided with fithesis. Though it uses some packages other than biblatex which might turn out to cause problems.

[joostkremers]

Well, perhaps this will help. Take the following .bib file:

@article{Haempke2002,
	author = {Hämpke, Johannes},
	title = {Irgendein Titel},
	journal = {Irgendein Journal},
	year = {2002},
	volume = {5},
	number = {6},
	pages = {7-8}
}

@article{Holbein2003,
	author = {Holbein, Frank},
	title = {Some Title},
	journal = {Some Journal},
	year = {2003},
	volume = {1},
	number = {2},
	pages = {3-4}
}

Note the author name Hämpke, which contains an umlaut.

Then use the following .tex file, which uses biblatex with BibTeX as backend:

\documentclass{article}

\usepackage[style=authoryear-comp,backend=bibtex]{biblatex}
\addbibresource{samp.bib}

\begin{document}

\textcite{Haempke2002, Holbein2003}

\printbibliography

\end{document}

I called the .bib file samp.bib. When you run this through xelatex --> bibtex --> xelatex, the result is this:

Note how in the list of references Hämpke is sorted after Holbein, even though according to German sorting rules (and indeed according to English sorting rules), their order should be reversed.

The biblatex manual says the following about using the BibTeX backend (I don't know what all of them really mean, TBH):

Key limitations of the BibTeX backend are:

• Sorting is global and is limited to Ascii ordering
• No re-encoding is possible and thus database entries must be in LICR form to work reliably
• The data model is fixed
• Cross-referencing is more limited and entry sets must be written into the .bib file
• Fixed memory capacity (using the --wolfgang option with bibtex8 is strongly recommended to minimize the likelihood of an issue here)

BTW, a quick test suggests that you may get better results using bibtex8 as backend. But I have no experience using biblatex / biber for documents that are not in English and have different sorting requirements (e.g., in Swedish, ä is a separate letter that is sorted after z), so I can't compare them to bibtex8.

[edwardabraham]

A bit late to the discussion. But yes please, biber. One reason we use it is that customising the style files is easier with biblatex than with bibtex.

[WtfJoke]

Is there any progress on this?

[pkgw]

@WtfJoke Unfortunately, no, there hasn't really been any change here. There are other areas where shell-escape support would make people's lives easier (example), though, so it's looking more and more like that will be a valuable feature to prioritize.

[Arunscape]

I ran into this issue as well, so for anyone else who needs a temporary solution, I pieced together a Makefile:

# Makefile

.PHONY: all

all: report.pdf

%.pdf: %.tex 
        tectonic --reruns 0 $<
        # Run biber if we find a .bcf file in the output
        if [ -f $(notdir $(<:.tex=.bcf)) ]; then \
                biber $(notdir $(<:.tex=)); \
        fi
        tectonic --reruns 0 $<
        rm $(basename $<).bbl $(basename $<).bcf $(basename $<).blg

edit: just wanted to say that I came across tectonic only last week, and I love what you've created here👏 💯

[rugk]

To gauge interest, everyone is invited to add a short comment on this issue if they use biber in their day-to-day TeX'ing.

Well… bibtex is just outdated legacy software. (no Unicode support, not flexible etc.) Just like the thing tectonic seeks to replace… :wink:

So as a modern tool, tectonic should somehow certainly support that.

[koalp]

Hello, I came across this awesome project and also use biber. I tried to use it along biber but have an incompatible version of biber installed (newer than needed by the packaged biblatex).

I will surely create a container image to be able to personally use and test this tool but don't think I will use it in makefiles or pipelines before biber is integrated.

[WtfJoke]

@koalp I experienced the same issue and created a docker image. You can find it here: https://github.com/WtfJoke/tectonic-docker

~However I didnt update the image in a while.~

[RicardoMonteiroSimoes]

Could this be the reason for my failing Glossary/References print? We're rather new to LaTeX and trying to get things running, from what I've tried tectonic is by a very large margin the fastest compiler to use as action, but it doesn't generate glossary etc. I've tried the https://github.com/WtfJoke/tectonic-docker action with biber install, but that leads to a fail:

note: Writing `main.ist` (918 B)
note: Writing `main.glo` (1.18 KiB)
note: Writing `main-blx.bib` (343 B)
note: Writing `main.out` (3.34 KiB)
note: Writing `main.toc` (5.88 KiB)
note: Writing `main.acn` (233 B)
note: Writing `main.aux` (16.85 KiB)
note: Writing `main.bbl` (513 B)
note: Writing `main.pdf` (1.88 MiB)
note: Writing `main.run.xml` (2.60 KiB)
note: Writing `main.tdo` (2.69 KiB)
INFO - This is Biber 2.16
INFO - Logfile is 'main.blg'
ERROR - Cannot find 'main.bcf'!
INFO - ERRORS: 1

Not entirely sure if the error is on my side, but considering normal compilation on desktop with TeXStudio

[WtfJoke]

@RicardoMonteiroSimoes
Im not sure if that could be the problem... Have you used tectonic-docker or the tectonic github action? Did you run tectonic using these commands?

1. tectonic --keep-intermediates --reruns 0 main.tex
2. biber main
3. tectonic main.tex

Do you have an example of your tex files?

[RicardoMonteiroSimoes]

@WtfJoke I managed to fix it, someone in my team had set the backend of biblatex to bibtex, which explains why it didnt compile.

Yet my problem still remains about the glossary/references part. They don't want to compile and print in the action:

- name: Setup tectonic with biber
        uses: wtfjoke/setup-tectonic@v1
        with:
          github-token: ${{ secrets.AUTO_RELEASE_TOKEN }}
          biber-version: "latest"
          
      - name: Run Tectonic + Biber
        run: |
          tectonic --keep-intermediates --reruns 0 main.tex
          tectonic main.tex

Currently trying to see if we can make the repo public with my team, but here are the important snippets:

% !TeX program = lualatex

\documentclass[a4paper]{article}

% Packages %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage[backend=biber, style=ieee, citestyle=ieee]{biblatex}

% Document structure %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
	\sloppy
	\pagenumbering{roman}
	\color{content_normal}
	
	\begin{titlepage}
		\maketitle
		\thispagestyle{empty}
	\end{titlepage}

	%\subfile{sections/disclaimer}

	\tableofcontents
	\newpage
	
	\pagenumbering{arabic}

	
	%\subfile{sections/example}
	\subfile{sections/productvision/introduction}
	\subfile{sections/productvision/produktidee}
	\subfile{sections/productvision/spielregeln}
	\subfile{sections/productvision/umsetzung}
	\subfile{sections/productvision/risiken}
	\subfile{sections/productvision/architektur}
	\subfile{sections/productvision/technologien}
	\subfile{sections/productvision/miscellaneous}	
	%\subfile{sections/context}
	%\subfile{sections/objectives_functional_overview}
	%\subfile{sections/quality_attributes_non_functional_requirements}
	%\subfile{sections/constraints}
	%\subfile{sections/principles}
	%\subfile{sections/architecture}
	%\subfile{sections/external_interfaces}
	%\subfile{sections/code}
	%\subfile{sections/data}
	%\subfile{sections/infrastructure_architecture}
	%\subfile{sections/deployment}
	%\subfile{sections/operation_support}
	%\subfile{sections/decision_log}	% add sections here
	
	\newpage

	\printglossary[type=\acronymtype]
	\printglossary
	
	\newpage
	\printbibliography[heading=bibintoc]
	
	\newpage
	\listoftodos[Notes]
\end{document}

a local compilation works fine, so it shouldnt be a bad file

[WtfJoke]

FYI: I took your tex file and tried to make it compile (by making some changes) and it worked... you can check out https://github.com/WtfJoke/setup-tectonic-sample feel free to compare the files. Not sure if its an issue with the setup-tectonic action (feel free to open an issue as this goes a bit offtopic here)

[aterenin]

+1 to this issue, for many of the same reasons above, including Unicode support.

Is the main difficulty here that Biber is implemented in Perl, which makes it difficult to "build from source and glue together" compared to BibTeX? Looking at the BibTeX engine, all I see is a single C file. Is this why we are talking about shell escape support?

[aterenin]

Sorry to double post (my phone doesn't want to let me edit) but wanted to add to the above: if this is remotely doable for a Rust-fluent TeX-internals-novice, I'd be happy to make an attempt here with some support on what needs to be done.

I'd love to use Tectonic as the day-to-day build tool for writing my PhD thesis!

[pkgw]

@aterenin That's right, I'd say that it is essentially impossible to try to offer Biber as a built-in option in the way that Tectonic handles bibtex. The least-impractical approach would probably be an independent, work-alike Rust implementation. I'm not going to try to stop anyone from undertaking such a project but it's certainly not high on my priority list.

We are decently close to working shell-escape with #708 but it turns out to be a subtler problem than I first appreciated, and I believe that this is something where it's important to provide a really robust, carefully-designed solution. I think I see that solution, but I personally won't be able to spend any time on it for a little while, ironically because I'm working on a major proposal where I'm finding myself quite annoyed by the limitations of plain bibtex ...

[aterenin]

That's too bad - sounds not particularly encouraging. Biber is far too large to be reimplementable by one person, though if this someday did happen, that would be awesome, as it would finally become just as fast as BibTeX on very large bib files.

I did a brief Google search, and Perl supports interop with C, including the ability to instantiate a Perl interpreter from within C code, which together with Rust FFI means that in principle one can start a Perl interpreter from within Rust, thereby running Biber's code, perhaps an appropriately patched version thereof.

However, whether this has any advantages over just downloading a Biber binary and invoking it as a shell command - this is unclear.

[aterenin]

I wanted to add that perlcc was mentioned in the Biber issue https://github.com/plk/biber/issues/338 as a way to get Biber to run on WebAssembly. Development made in pursuit of this goal could also make Biber+Tectonic integration more viable.

For the moment, I've been running Biber separately from VS Code by running tectonic --reruns 0 to generate the bcf file, then running Biber to generate the bbl file, and running tectonic to build the remaining document. This works great! It's even possible to create a hash of the bcf file to avoid re-running Biber if this file is unchanged, which can greatly improve compile times for documents with large bib files - this can be achieved with the one-liner if [ \"$(openssl md5 %DOCFILE%.bcf)\" != \"$(cat %DOCFILE%.bcf.hash)\" ]; then; biber --nolog %DOCFILE%; openssl md5 %DOCFILE%.bcf > %DOCFILE%.bcf.hash; fi.

This said, integration would be a step up, because this could all be theoretically done from within Tectonic, which could handle all of this automatically in addition to keeping Biber up-to-date in a reproducible manner.

[vadimkantorov]

I found perlcc / staticperl not reliable enough to compile all required packages. And an easier to reason/debug route may be as proposed by @aterenin:

1. Compile statically Perl and all dependencies, make sure they run biber on x64
2. Prepare a small C-based Perl embedded interpreter runner - make sure it works
3. Compile it all with Emscripten

I have advanced towards (1) in https://github.com/vadimkantorov/buildbiber/blob/master/emperl.sh, but the script is in some broken, unreadable shape for now

[pkgw]

OK, as of the 0.7 release, Tectonic now supports execution of biber as an external tool. The shell-escape work set up the technical pieces needed, and my overall feeling now is that it's better to probably compile a document correctly as opposed to definitely compile it incorrectly.

The current support doesn't let you pass command-line arguments to biber. It ought to be straightforward/nice to make that possible using the V2 Tectonic.toml scheme or a new -Z unstable option.

In principle I would certainly like to avoid having to invoke an external program, but as per the recent discussion, my guess is that it would be a pain to get that to work correctly.

[aterenin]

Amazing! 🎉🎉🎉

Unfortunately, I'm getting an error when using this.

aterenin@MacBook-Pro Thesis % tectonic --synctex thesis.tex
note: generating format "latex"
Running TeX ...
warning: thesis.tex:148: Overfull \hbox (16.34637pt too wide) in paragraph at lines 148--148
warning: thesis.tex:174: Overfull \hbox (18.73097pt too wide) in paragraph at lines 174--174
warning: thesis.tex:214: Overfull \hbox (13.0286pt too wide) in paragraph at lines 214--214
warning: thesis.tex:230: Overfull \hbox (4.78604pt too wide) in paragraph at lines 230--230
warning: thesis.tex:245: Overfull \hbox (20.94623pt too wide) in paragraph at lines 245--245
Running external tool biber ...
error: the external tool exited with an error code; its stdout was:

===============================================================================
INFO - This is Biber 2.16
INFO - Logfile is 'thesis.blg'
INFO - Reading 'thesis.bcf'
INFO - Found 5 citekeys in bib section 0
INFO - Processing section 0
INFO - Looking for bibtex format file 'citations.bib' for section 0
ERROR - Cannot find 'citations.bib'!
INFO - ERRORS: 1
===============================================================================
error: its stderr was:

===============================================================================
===============================================================================
error: the external tool exited with error code 2