RFC: Introduce `bundle serve` command.

[let-def]

This pull request adds a new bundle serve command, an interactive version of bundle cat. It allows querying the contents of multiple files without spawning a new process each time.

Motivation

While working on TeXpresso, a real-time LaTeX file previewing tool, we've encountered a bottleneck in our current workflow. It is possible to use Tectonic as a TeX distribution via the bundle cat command, but querying the contents of multiple files spawns a new process each time and leads to performance issues. We propose introducing a new bundle serve command to address this problem.

Background

TeXpresso is an experimental tool that allows users to preview LaTeX files in real-time. It achieves using a patched XeTeX and a specialized viewer. You can see it in action with Vim, Emacs and VSCode. The viewer requires a TeX distribution, which is where Tectonic comes in. We started by using bundle cat, but we need a more efficient solution.

Proposal

The bundle serve command is an interactive version of bundle cat that allows querying the contents of multiple files without spawning a new process each time. To achieve this, we need to delimit the different requests.

We implemented a simple protocol to stream all contents over a single stream:

1. On standard input, bundle serve reads a list of file names, one per line (delimited by \n).
2. For each file name, in order, bundle serve responds with a 9-byte header followed by a payload:
   • The first byte indicates the type of response: C for contents, P for path, or E for errors.
   • The next 8 bytes represent the length of the payload as a little-endian 64-bit unsigned integer.
   • The payload contents depend on the first byte:
     • C indicates the contents of the requested file.
     • P indicates a path on the local file system where the requested file can be found.
     • E indicates an error message explaining why the request failed.

Alternative

If a more complex protocol is not desirable, a simpler line-based protocol reading file names and writing either paths or error messages would be sufficient. This approach is similar to kpsewhich -interactive (with improved error handling).

Conclusion

We believe introducing the bundle serve command and its accompanying protocol would significantly improve the performance of TeXpresso and, potentially, other tools that rely on bundle cat. We're open to feedback and alternative solutions.

[let-def]

Notes:

1. At the moment, TeXpresso bundles a fork of Tectonic. I am cleaning that up as I would like it to work with vanilla tectonic. This PR would allow us to get rid of the fork without impacting performance.

2. I don't know of a standard solution to this "file delimitation" problem. HTTP 1.1 solved a similar one with Chunked Transfer Encoding, though it is both more flexible and more complex. The proposed protocol can be implemented in a few lines of code. The alternative has the benefits of playing better with shell utils/scripts.

[codecov[bot]]

Codecov Report

:x: Patch coverage is 6.89655% with 54 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 46.96%. Comparing base (19654bf) to head (000e277). :warning: Report is 240 commits behind head on master.

Files with missing lines	Patch %	Lines
src/bin/tectonic/v2cli/commands/bundle.rs	7.69%	48 Missing :warning:
crates/bundles/src/cache.rs	0.00%	3 Missing :warning:
crates/io_base/src/lib.rs	0.00%	3 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1195      +/-   ##
==========================================
- Coverage   46.99%   46.96%   -0.04%     
==========================================
  Files         176      176              
  Lines       65196    65255      +59     
==========================================
+ Hits        30639    30644       +5     
- Misses      34557    34611      +54     

Components	Coverage Δ
tectonic	66.83% <7.69%> (-1.42%)	:arrow_down:
bridge_core	68.08% <ø> (ø)
bridge_flate	33.89% <ø> (ø)
bridge_fontconfig	∅ <ø> (∅)
bridge_freetype2	∅ <ø> (∅)
bridge_graphite2	∅ <ø> (∅)
bridge_harfbuzz	∅ <ø> (∅)
bridge_icu	∅ <ø> (∅)
bridge_png	∅ <ø> (∅)
bundles	69.30% <0.00%> (-0.75%)	:arrow_down:
cfg_support	∅ <ø> (∅)
dep_support	∅ <ø> (∅)
docmodel	75.08% <ø> (ø)
engine_bibtex	64.42% <ø> (-0.03%)	:arrow_down:
engine_spx2html	0.00% <ø> (ø)
engine_xdvipdfmx	73.17% <ø> (ø)
engine_xetex	72.65% <ø> (ø)
errors	0.00% <ø> (ø)
geturl	59.49% <ø> (ø)
io_base	73.23% <0.00%> (-0.42%)	:arrow_down:
pdf_io	23.57% <ø> (+0.01%)	:arrow_up:
status_base	71.05% <ø> (ø)
xdv	0.00% <ø> (ø)
xetex_format	∅ <ø> (∅)
xetex_layout	58.02% <ø> (ø)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[CraftSpider]

Please rebase on recent master, then I'll try to review this as soon as I'm able

[let-def]

I have found workarounds such that these changes are no longer required by TeXpresso to work with Tectonic.

I still think that a more efficient interface for querying Tectonic bundles and/or local cache would be useful, but this PR is not the right place for discussing that and I don't have the time to look at this at the moment. I will close the PR for now.

If you have plans for related features in the future, I will be happy to discuss again. Thanks.