Support attachfile Package

[bradcarman]

If possible, it would be great if the "attachfile" package could be supported by Tectonic.

[Neved4]

Hey @bradcarman, I've personally never worked with attached files on PDF before, could you provide a MWE and which PDF viewer you're using that supports them?

[Neved4]

@bradcarman I got some time to check this with Adobe Acrobat DC

Tecctonic already includes the attachfile package in our last 2022 bundle.

When compiling with either tectonic -p or XeTeX we get:

Package attachfile Warning: attachfile works _only_ with pdfLaTeX and LuaLaTeX
(attachfile)                and _only_ in PDF-generating mode. For this run,
(attachfile)                placeholders will be substituted for all
(attachfile)                attachfile commands.

It seems that XeTeX/Tectonic are missing the engine macros for attachfile to work, so it fails gracefully. I'm not sure if investigating which macros specifically is the package using but seems something suited for upstream so the package provides support for XeTeX/tectonic. Changing an engine may be harder than changing a package 💜

From attachfile documentation:

"attachfile requires either pdfLATEX version 0.14 or later or LuaLATEX. [...] While there are many other ways to produce pdf files from LATEX source, attachfile v1.9 supports only pdfLATEX and LuaLATEX."

Seems this lack of support was a deliberate decision from the package author.

---

Looking for alternatives, there's the embedfile package, but that only supports pdfTEX/LuaTEX also.

Finally, there's the attachfile2 package, which is less ancient than its v1 brother and has code that supports XeTeX.

MWE:

\documentclass{article}
  \usepackage{attachfile2}

\begin{document}
 \attachfile{Test.png}
\end{document}

Using it produces a file with an attachment, but when opening it with Acrobat DC the attachment name is an uncanny ˛ˇ and it fails to open. Since attachfile2 used to work my guess is that its in the need of an update since last one was in 2019.

Do you know of other PDF readers that display attachments we could try?

[9SMTM6]

Firefox works just fine for that purpose (tested with the attachfile package documentation ).

In my experimentation in windows attachfile2 produces a file with the correct name etc, ist just that it contains only "null".

Which is the same behavior as when I reference a non-existing file (which fails silently btw, not great).

I suspect that that is simply due to me using the v2 cli, where I'm not sure where tectonic is building the output? I also experienced issues with minted, cant get it working referencing external files with \inputminted, while \inlineminted etc work fine once shell_escape is enabled.

Did you enable shell_escape? I think attachfile2 wants that.

EDIT: Yeah no, with v1 the document also only contains "null"

[Neved4]

Thanks, yes I can reproduce the issue with Firefox. Using attachfile and attachfile2 work with other LaTeX engines, but not with tectonic. I've checked and shell-escape is not needed for attachfile2.

When tectonic compiles the document but the attached file is empty, in my case a null image.

Tectonic is in a situation where none of the attaching files packages work, I believe that attachfile2 is the way to go for support and the one we'd be better of targeting.

Also, whenever using XeTeX or a XeTeX-based engine, something of the attachment prevents it being displayed correctly under some PDF viewers like Adobe Acrobat DC. We might need to tackle that issue separately.

@vlasakm You're good with the internals, any insight on this appreciated 💜

Output:

warning: Tectonic does not support `config' special. Ignored.
warning: File "./Test.png" not found.
warning: Interpreting special command fstream (pdf:) failed.
warning: >> at page="1" position="(133.768, 667.198)" (in PDF)
warning: >> xxx "pdf:fstream @atfi_obj_1(./Test.png)<</Type/EmbeddedFile/Para..."
warning: >> Reading special command stopped around >><</Type/EmbeddedFile/Params<</Size 251184/ModDate(D:20221005...<<
warning: Cannot close undefined object @atfi_obj_1.
warning: Object @atfi_obj_1 used, but not defined. Replaced by null.

[vlasakm]

I agree that attachfile2 is the right alternative to attachfile for XeTeX (generally one needs to look for XeTeX support when using Tectonic).

For these kinds of low level packages that deal with the output format (PDF, PostScript, HTML, DVI) one needs to look for xetex, dvipdfmx so called "backend" (or "driver") options. This is the case with attachfile2, but for example also with TikZ. Though often the packages autodetect the "backend", in some cases it can fail.

My opinions, point by point:

If possible, it would be great if the "attachfile" package could be supported by Tectonic.

attachfile doesn't support XeTeX, apart from large patching effor in the bundle, Tectonic can't really do much. And the TeX community already answered with attachfile2 which solves that. From that I gather that attachfile won't be changed retroactively, so best bet is probably to avoid it for new projects and use the alternatives.

Since this github issue is about attachfile specifically, I suggest for it to be closed or renamed, to reflect the real issue being discussed.

When tectonic compiles the document but the attached file is empty, in my case a null image.

There are multiple things from the warnings:

warning: Tectonic does not support `config' special. Ignored.

This is https://github.com/tectonic-typesetting/tectonic/issues/904. I didn't investagate how much this influences attachfile2, but it should be fixed with #953.

warning: File "./Test.png" not found.

This is the most crucial line. Though I must admit, that I missed it too. The next few lines is just the backtrace and the end is just the result of the file not being embedded (since it was not found). The fact that the file is not found is a mistake in the code, which unconditionally failed any attempt to find files to embed. This should also be fixed with #953.

[Neved4]

@vlasakm Thanks a lot!

[vlasakm]

I forgot to mention that although the file gets embedded just fine, I get:

warning: 1024-byte read failed
caused by: failed to fill whole buffer

To be honest I have no previous experience with buffered reads in Rust, and the issue might be something in the layers of abstraction, so I didn't investigate further.

[pkgw]

Thanks again for investigating and working on this @mnrvwl and @vlasakm! Just to confirm, does that 1024-byte read complaint happen after your pdf_io fix is applied, @vlasakm?

[Neved4]

@9SMTM6 I had almost forgotten about minted! Could you provide me me a MWE to test it?

[9SMTM6]

@mnrvwl Sure. It appears to mostly work with v0.12.0 on windows

% index.tex
\documentclass{article}
\usepackage{minted}
\begin{document}

\begin{minted}{python}
print("inline")
\end{minted}

\inputminted{python}{./test.py}

\end{document}

# test.py
print("external")

with v1 CLI: tectonic -Z shell-escape -Z shell-escape-cwd=. index.tex, with v2 CLI config (and emptying pre and postamble, adjusting path of inputminted to ../test.py):

[doc]
name = 'v2cli'
bundle = 'https://data1.fullyjustified.net/tlextras-2022.0r0.tar'

[[output]]
name = 'default'
type = 'pdf'
shell_escape = true
shell_escape_cwd = "."

As I said, it appears to mostly be working if one is setting shell_escape_cwd, in that it looks like a correct PDF but its still throwing some shell errors for me. Without shell_escape_cwd, inputminted does not work, the other one is.

Also the directory of shell_escape_cwd is getting some intermediate file waste, but that probably is to be expected with its approach.

With xelatex --shell-escape index.tex both are working with no reported errors.

Perhaps xelatex just doesnt report execution errors?

The reason I think that might be the case is the error:

failed to execute the shell-escape command "for ^%i in (pygmentize.exe pygmentize.bat pygmentize.cmd) do set > default.aex <nul: /p x=^%~$PATH:i>> default.aex": execution of the request failed

this looks like its just running a commend for every os (not an expert in whatever language that is tho) and looking what sticks.

[9SMTM6]

@pkgw

Just to confirm, does that 1024-byte read complaint happen after your pdf_io fix is applied

As far as I understand it from tracing it through release notes, merges etc, that commit is part of 0.12.0. If that is so, on 0.12.0 I'm still getting the same error when trying to use attachfile2.