`\inputminted` support: files missing from shell-escape working directory

[kwshi]

trafficstars

#708 introduced shell-escape support, which (as motivated in #38) is needed for the exceedingly popular minted package for typesetting syntax-highlighted source code.

However, shell-escape support alone is not enough to support minted--in particular, the \inputminted command does not work correctly. Here is a simple project demonstrating the issue: tectonic-minted-test.tar.gz. Its contents are:

• hello.tex

  \documentclass{article}
  \usepackage{minted}
  \begin{document}
  \inputminted{python}{world.py}
  \end{document}
  

• world.py

  print("hello world")
  

When running tectonic -Z shell-escape in the above project, the following error is returned:

Error: cannot read infile: [Errno 2] No such file or directory: 'world.py'
warning: command exited with error code 1
error: failed to execute the shell-escape command "pygmentize -l python -f latex -P commandprefix=PYG -F tokenmerge -o _minted-hello/72214DB4E1E543018D1BAFE86EA3B444BA1FB043E1554A6696271B99677258E5.pygtex world.py": execution of the request failed
error: hello.tex:4: Package minted Error: Missing Pygments output; \inputminted was
probably given a file that does not exist--otherwise, you may need
the outputdir package option, or may be using an incompatible build tool,
or may be using frozencache with a missing file.

See the minted package documentation for explanation.
Type  H <return>  for immediate help
error: halted on potentially-recoverable error as specified

After some digging around, I think I've figured the reason for this error.

• Unlike the "original" LaTeX compilers, which run shell-escape commands from the same working directory as the project, Tectonic runs shell-escape commands from fresh, empty temporary directories (I checked this by making a blank LaTeX file containing some dummy text and \write18{pwd}).
• Other files in the project directory are not copied over to this temporary directory (\write18{ls -l} shows an empty list).
• As such, when minted invokes the pygmentize shell command, it tries to load the source file (expecting it to be in the current project working directory) but fails to find it because its working directory is actually set to something like /tmp/.tmpVGQl7T.

This being the cause gives rise to a temporary workaround: passing \inputminted an absolute rather than relative path makes Tectonic run correctly.

However, I think this workaround is somewhat suboptimal (limits portability of the project/depends on the absolute path of the project; clunky to type; etc.); I think it would be ideal to somehow allow minted to access files from the project directory.

I'm not sure, technically, how this would be done--maybe special-case support, e.g. what Tectonic did for Biber, may be required. Or maybe there's a simpler solution. I'm raising this issue because I imagine, given the popularity of minted, I must not be the only one facing this trouble.

[pkgw]

Thanks for reporting! Your diagnosis sounds correct, and I agree that using absolute paths should work around the issue but definitely isn't a real solution.

I think the first step here would be to check out how \inputminted works and see if there is some way to get it to indicate to the engine that the source file (in this case, world.py) will be needed for the shell-escape session. Fundamentally Tectonic will need to know that the file needs to be copied to the shell-escape working directory. Maybe there will be a clever way to allow the engine to "figure out" that it needs to copy an extra file automagically ... but I have a hunch that it might be necessary to patch our version of Minted, and we might even need to add a special primitive to provide the proper hooks. That's certainly not my ideal case, but I think it's really important that shell-escape tools be run in a tightly controlled environment, so I really want to find a way to make this use-case work using the temporary directory.

[DanySK]

Hi, I often use \write18 to call scripts that scrape the web and prepare LaTeX files, and I can not use absolute paths. I think that it should be possible to configure tectonic to copy over in the temporary directory more files than the core ones, or, maybe better, copy everything. Alternatively or additionally, it would be nice to have a -inplace flag or similar to run into the document folder directly.

I also opened a discussion here.