reducing dependency on shell escape

[davidcarlisle]

This is a feature request rather than a bug report.

I am trying to get minted to run in the default restricted shell escape mode of texlive.

I have a latex server (see https://www.learnlatex.org/) returning pdfs generated from user supplied tex source. It would be nice to be able to include minted examples...

It's running in a fairly sandboxed environment and I'd be prepared to allow pygmentize (which is installed) but not unrestricted shell escape.

So as an experiment I added pygmentize to the list of allowed commands on texmf.cnf

shell_escape_commands = \
bibtex,bibtex8,\
extractbb,\
gregorio,\
kpsewhich,\
makeindex,\
repstopdf,\
r-mpost,\
texosquery-jre8,\
pygmentize,\

Unfortunately this doesn't work and minted wants access to \ShellEscape{rm and a host of other things that can't be added to the list of allowed shell commands.

In such a context I don't need minted to clean up after itself as the entire working directory tree will be deleted the moment the job finishes, so I probably(?) don't need it to have access to rm.

I started to trace what would need to be changed but got a bit lost, does this sound feasible or are there too many essential shell operations the code relies on?

[muzimuzhi]

It seems that in restricted write18, redirection > is not supported.

With pygmentize appended to shell_escape_commands, the following example doesn't work:

\documentclass{article}
\usepackage{shellesc}

\begin{document}
\ShellEscape{pygmentize -S default -f latex -P commandprefix=PYGdefault > ./default.pygstyle}
\end{document}

This can be further simplified to \ShellEscape{echo "abc" > abc.txt}, with echo appended to shell_escape_commands.

From the doc of pygments, it uses stdout when option -S is used

With the -S option, print out style definitions for style <style>
for formatter <formatter>. The argument given by -a is formatter
dependent.

Of course, one can write a wrapper python script to support syntax of

pygmentize -S default -f latex -P commandprefix=PYGdefault -o default.pygstyle

[gpoore]

All the pygmentize commands could be switched to use -o <outfile> to eliminate the redirect issue @muzimuzhi pointed out.

If cleanup isn't an issue, then \DeleteFile isn't needed, which removes the rm usage. That leaves \ProvideDirectory and \TestAppExists as the other typical uses of \ShellEscape. \ProvideDirectory could be avoided by using a hardcoded cache dir, or can be skipped altogether if caching is turned off. \TestAppExists can be disabled if you know pygmentize is installed.

So actually using -o <outfile> plus no cache plus \def\DeleteFile#1{} and \def\TestAppExists#1{} should cover most cases. The other (rarer) case is the autogobble option that requires running Python. Some or all of this may be worth incorporating into minted as a package option. It might also be possible to handle some of this by having the server always include some sort of patching package that does a few \AtEndPreamble etc. checks and patches.

[davidcarlisle]

Thanks. I'd started adjusting \DeleteFile, \TestAppExist and friends but missed the redirect, from the comments it seems that it should be feasible; I will probably try again and report back here but probably not this week.

[muzimuzhi]

All the pygmentize commands could be switched to use -o <outfile> to eliminate the redirect issue @muzimuzhi pointed out.

@gpoore Can you give some hints? I checked the implementation of -S option in pygments, it directly calls print(). https://github.com/pygments/pygments/blob/a486735f2b485ed35bcb60101fc9a3e2a0eca11d/pygments/cmdline.py#L295-L315

@davidcarlisle Here (https://github.com/muzimuzhi/minted/tree/reduce-shell-escape) is a first stage implementation. It adds a new pkg option "reduceshellescape", which now just sets cache=. and disables \DeleteFile, TestAppExist, \ProvideDirectory, and the check for \pdf@shellescape. Apart from the redirection issue, it works well.

[gpoore]

@muzimuzhi You're right about -S...I think I misread the pygmentize help. In that case, getting this to work would probably involve creating a special wrapper script for pygmentize to create the styles. The wrapper script could be on the list of allowed commands and would basically just call pygmentize but capture its stdout and save to a file. Another, possibly simpler, option would be to just pregenerate all styles, so the style-generating code is never called. That should be possible in a server context like this.

[davidcarlisle]

On my own server I can add pygmentize to the restricted shell list (in fact I have done so) but if I were to try to get it added by default so minted could be used by normal setup without shell escape then I'd have to answer this question which has been in the default config for years

shell_escape_commands = \
bibtex,bibtex8,\
extractbb,\
gregorio,\
kpsewhich,\
makeindex,\
repstopdf,\
r-mpost,\
texosquery-jre8,\

% we'd like to allow:
% dvips - but external commands can be executed, need at least -R1.
% epspdf, ps2pdf, pstopdf - need to respect openout_any,
%   and gs -dSAFER must be used and check for shell injection with filenames.
% pygmentize - but is the filter feature insecure?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

So if we were to have a wrapper script that added -o perhaps it could also simply disable some of the options, I note even pygmentize --help warns about x option

Users should be very careful not to use this option with untrusted files, because it will import and run them.

so pygmentize would not be allowed in the default texlive list as is. This wouldn't be unusual: epstopdf and mpost have r wrappers with restricted functionality that are included here for the same reason.

@gpoore wrote

it might also be possible to handle some of this by having the server always include some sort of patching package that does a few \AtEndPreamble etc. checks and patches.

Yes I already do some of that (for other packages) although I try to avoid it too much as applying regex replace to incoming documents has a potential to mess things up and confuse the user greatly:-)

But while my immediate concern is a server setup I think this would be useful in general. minted is often the answer to code listing questions on sites like https://tex.stackexchange.com but it always concerns me when users are told "run this obscure fragment of tex code and use an option that gives it write access to all your files" in practice nothing bad happens and code gets syntax highlighted but the community generally seems to have a lack of awareness about why shell escape is disabled by default. If most uses of minted could be run in the default mode I think it would be a good thing.

[gpoore]

@davidcarlisle Thanks for pointing out the reason pygmentize isn't on the restricted shell list! I've actually been thinking about a wrapper for pygmentize to add some additional functionality (mostly simple things like autogobble that are trivial in Python but difficult in LaTeX--this currently involves its own \ShellEsc, increasing complexity and overhead). If a wrapper could also eliminate the need for --shell-escape, that would be really great!

[muzimuzhi]

(a little off-topic) One concern, will texlive accept to add a new command to shell_escape_commands which is not distributed with texlive itself and must be manually installed?

[davidcarlisle]

@muzimuzhi well I can ask Karl:-) epstopdf relies on gs and (on most platforms) that isn't part of texlive but needs to be in installed in the system. texosquery is in the list and needs java, so I don't think it is impossible to get this added if it is sufficiently restricted.

[gpoore]

Update: minted version 3.0 is now under development, thanks to a grant from the TeX Users Group. It will replace pygmentize with a new Python executable that is designed to be compatible with restricted shell escape. Initial beta releases of minted version 3.0 are expected by early 2024.