TikZ external does not work

[KunihiroWASA]

I tried to compile the following tex source

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{external}
\tikzset{
    external/system call={xelatex \tikzexternalcheckshellescape -halt-on-error -interaction=batchmode -jobname "\image" "\texsource"}}
\tikzexternalize % activate!

\begin{document}
\begin{tikzpicture}
    \draw (0,0) node [draw=black] {My text};
\end{tikzpicture}
\end{document}

by tectonic -X compile main.tex -Z shell-escape

However, I reciveded the following error message:

note: "version 2" Tectonic command-line interface activated
Running TeX ...
note: running shell command: `xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "main-figure0" "\def\tikzexternalrealjob{main}\input{main}"`
This is XeTeX, Version 3.141592653-2.6-0.999993 (TeX Live 2021) (preloaded format=xelatex)
 \write18 enabled.
entering extended mode
warning: command exited with error code 1
error: failed to execute the shell-escape command "xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "main-figure0" "\def\tikzexternalrealjob{main}\input{main}"": execution of the request failed
error: main.tex:11: Package tikz Error: Sorry, the system call 'xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "main-figure0" "\def\tikzexternalrealjob{main}\input{main}"' did NOT result in a usable output file 'main-figure0' (expected one of .pdf:.jpg:.jpeg:.png:.bmp:). Please verify that you have enabled system calls. For pdflatex, this is 'pdflatex -shell-escape'. Sometimes it is also named 'write 18' or something like that. Or maybe the command simply failed? Error messages can be found in 'main-figure0.log'. If you continue now, I'll try to typeset the picture.

See the tikz package documentation for explanation.
Type  H <return>  for immediate help
error: halted on potentially-recoverable error as specified

The above sorce actually works when compiling with xelatex -shell-escape main.tex This also works when removing external package and compiling with tectonic -X compile main.tex -Z shell-escape In addition, when I compile another complicated tex source, I received the following message:

...
===== 'mode=convert with system call': Invoking 'xelatex -shell-escape -halt-on
-error -interaction=batchmode -jobname "ext/kernel_outline-1" "\def\tikzexterna
lrealjob{main}\input{main}"' ========
error: failed to create file `/var/folders/s8/6qr2kxt14vlbf22ybs_vm2rh0000gn/T/.tmpQgK6F5/ext/kernel_outline-1.md5`
caused by: No such file or directory (os error 2)
error: failed to execute the shell-escape command "xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "ext/kernel_outline-1" "\def\tikzexternalrealjob{main}\input{main}"": execution of the request failed
...

Is this caused by my environment? My environment:

• Apple M1
• Tectonic 0.8.0
• xelatex: XeTeX 3.141592653-2.6-0.999993 (TeX Live 2021)

Anyway, thank you for providing such an excellent TeX engine. I love it.

[Neved4]

@KunihiroWASA Hello and thank you so much for reporting this 💜 I was able to reproduce that this MWE is not being compiled correctly with tectonic -p -Z shell-escape main.tex I have no idea how it can be solved but I can say it's not your environment ✌🏻

[phtalo]

Hi, I wanted to chip in and say that the same happens to me on Arch Linux. I get the following error:

note: running shell command: `xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "exampl-figure0" "\def\tikzexternalrealjob{exampl}\input{exampl}"`
This is XeTeX, Version 3.141592653-2.6-0.999994 (TeX Live 2022/Arch Linux) (preloaded format=xelatex)
 \write18 enabled.
entering extended mode
warning: command exited with error code 1
error: failed to execute the shell-escape command "xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "exampl-figure0" "\def\tikzexternalrealjob{exampl}\input{exampl}"": execution of the request failed
error: exampl.tex:26: Package tikz Error: Sorry, the system call 'xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "exampl-figure0" "\def\tikzexternalrealjob{exampl}\input{exampl}"' did NOT result in a usable output file 'exampl-figure0' (expected one of .pdf:.jpg:.jpeg:.png:.bmp:). Please verify that you have enabled system calls. For pdflatex, this is 'pdflatex -shell-escape'. Sometimes it is also named 'write 18' or something like that. Or maybe the command simply failed? Error messages can be found in 'exampl-figure0.log'. If you continue now, I'll try to typeset the picture.

Also here is my MWE:

\documentclass{article}
\usepackage{graphicx}
\usepackage[utf8]{inputenc}
\usepackage{tikz,pgfplots}
\pgfplotsset{compat=newest}
\usepgfplotslibrary{external} 
\tikzexternalize


\begin{document}
\begin{figure}
    \centering
    \begin{tikzpicture}
        \begin{axis}[]
            \addplot[] coordinates{
                (-1,1)
                (1,1)
            }; 
        \end{axis}
    \end{tikzpicture}
    \caption{}
\end{figure}

\end{document}

Thank you for your amazing work!

[Neved4]

@KunihiroWASA @FilCo98 I've checked again and none of your MWEs seem to be working with either pdfLaTex, XeLaTeX, LuaLaTeX or Tectonic. I'm running TeX Live 2022 currently and can't get none of the above to work.

Investigating a deep further the problem seems to lie in the usage of \usepgfplotslibrary, \usetikzlibrary and \tikzexternalize; meaningly, the external library of tikz. Removing those lines get the document to compile, but that won't render the desired behavior.

Perhaps something changed with tizk on recent versions and now external libs need to be invoked in a different way.

Could you give some tests under last TeX Live and come back to me?

Details

pdfTeX: 3.141592653-2.6-1.40.24 XeTeX: 3.141592653-2.6-0.999994 LuaHBTeX: 1.15.0 Tectonic: 0.11.0

[KunihiroWASA]

@mnrvwl Today, I also checked whether the external library and the minted package work, and I confirmed that both indeed work! To this end, I used -Z shell-escape-cwd=<path>, e.g., tectonic -X compile main.tex -Z shell-escape-cwd=./ This option is added on version 0.10.0 (https://github.com/tectonic-typesetting/tectonic/releases/tag/tectonic%400.10.0). Note that for my MWE, tectonic -X compile main.tex -Z shell-escape still doesn't work with XeTeX: 3.141592653-2.6-0.999994. Thank you very much!

[Neved4]

@KunihiroWASA you're absolutely right! 🎉 I couldn't get the examples running earlier 😅

I didn't know that TikZ external needed a cwd of ./ for tectonic to work, in LuaLaTeX/XeLaTeX it works with just a regular -shell-escape call, do you have a hint why this is? It'd be great to reflect this on Tectonic documentation for people trying to compile TikZ documents with it 💜

Let me know if we can consider the issue closed. I'm personally not familiar using tikz external libraries, do you know of any other examples that could fail?

[Neved4]

@FilCo98 check out the -Z shell-escape-cwd=./ option above, it will make your MWE work 🎊

[Neved4]

@pkgw is likely that future PGF/TikZ users will find that the shell-escape they're used to in pdfTeX/XeTeX/LuaTeX land is not enough for this use case and that they'll have to resort to shell-escape-cwd.

Not sure if it's something as big to deserve changing tectonic so it works without the cwd, to make the transition to tectonic more seamless and give new users a better experience.

We could hint it somewhere in documentation to guide new people towards that solution and avoid opening of duplicate issues, which would be the place best to add it?

[KunihiroWASA]

@mnrvwl I'm not sure what I'm saying is correct, but the main issue is that XeTeX does not have permission to access temporary files. Now, we can use shell-escape-cwd option. Although the appropriate cwd depends on the situation, I believe similar problems (related to the external library and the minted package) won't happen. I think we can close this issue unless @FilCo98 gives a new MWE.

[pkgw]

@mnrvwl Thank you for investigating this issue (and many others; I have been letting the backlog build up but aim to follow up on various things as best I can).

One place to document this could potentially be a new "Shell Escape" page of the "Concept Reference" section of the book. My goal is for that to be the sort of reference-information section of the book, which could/should define things like how Tectonic's shell-escape support works. The key thing being that by default Tectonic isolates the shell-escape process in a temporary directory, which is behavior that I would really like to preserve as the default if possible.

But also, one reality that we must accept is that people often don't read the manual :-) So it might also make sense to patch tikz, and other packages with similar issues, in the Tectonic bundle to warn people about its behavior.

Finally, it's possible that Tectonic's implementation of shell-escape isolation could be improved to just make this problem go away! I haven't sat down and analyzed why they're failing.

[Neved4]

@pkgw You're very welcome! Thanks to you and your work!

[phtalo]

First of all thank you for your amazing contributions. Unfortunately I am not yet able to compile the MWE with tectonic due to the fact I have indeed XeTeX 3.141592653-2.6-0.999994, which @KunihiroWASA mentioned. In my case the command getting pushed to shell is the same even with the flag -Z shell-escape-cwd=./ hence the error that follows is also the same.

I would be grateful if @KunihiroWASA (or anybody) could recommend me a version of XeTeX that proved effective or how to change engine run by tectonic.

[Neved4]

@phtalo You're welcome! As of changing engines I'm afraid it may not be easy since tectonic is its own engine 😅 You could try pull a different version of XeTeX and patch it into tectonic source code... if that'd work.

Compiling the following with tectonic -p -Z shell-escape -Z shell-escape-cwd=./ test.tex goes correctly for me:

\documentclass{article}
\usepackage{graphicx}
\usepackage{tikz, pgfplots}
  \pgfplotsset{compat=newest}
  \usepgfplotslibrary{external}
  \tikzexternalize

\begin{document}
  \begin{figure}
    \centering
    \begin{tikzpicture}
      \begin{axis}[]
        \addplot[] coordinates{(-1,1) (1,1)};
      \end{axis}
    \end{tikzpicture}
    \caption{}
  \end{figure}
\end{document}

Could you give it a try with tectonic 0.12.0 and tell me what the error message is?

[phtalo]

@mnrvwl I have tried running your MWE with your command, but on my machine (also using 0.12.0) it keeps failing. Interestingly, the call to system doesn't fallback to tectonic but calls xelatex, such that it has two different errors whether I have texlive-bin (which holds the xelatex binary) installed or not. In the case the latter is not installed, the error is that the command xelatex ... used by \tikzexternalize to generate the image fails since there is no binary with such name in path. In the other case it fails for an unclear error as it did before. May it be the case that I need to install both tectonic and texlive and all of the packages for both systems? Why doesn't tectonic call itself?

These are the errors:

• TeX Live not installed

===== 'mode=convert with system call': Invoking 'xelatex -shell-escape -halt-on
-error -interaction=batchmode -jobname "mwe-figure0" "\def\tikzexternalrealjob{
mwe}\input{mwe}"' ========
note: running shell command: `xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "mwe-figure0" "\def\tikzexternalrealjob{mwe}\input{mwe}"`
sh: line 1: xelatex: command not found
warning: command exited with error code 127
error: failed to execute the shell-escape command "xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "mwe-figure0" "\def\tikzexternalrealjob{mwe}\input{mwe}"": execution of the request failed

! Package tikz Error: Sorry, the system call 'xelatex -shell-escape -halt-on-er
ror -interaction=batchmode -jobname "mwe-figure0" "\def\tikzexternalrealjob{mwe
}\input{mwe}"' did NOT result in a usable output file 'mwe-figure0' (expected o
ne of .pdf:.jpg:.jpeg:.png:.bmp:). Please verify that you have enabled system c
alls. For pdflatex, this is 'pdflatex -shell-escape'. Sometimes it is also name
d 'write 18' or something like that. Or maybe the command simply failed? Error 
messages can be found in 'mwe-figure0.log'. If you continue now, I'll try to ty
peset the picture.

See the tikz package documentation for explanation.
error: mwe.tex:15: Package tikz Error: Sorry, the system call 'xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "mwe-figure0" "\def\tikzexternalrealjob{mwe}\input{mwe}"' did NOT result in a usable output file 'mwe-figure0' (expected one of .pdf:.jpg:.jpeg:.png:.bmp:). Please verify that you have enabled system calls. For pdflatex, this is 'pdflatex -shell-escape'. Sometimes it is also named 'write 18' or something like that. Or maybe the command simply failed? Error messages can be found in 'mwe-figure0.log'. If you continue now, I'll try to typeset the picture.

• TeX Live installed

===== 'mode=convert with system call': Invoking 'xelatex -shell-escape -halt-on
-error -interaction=batchmode -jobname "mwe-figure0" "\def\tikzexternalrealjob{
mwe}\input{mwe}"' ========
note: running shell command: `xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "mwe-figure0" "\def\tikzexternalrealjob{mwe}\input{mwe}"`
This is XeTeX, Version 3.141592653-2.6-0.999994 (TeX Live 2022/Arch Linux) (preloaded format=xelatex)
 \write18 enabled.
entering extended mode
warning: command exited with error code 1
error: failed to execute the shell-escape command "xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "mwe-figure0" "\def\tikzexternalrealjob{mwe}\input{mwe}"": execution of the request failed

! Package tikz Error: Sorry, the system call 'xelatex -shell-escape -halt-on-er
ror -interaction=batchmode -jobname "mwe-figure0" "\def\tikzexternalrealjob{mwe
}\input{mwe}"' did NOT result in a usable output file 'mwe-figure0' (expected o
ne of .pdf:.jpg:.jpeg:.png:.bmp:). Please verify that you have enabled system c
alls. For pdflatex, this is 'pdflatex -shell-escape'. Sometimes it is also name
d 'write 18' or something like that. Or maybe the command simply failed? Error 
messages can be found in 'mwe-figure0.log'. If you continue now, I'll try to ty
peset the picture.

See the tikz package documentation for explanation.
error: mwe.tex:15: Package tikz Error: Sorry, the system call 'xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "mwe-figure0" "\def\tikzexternalrealjob{mwe}\input{mwe}"' did NOT result in a usable output file 'mwe-figure0' (expected one of .pdf:.jpg:.jpeg:.png:.bmp:). Please verify that you have enabled system calls. For pdflatex, this is 'pdflatex -shell-escape'. Sometimes it is also named 'write 18' or something like that. Or maybe the command simply failed? Error messages can be found in 'mwe-figure0.log'. If you continue now, I'll try to typeset the picture.

[pkgw]

Interesting! It looks like tikz embeds code that re-invokes the TeX engine within the document processing run. In Tectonic we could handle that in some clever way, but unfortunately the only way to make that happen would be to either add some very specialized code to detect these scenarios, or patch the package in the bundle creation code (and still have some specialized recursion code). Neither of those things has happened yet.

I have an idea that Tectonic's document workspace support could one day support this sort of thing in a somewhat cleaner way, but that support is still a bit of a sketch.