When used in latex2e hook "shipout/background", soft masks not shown on first page

[muzimuzhi]

Brief outline of the bug

When a tikzpicture is added to latex2e hook shipout/background, soft masks (e.g., shading, opacity setting) are not shown on the first page. This only happens with xelatex.

The output .xdv or .pdf shows that @pgfextgs is not added to the first page. .xdv can be dumped with dviasm, and .pdf can be transformed to a human editable form using mutool.

$ dviasm ltshipout-tikz-opacity-on-first-page.xdv | grep -nA 4 'pdf:obj'
26:        xxx: 'pdf:obj @pgfcolorspaces <<>>'
27-        xxx: 'pdf:put @resources << /ColorSpace @pgfcolorspaces >>'
28-        xxx: 'pdf:put @pgfcolorspaces <<  /pgfprgb [/Pattern /DeviceRGB]  >>'
29-      pop:
30-      push:
--
132:        xxx: 'pdf:obj @pgfextgs <<>>'
133-        xxx: 'pdf:put @resources << /ExtGState @pgfextgs /ColorSpace @pgfcolorspaces >>'
134-        xxx: 'pdf:put @pgfextgs << /[email protected] << /CA .5 >> >>'
135-        xxx: 'pdf:put @pgfextgs << /[email protected] << /ca .5 >> >>'
136-      pop:

This issue was initially reported to https://github.com/CTeX-org/forum/issues/173.

Minimal working example (MWE)

\documentclass{article}
\usepackage{tikz}

\AddToHook{shipout/background}{%
  \put (3cm,-10cm) {\tikz[opacity=.5] \node[scale=10] {test};}%
}

\begin{document}
  abc \newpage
  abc
\end{document}

[hmenke]

Resources allocation for the current page also takes place in shipout/background but is ordered before your drawing. I don't know whether it's possible to reorder them.

[hmenke]

Something like this

diff --git a/tex/generic/pgf/utilities/pgfutil-latex.def b/tex/generic/pgf/utilities/pgfutil-latex.def
index a21f3b4d6..a9d97b2fc 100644
--- a/tex/generic/pgf/utilities/pgfutil-latex.def
+++ b/tex/generic/pgf/utilities/pgfutil-latex.def
@@ -124,11 +124,12 @@
 \let\pgfutil@abe\pgfutil@empty%
 
 \pgfutil@ifl@t@r\fmtversion{2020/10/01}{%
-  \AddToHook{shipout/background}{%
+  \AddToHook{shipout/background}[pgf]{%
     \put(\hoffset-1in,\voffset-1in){%
       \pgfutil@abe\unhbox\pgfutil@abb\pgfutil@abc\global\let\pgfutil@abc\pgfutil@empty
     }%
   }%
+  \DeclareHookRule{shipout/background}{pgf}{after}{top-level}%
 }{%
   \RequirePackage{everyshi}
   \EveryShipout{%

but that doesn't fix the problem when everyshi is in use (or did that even have this problem?)

[muzimuzhi]

Resources allocation for the current page also takes place in shipout/background but is ordered before your drawing. I don't know whether it's possible to reorder them.

Since the above example works with both pdflatex and lualatex, I guess there is something wrong (or different) in and specific to pgf's (x)dvipdfmx backend.

[hmenke]

Yes, it looks like the pdftex and luatex driver unconditionally allocate these resources.

[u-fischer]

pgf doesn't need to set a label, as the code is loaded in a package it has a default label (pgfrcs). But the rule will not work with the next release, as then top-level will always be last. I need to think about this a bit (it is also relevant for the new pdf resource management).

[vitaminace33]

Hi! Does anyone know if this has been solved? Bug is still present in TeXLive 2021.58710.

Is there any workaround? I considered dropping the first page.

[u-fischer]

@vitaminace33 it works if you use the new pdf management:

\RequirePackage{pdfmanagement-testphase}
\DeclareDocumentMetadata{} 

\documentclass{article}
\usepackage{tikz}

\AddToHook{shipout/background}{%
  \put (3cm,-10cm) {\tikz[opacity=.5] \node[scale=10] {test};}%
}

\begin{document}
  abc \newpage
  abc
\end{document}

[vitaminace33]

Hi! Thx, however...

It works with plain text, like test, but not with graphics, \includegraphics[width=.9\linewidth]{LaTeX_logo.pdf}.

[u-fischer]

Well I don't have LaTeX_logo.pdf but I tested with

\AddToHook{shipout/background}{%
  \put (3cm,-10cm) {\tikz[opacity=.5] \node[scale=1] {\includegraphics{example-image-duck}};}%
}

and I get the right opacity: (the other duck is without opacity in the main page for comparision):

[vitaminace33]

Did you compile with XeLaTeX? For me it only works with pdfLaTeX.

Even this hook-less test fails...

\RequirePackage{pdfmanagement-testphase}
\DeclareDocumentMetadata{}

\documentclass{article}
\usepackage{tikz}

\begin{document}
\tikz\node[opacity=0.05]{
  \includegraphics[width=.9\linewidth]{LaTeX_logo.pdf}
};
\end{document}

LaTeX_logo.pdf from LaTeX_logo.svg using ctan:svg (Inkscape)

[u-fischer]

Well it fails. But it fails also if you move the image to page 2 so it is quite unrelated to the issue here which handles the special case of a page 1 problem.

[hmenke]

You LaTeX_logo.pdf is broken. When I feed it through Ghostscript

gs -o LaTeX_logo_fixed.pdf -sDEVICE=pdfwrite LaTeX_logo.pdf 

and then use LaTeX_logo_fixed.pdf it shows fine.

[vitaminace33]

@hmenke you are right, that worked. As @u-fischer says, I see now that this is unrelated to this bug, my apologies.

@hmenke how can I check if a pdf is broken? This was generated by svg/inkscape, as I said...

[vitaminace33]

BTW it did work with pdfLaTeX, hence my confusion. If it is broken, it should work with pdfLaTeX either, no?

[hmenke]

how can I check if a pdf is broken?

When included PDFs don't work correctly I just feed them through Ghostscript and see if it works then.

If it is broken, it should work with pdfLaTeX either, no?

That's something you have to ask the dvipdfmx developers. With some things I just don't know why they don't work there. My personal recommendation is to not use XeTeX if you don't have to. Unless you are typesetting CJK you are better served using LuaTeX. See also https://tex.stackexchange.com/questions/593031/what-are-the-downsides-of-using-xetex/593217#593217

[vitaminace33]

@hmenke Thx for the link, I think I will have to work with LuaTeX from now on...

Just as feedback, I ran some tests again and XeLaTeX requires pdfmanagement and the "fixed" pdf, while pdfLaTeX and LuaLaTeX work without pdfmanagement and the "broken" pdf file.

I resist to say that the pdf file is broken being generated by Inkscape, but it deserves a look.

Thanks guys for the help and enlightenment.

[u-fischer]

@hmenke I don't think that the pdf is "broken". It looks like a valid pdf. But it contains an explicit opacity setting and a /Group entry and xelatex and pdflatex handles this differently when embedding it. xelatex drops the /Group entry, and imho this is the reason why external opacity setting can no longer change the picture. When you feed the picture through ghostscript the opacity is removed and so it works again.