Examples in the manual as MWE

[pgf-tikz-bot]

Migrated from SourceForge Author: krichter722 Timestamp: 2018-07-03 19:40:40.026000

I'm aware that the following is a huge pile of work, however I hope that you devs don't close it immediately because of that. It's intended to make this awesome software easier to use for beginners and trained users which tend to forget or can't keep up with the latest major changes:

It'd be nice to have a link to a Short, Self Contained, Correct (Compilable), Example for every code sample so that if you click on a link to a code sample like

\begin{tikzpicture}
\draw[step=.5cm,gray,very thin] (-1.4,-1.4) grid(1.4,1.4);
\draw (-1.5,0) -- (1.5,0);
\draw (0,-1.5) -- (0,1.5);
\draw (0,0) circle[radius=1cm];
\draw (3mm,0mm) arc[start angle=0,end angle=30,radius=3mm];
\end{tikzpicture}

on page xy you're redirected to the plain text file https://sourceforge.net/p/pgf/wiki/samples/001.tex which contains a SSCCE which can be directly passed to the compiler.

The manual describes what the user needs to do in a satisfying way, starting with

\documentclass{article}% say
\usepackage{tikz}
\begin{document}
We are working on
\begin{tikzpicture}
\draw (-1.5,0) -- (1.5,0);
\draw (0,-1.5) -- (0,1.5);
\end{tikzpicture}.
\end{document}

However, it's sometimes hard to follow all the packages which have been introduced and settings which have been changed over the course of the chapter where the snippet is shown (which can be dozens of pages long). The plain text links would avoid this and reduce the number of mistakes can make while using the snippet almost to zero.

This would also improve the incapacity of most PDF viewers to copy code accurate, i.e. without introducing unwanted linebreaks and replacing spaces with linebreaks, even though that's not PGF's job.

If you think that's a good idea regardless of the effort, then you might want to keep those SSCCEs coming piece by piece, maybe through merge requests. I can provide some, however, I often find myself trying to get the snippets to compile for hours which is the main reason why I'm suggesting this.

The SSCCEs could be provided for LaTeX, TeX, ConTeX and LuaLaTeX.

[hmenke]

@tallmarmot @Mo-Gul I have now more or less finished the extractor script. You can give it a go using the following bash script

#!/bin/bash
mkdir -p mwe
texlua extract.lua text-en/ mwe/
cd mwe
n=1
N=$(ls -1q *.tex | wc -l)
for i in *.tex; do
    echo -n "Processing $n/$N $i"
    if [ -f "${i%%.tex}.pdf" ]; then
        echo -n " Skipped." # skip existing
    else
        lualatex -interaction=batchmode -halt-on-error "$i" > /dev/null
        if [ "x$?" == "x1" ]; then
            echo -n " Failed!"
        fi
    fi
    echo ""
    n=$((n+1))
done

Save this script as, say build.sh in doc/generic/pgf/ next to extract.lua. Then you can run it using bash build.sh. Since this typesets all the examples individually, it takes a lot of time. The output will look like

[...snip...]
Processing 59/2128 pgfmanual-en-base-animations-8.tex
Processing 60/2128 pgfmanual-en-base-animations-9.tex
Processing 61/2128 pgfmanual-en-base-arrows-17.tex Failed!
Processing 62/2128 pgfmanual-en-base-arrows-8.tex Failed!
Processing 63/2128 pgfmanual-en-base-arrows-9.tex Failed!
Processing 64/2128 pgfmanual-en-base-decorations-11.tex
Processing 65/2128 pgfmanual-en-base-decorations-12.tex
Processing 66/2128 pgfmanual-en-base-decorations-1.tex
[...snip...]

As you can see not all of the examples typeset out-of-the-box. For the failed ones, you can inspect the log to find the error. As you can see from the extractor script

https://github.com/pgf-tikz/pgf/blob/710bd7fceaa8735fc3c0fb42e509dd919308cb0d/doc/generic/pgf/extract.lua#L2-L18

I currently just include all libraries in the preamble. This of course has to change. We only want to use the libraries which are necessary for each example.

[Mo-Gul]

@hmenke, thank you very much for providing the extractor script. I gave it a try and it works perfectly fine. And you are right, it takes a lot of time. Since I don't have clue about Bash scripts: Is there a chance to parallize the compilation?

@tallmarmot, what do you think about the extractor script? Do you see a chance to modify it according to "our understanding"?

[hmenke]

Here is a script which runs in parallel on all available processors.

#!/bin/bash
mkdir -p mwe
texlua extract.lua text-en/ mwe/
cd mwe

ls -1q *.tex | sort | xargs -I @ -P `nproc` bash -c '
    i="@"
    str="Building $i"
    if [ -f "${i%%.tex}.pdf" ]; then
        str="$str Skipped."
    else
        lualatex -interaction=batchmode -halt-on-error "$i" > /dev/null
        if [ "x$?" == "x1" ]; then
            str="$str Failed!"
        fi
    fi
    echo $str'

[Mo-Gul]

Works like a charm. Thank you for that. The only downside is that now one doesn't have a clue about the progress. But I think one has to live with this minor issue.

[hmenke]

I could add a counter again but that would require some sort of locking mechanism because xargs runs in parallel and I don't think bash variables are atomic.

[hmenke]

@Mo-Gul I needed this for something else, so I had a look at how to lock file descriptors in Bash. Here is a new script with counter.

#!/bin/bash
mkdir -p mwe
texlua extract.lua text-en/ mwe/
cd mwe

touch /tmp/pgfmanuallock
echo 1 > /tmp/pgfmanualcount
ls -1q *.tex | sort | xargs -I @ -P `nproc` bash -c '
    i="@"
    { flock -x 200;
        n=$(cat /tmp/pgfmanualcount);
        echo $((n+1)) > /tmp/pgfmanualcount;
    } 200> /tmp/pgfmanuallock
    str="Building $n/$(ls -1q *.tex | wc -l) $i"
    if [ -f "${i%%.tex}.pdf" ]; then
        str="$str Skipped."
    else
        lualatex -interaction=batchmode -halt-on-error "$i" > /dev/null
        if [ "x$?" == "x1" ]; then
            str="$str Failed!"
        fi
    fi
    echo $str'

[hmenke]

I have also run the script to see how many examples currently fail even in the presence of all libraries and there are luckily only few (181 out of 2128).

grep Failed build.log

Building 62/2128 pgfmanual-en-base-arrows-8.tex Failed!
Building 61/2128 pgfmanual-en-base-arrows-17.tex Failed!
Building 63/2128 pgfmanual-en-base-arrows-9.tex Failed!
Building 75/2128 pgfmanual-en-base-images-3.tex Failed!
Building 77/2128 pgfmanual-en-base-images-4.tex Failed!
Building 128/2128 pgfmanual-en-base-patterns-4.tex Failed!
Building 133/2128 pgfmanual-en-base-plots-6.tex Failed!
Building 161/2128 pgfmanual-en-base-quick-3.tex Failed!
Building 217/2128 pgfmanual-en-base-transformations-31.tex Failed!
Building 218/2128 pgfmanual-en-base-transformations-32.tex Failed!
Building 219/2128 pgfmanual-en-base-transformations-33.tex Failed!
Building 220/2128 pgfmanual-en-base-transformations-34.tex Failed!
Building 221/2128 pgfmanual-en-base-transformations-35.tex Failed!
Building 222/2128 pgfmanual-en-base-transformations-36.tex Failed!
Building 235/2128 pgfmanual-en-base-transparency-11.tex Failed!
Building 245/2128 pgfmanual-en-base-transparency-9.tex Failed!
Building 293/2128 pgfmanual-en-dv-axes-76.tex Failed!
Building 294/2128 pgfmanual-en-dv-axes-78.tex Failed!
Building 295/2128 pgfmanual-en-dv-axes-79.tex Failed!
Building 296/2128 pgfmanual-en-dv-axes-80.tex Failed!
Building 297/2128 pgfmanual-en-dv-axes-81.tex Failed!
Building 301/2128 pgfmanual-en-dv-axes-85.tex Failed!
Building 303/2128 pgfmanual-en-dv-axes-87.tex Failed!
Building 304/2128 pgfmanual-en-dv-axes-88.tex Failed!
Building 305/2128 pgfmanual-en-dv-axes-89.tex Failed!
Building 308/2128 pgfmanual-en-dv-axes-92.tex Failed!
Building 323/2128 pgfmanual-en-dv-main-15.tex Failed!
Building 339/2128 pgfmanual-en-dv-stylesheets-10.tex Failed!
Building 340/2128 pgfmanual-en-dv-stylesheets-11.tex Failed!
Building 343/2128 pgfmanual-en-dv-stylesheets-6.tex Failed!
Building 344/2128 pgfmanual-en-dv-stylesheets-7.tex Failed!
Building 345/2128 pgfmanual-en-dv-stylesheets-9.tex Failed!
Building 354/2128 pgfmanual-en-gd-algorithm-layer-1.tex Failed!
Building 355/2128 pgfmanual-en-gd-algorithm-layer-2.tex Failed!
Building 480/2128 pgfmanual-en-library-circuits-25.tex Failed!
Building 482/2128 pgfmanual-en-library-circuits-27.tex Failed!
Building 500/2128 pgfmanual-en-library-circuits-47.tex Failed!
Building 502/2128 pgfmanual-en-library-circuits-49.tex Failed!
Building 515/2128 pgfmanual-en-library-circuits-61.tex Failed!
Building 516/2128 pgfmanual-en-library-circuits-62.tex Failed!
Building 518/2128 pgfmanual-en-library-circuits-64.tex Failed!
Building 520/2128 pgfmanual-en-library-circuits-66.tex Failed!
Building 521/2128 pgfmanual-en-library-circuits-67.tex Failed!
Building 522/2128 pgfmanual-en-library-circuits-68.tex Failed!
Building 523/2128 pgfmanual-en-library-circuits-69.tex Failed!
Building 525/2128 pgfmanual-en-library-circuits-70.tex Failed!
Building 526/2128 pgfmanual-en-library-circuits-71.tex Failed!
Building 527/2128 pgfmanual-en-library-circuits-72.tex Failed!
Building 528/2128 pgfmanual-en-library-circuits-73.tex Failed!
Building 529/2128 pgfmanual-en-library-circuits-74.tex Failed!
Building 670/2128 pgfmanual-en-library-fpu-2.tex Failed!
Building 731/2128 pgfmanual-en-library-perspective-10.tex Failed!
Building 732/2128 pgfmanual-en-library-perspective-11.tex Failed!
Building 735/2128 pgfmanual-en-library-perspective-14.tex Failed!
Building 740/2128 pgfmanual-en-library-perspective-4.tex Failed!
Building 741/2128 pgfmanual-en-library-perspective-5.tex Failed!
Building 742/2128 pgfmanual-en-library-perspective-6.tex Failed!
Building 743/2128 pgfmanual-en-library-perspective-7.tex Failed!
Building 744/2128 pgfmanual-en-library-perspective-8.tex Failed!
Building 745/2128 pgfmanual-en-library-perspective-9.tex Failed!
Building 782/2128 pgfmanual-en-library-rdf-23.tex Failed!
Building 783/2128 pgfmanual-en-library-rdf-24.tex Failed!
Building 811/2128 pgfmanual-en-library-shapes-12.tex Failed!
Building 810/2128 pgfmanual-en-library-shapes-11.tex Failed!
Building 814/2128 pgfmanual-en-library-shapes-15.tex Failed!
Building 816/2128 pgfmanual-en-library-shapes-17.tex Failed!
Building 820/2128 pgfmanual-en-library-shapes-21.tex Failed!
Building 822/2128 pgfmanual-en-library-shapes-23.tex Failed!
Building 824/2128 pgfmanual-en-library-shapes-25.tex Failed!
Building 826/2128 pgfmanual-en-library-shapes-27.tex Failed!
Building 829/2128 pgfmanual-en-library-shapes-2.tex Failed!
Building 833/2128 pgfmanual-en-library-shapes-33.tex Failed!
Building 840/2128 pgfmanual-en-library-shapes-3.tex Failed!
Building 841/2128 pgfmanual-en-library-shapes-40.tex Failed!
Building 844/2128 pgfmanual-en-library-shapes-43.tex Failed!
Building 846/2128 pgfmanual-en-library-shapes-45.tex Failed!
Building 851/2128 pgfmanual-en-library-shapes-4.tex Failed!
Building 852/2128 pgfmanual-en-library-shapes-50.tex Failed!
Building 856/2128 pgfmanual-en-library-shapes-54.tex Failed!
Building 861/2128 pgfmanual-en-library-shapes-59.tex Failed!
Building 862/2128 pgfmanual-en-library-shapes-5.tex Failed!
Building 866/2128 pgfmanual-en-library-shapes-63.tex Failed!
Building 869/2128 pgfmanual-en-library-shapes-66.tex Failed!
Building 871/2128 pgfmanual-en-library-shapes-68.tex Failed!
Building 874/2128 pgfmanual-en-library-shapes-70.tex Failed!
Building 875/2128 pgfmanual-en-library-shapes-71.tex Failed!
Building 882/2128 pgfmanual-en-library-shapes-78.tex Failed!
Building 885/2128 pgfmanual-en-library-shapes-80.tex Failed!
Building 887/2128 pgfmanual-en-library-shapes-82.tex Failed!
Building 888/2128 pgfmanual-en-library-shapes-83.tex Failed!
Building 890/2128 pgfmanual-en-library-shapes-85.tex Failed!
Building 893/2128 pgfmanual-en-library-shapes-88.tex Failed!
Building 899/2128 pgfmanual-en-library-shapes-93.tex Failed!
Building 904/2128 pgfmanual-en-library-shapes-98.tex Failed!
Building 1099/2128 pgfmanual-en-module-parser-2.tex Failed!
Building 1100/2128 pgfmanual-en-module-parser-3.tex Failed!
Building 1161/2128 pgfmanual-en-pgfkeysfiltered-10.tex Failed!
Building 1165/2128 pgfmanual-en-pgfkeysfiltered-2.tex Failed!
Building 1166/2128 pgfmanual-en-pgfkeysfiltered-5.tex Failed!
Building 1167/2128 pgfmanual-en-pgfkeysfiltered-6.tex Failed!
Building 1168/2128 pgfmanual-en-pgfkeysfiltered-7.tex Failed!
Building 1169/2128 pgfmanual-en-pgfkeysfiltered-9.tex Failed!
Building 1170/2128 pgfmanual-en-pgfsys-animations-10.tex Failed!
Building 1171/2128 pgfmanual-en-pgfsys-animations-11.tex Failed!
Building 1172/2128 pgfmanual-en-pgfsys-animations-12.tex Failed!
Building 1173/2128 pgfmanual-en-pgfsys-animations-13.tex Failed!
Building 1174/2128 pgfmanual-en-pgfsys-animations-14.tex Failed!
Building 1175/2128 pgfmanual-en-pgfsys-animations-15.tex Failed!
Building 1176/2128 pgfmanual-en-pgfsys-animations-17.tex Failed!
Building 1177/2128 pgfmanual-en-pgfsys-animations-18.tex Failed!
Building 1178/2128 pgfmanual-en-pgfsys-animations-19.tex Failed!
Building 1179/2128 pgfmanual-en-pgfsys-animations-20.tex Failed!
Building 1180/2128 pgfmanual-en-pgfsys-animations-21.tex Failed!
Building 1181/2128 pgfmanual-en-pgfsys-animations-22.tex Failed!
Building 1182/2128 pgfmanual-en-pgfsys-animations-3.tex Failed!
Building 1183/2128 pgfmanual-en-pgfsys-animations-4.tex Failed!
Building 1184/2128 pgfmanual-en-pgfsys-animations-5.tex Failed!
Building 1185/2128 pgfmanual-en-pgfsys-animations-6.tex Failed!
Building 1186/2128 pgfmanual-en-pgfsys-animations-7.tex Failed!
Building 1187/2128 pgfmanual-en-pgfsys-animations-8.tex Failed!
Building 1188/2128 pgfmanual-en-pgfsys-animations-9.tex Failed!
Building 1217/2128 pgfmanual-en-tikz-actions-38.tex Failed!
Building 1218/2128 pgfmanual-en-tikz-actions-39.tex Failed!
Building 1220/2128 pgfmanual-en-tikz-actions-40.tex Failed!
Building 1231/2128 pgfmanual-en-tikz-actions-50.tex Failed!
Building 1357/2128 pgfmanual-en-tikz-arrows-38.tex Failed!
Building 1358/2128 pgfmanual-en-tikz-arrows-39.tex Failed!
Building 1360/2128 pgfmanual-en-tikz-arrows-40.tex Failed!
Building 1361/2128 pgfmanual-en-tikz-arrows-41.tex Failed!
Building 1362/2128 pgfmanual-en-tikz-arrows-42.tex Failed!
Building 1363/2128 pgfmanual-en-tikz-arrows-43.tex Failed!
Building 1364/2128 pgfmanual-en-tikz-arrows-44.tex Failed!
Building 1365/2128 pgfmanual-en-tikz-arrows-45.tex Failed!
Building 1366/2128 pgfmanual-en-tikz-arrows-46.tex Failed!
Building 1367/2128 pgfmanual-en-tikz-arrows-47.tex Failed!
Building 1368/2128 pgfmanual-en-tikz-arrows-48.tex Failed!
Building 1369/2128 pgfmanual-en-tikz-arrows-49.tex Failed!
Building 1371/2128 pgfmanual-en-tikz-arrows-50.tex Failed!
Building 1373/2128 pgfmanual-en-tikz-arrows-52.tex Failed!
Building 1372/2128 pgfmanual-en-tikz-arrows-51.tex Failed!
Building 1413/2128 pgfmanual-en-tikz-coordinates-19.tex Failed!
Building 1546/2128 pgfmanual-en-tikz-graphs-153.tex Failed!
Building 1665/2128 pgfmanual-en-tikz-matrices-23.tex Failed!
Building 1729/2128 pgfmanual-en-tikz-pics-11.tex Failed!
Building 1730/2128 pgfmanual-en-tikz-pics-12.tex Failed!
Building 1732/2128 pgfmanual-en-tikz-pics-16.tex Failed!
Building 1735/2128 pgfmanual-en-tikz-pics-2.tex Failed!
Building 1736/2128 pgfmanual-en-tikz-pics-3.tex Failed!
Building 1737/2128 pgfmanual-en-tikz-pics-4.tex Failed!
Building 1738/2128 pgfmanual-en-tikz-pics-5.tex Failed!
Building 1739/2128 pgfmanual-en-tikz-pics-6.tex Failed!
Building 1740/2128 pgfmanual-en-tikz-pics-7.tex Failed!
Building 1811/2128 pgfmanual-en-tikz-shapes-121.tex Failed!
Building 1812/2128 pgfmanual-en-tikz-shapes-122.tex Failed!
Building 1865/2128 pgfmanual-en-tikz-shapes-55.tex Failed!
Building 1949/2128 pgfmanual-en-tikz-transparency-22.tex Failed!
Building 2045/2128 pgfmanual-en-tutorial-chains-10.tex Failed!
Building 2046/2128 pgfmanual-en-tutorial-chains-11.tex Failed!
Building 2047/2128 pgfmanual-en-tutorial-chains-12.tex Failed!
Building 2048/2128 pgfmanual-en-tutorial-chains-13.tex Failed!
Building 2049/2128 pgfmanual-en-tutorial-chains-14.tex Failed!
Building 2050/2128 pgfmanual-en-tutorial-chains-15.tex Failed!
Building 2051/2128 pgfmanual-en-tutorial-chains-16.tex Failed!
Building 2053/2128 pgfmanual-en-tutorial-chains-18.tex Failed!
Building 2054/2128 pgfmanual-en-tutorial-chains-19.tex Failed!
Building 2056/2128 pgfmanual-en-tutorial-chains-20.tex Failed!
Building 2058/2128 pgfmanual-en-tutorial-chains-22.tex Failed!
Building 2059/2128 pgfmanual-en-tutorial-chains-23.tex Failed!
Building 2060/2128 pgfmanual-en-tutorial-chains-24.tex Failed!
Building 2062/2128 pgfmanual-en-tutorial-chains-26.tex Failed!
Building 2061/2128 pgfmanual-en-tutorial-chains-25.tex Failed!
Building 2065/2128 pgfmanual-en-tutorial-chains-4.tex Failed!
Building 2066/2128 pgfmanual-en-tutorial-chains-5.tex Failed!
Building 2067/2128 pgfmanual-en-tutorial-chains-6.tex Failed!
Building 2068/2128 pgfmanual-en-tutorial-chains-7.tex Failed!
Building 2069/2128 pgfmanual-en-tutorial-chains-8.tex Failed!
Building 2070/2128 pgfmanual-en-tutorial-chains-9.tex Failed!
Building 2087/2128 pgfmanual-en-tutorial-map-11.tex Failed!
Building 2090/2128 pgfmanual-en-tutorial-map-15.tex Failed!
Building 2091/2128 pgfmanual-en-tutorial-map-16.tex Failed!
Building 2128/2128 pgfmanual-en-xxcolor-1.tex Failed!

That means for all the other examples you can go ahead and check which libraries are actually needed. Let's choose for example

pgfmanual-en-base-decorations-1.tex

\documentclass{article}
\usepackage{fp,pgf,tikz,xcolor}
\usetikzlibrary{3d,arrows,arrows.spaced,arrows.meta,bending,babel,calc,
  fit,patterns,plotmarks,shapes.geometric,shapes.misc,shapes.symbols,
  shapes.arrows,shapes.callouts,shapes.multipart,shapes.gates.logic.US,
  shapes.gates.logic.IEC,circuits.logic.US,circuits.logic.IEC,
  circuits.logic.CDH,circuits.ee.IEC,datavisualization,
  datavisualization.polar,datavisualization.formats.functions,er,automata,
  backgrounds,chains,topaths,trees,petri,mindmap,matrix,calendar,folding,
  fadings,shadings,spy,through,turtle,positioning,scopes,
  decorations.fractals,decorations.shapes,decorations.text,
  decorations.pathmorphing,decorations.pathreplacing,decorations.footprints,
  decorations.markings,shadows,lindenmayersystems,intersections,
  fixedpointarithmetic,fpu,svg.path,external,graphs,graphs.standard,quotes,
  math,angles,views,animations,rdf,perspective}
\usetikzlibrary{graphdrawing}
\usegdlibrary{trees,circular,layered,examples,force,phylogenetics,routing}
\begin{document}
\makeatletter
\tikz \draw decorate[decoration=zigzag] {(0,0) -- (3,0)};
\end{document}

For this we only need \usetikzlibrary{decorations,decorations.pathmorphing}, so I go to the corresponding location in the manual and add the libraries to the options of the codeexample. https://github.com/pgf-tikz/pgf/blob/4c3ab05b40608c4b2f499b9aedae0fcf30c738fa/doc/generic/pgf/text-en/pgfmanual-en-base-decorations.tex#L27-L29 It will show up in the manual like this: For now this is only a preliminary solution and the details might change in the future, but if you could add all the required libraries for each codeexample that would be a great help. Right now, libraries/tikz and libraries/pgf are available, so it is best to start with those examples which do not require \usegdlibrary or \usepackage in the preamble.

[Mo-Gul]

@hmenke, great work (as always). It seems that the both keys libraries/tikz and libraries/pgf are not enough (for automated testing). We need another key that holds \tikzset stuff and doesn't show up in the manual. This is needed e.g. for the tutorial examples. Have a look at e.g. https://github.com/pgf-tikz/pgf/blob/4c3ab05b40608c4b2f499b9aedae0fcf30c738fa/doc/generic/pgf/text-en/pgfmanual-en-tutorial-chains.tex#L253-L266 (which corresponds to the extracted pgfmanual-en-tutorial-chains-4.tex) where the terminal style is no longer defined/repeated ...

The definitions of the styles are given at the beginning of the file https://github.com/pgf-tikz/pgf/blob/4c3ab05b40608c4b2f499b9aedae0fcf30c738fa/doc/generic/pgf/text-en/pgfmanual-en-tutorial-chains.tex#L25-L50 Maybe you have an idea how to add grab/store them and then either state the needed styles or just all of them. But in the latter case there is needed some kind of a reset thus that not all following examples contain stuff that is not needed any more.

To make it not too easy there is a need to grab several \tikzsets. E.g. for the given example the definition of the styles is updated later in the file https://github.com/pgf-tikz/pgf/blob/4c3ab05b40608c4b2f499b9aedae0fcf30c738fa/doc/generic/pgf/text-en/pgfmanual-en-tutorial-chains.tex#L336-L338

Also we need to grab definitions https://github.com/pgf-tikz/pgf/blob/4c3ab05b40608c4b2f499b9aedae0fcf30c738fa/doc/generic/pgf/text-en/pgfmanual-en-tutorial-chains.tex#L542-L573 (which corresponds to the extracted pgfmanual-en-tutorial-chains-15.tex).

If my above suggestion cannot be implemented (easily), maybe there is a possibility to abuse the codeexamples pre key (see previous code snippet) for our needs?

[ghost]

We also need something that allows us to load pgfmodules (e.g. for nonlinear transformations) unless we agree on changing the corresponding libraries in such a way that they always load the corresponding modules automatically. Whether or not we need the \tikzset I do not have a strong opinion on because after all at the corresponding place in the manual this is explained very well. However, one conceivable way to go would be a key "other" which can be used to add everything else like \usepgfmodule or \tikzset such that all commands run through. (If I would know how I can check out the current version of the repository and commit changes, I would be able to test all the above. Is there a tutorial which is not entirely stupid and allows one to see how that works?)

[Mo-Gul]

@tallmarmot, you are right. Just having a look at the manual you don't need to "catch" the \tikzsets. But for the automatic testing using the above bash script you need them ... (That I already tried to say in my previous comment, but maybe it wasn't clear enough.)

[Mo-Gul]

Ok, for now I use the pre key and added all styles and definitions. This works fine. Only drawback is that styles and definitions containing hashes need to be provided in double form, e.g.

https://github.com/pgf-tikz/pgf/blob/5d179300a9dcc879414cb86bca28d6a3f2fc0a08/doc/generic/pgf/text-en/pgfmanual-en-tutorial-chains.tex#L676-L679

But of course then also in the extracted files the hashes are doubled, but need to be in single form. In all these cases I added comment lines on top, thus the user has an idea what he has to do to make the example work. Of course this doesn't help for the automatic testing. These lines start with % !!! (and in the code of course with pre={ % !!!).

Also I have found out that

https://github.com/pgf-tikz/pgf/blob/5d179300a9dcc879414cb86bca28d6a3f2fc0a08/doc/generic/pgf/text-en/pgfmanual-en-xxcolor.tex#L36-L50

isn't extracted. When I remove the optional argument it is. Is this a bug in extract.lua?

[hmenke]

width={4cm} should help as a workaround.

[hmenke]

Fixed in 0a6f19790dd3a99f8a9a59479416aa21954f7334

[hmenke]

Following the request by @tallmarmot I have changed the way libraries are listed. I have removed the libraries/tikz and libraries/pgf options and replaced it with a single option preamble in which you can put anything that should go in the preamble, e.g. https://github.com/pgf-tikz/pgf/blob/36803f77cc11e6a04269e53d26c6f9f396a5cfac/doc/generic/pgf/text-en/pgfmanual-en-base-decorations.tex#L27-L29 Right now the content of preamble is simply piped through \detokenize. I might add more elaborate formatting in the future but I think this approach is flexible enough for now. Remember, the main objective currently is to identify which libraries are required to make each example compile. Pretty printing in the manual is only secondary at this point.

[Mo-Gul]

@hmenke, it would be nice if you could add some more features

1. Possibility to empty stuff collected by setup code.
2. An option for codeexample that the collected setup code can be ignored in this codeexample.

To 1: Background is that sometimes styles or definitions are only needed for a section and not for the whole chapter. Simply enclosing this codeexample in (curly) braces like

{
\begin{codeexample}[setup code,hidden]
    ....
\end{codeexample}
...
}

will nonetheless add this collected stuff to a codeexample after the closing (curly) brace.

To 2: Sometimes there are simple examples to introduce some stuff before this is applied to the "real" example. Of course for these all the collected stuff is not needed and might confuse users when they see all these definitions.

[hmenke]

You have clearly not understood what this task is about. Let me make this clear: The aim is to annotate the examples in the manual with the necessary \usetikzlibrary etc. such that a user reading the manual can easily infer which libraries and packages to include in their document to obtain the demonstrated functionality. The aim is not to make the examples copy-pastable or anything like that. Also the purpose of the extract.lua is not to generate the most minimal example from the code in the manual, but to extract a compilable example that we can use for regression testing.

[Mo-Gul]

Understood. Step 1 is: Just annotate such that codeexamples are compilable after extraction.

[Mo-Gul]

There is another source for the manual files used for the graphdrawing library. They are found at https://github.com/pgf-tikz/pgf/tree/master/tex/generic/pgf/graphdrawing/lua/pgf/gd.

You decide if its best to adapt extract.lua such that it grabs these examples, too or if you add another script in that location only for these examples.

[Mo-Gul]

Ok, I am through the codeexamples found in https://github.com/pgf-tikz/pgf/tree/master/doc/generic/pgf/text-en. There are 171 remaining instances. Shall I first merge the branch I created to the master and then do a pull request or directly do a pull request?

Some comments to the remaining instances:

• [x] 70 are in in pgfmanual-en-dv-stylesheets.tex where https://github.com/Mo-Gul/pgf/blob/f1b8890a941d5f89e36f7cd6ca98001faf4b998d/doc/generic/pgf/text-en/pgfmanual-en-dv-stylesheets.tex#L78-L99 is needed in (almost) all codeexamples as pre. So I guess you need to add another handler for that.
• [x] 12 are in pgfmanual-en-dv-axes.tex where https://github.com/Mo-Gul/pgf/blob/da503508e88f3d377475ff2f62634da9429bd868/doc/generic/pgf/text-en/pgfmanual-en-dv-axes.tex#L2717-L2750 is needed in the remaining codeexamples as pre. --> see previous item.
• [x] 43 are in pgfmanual-en-pgfsys-animations.tex
• [ ] some (most likely) cannot work on their own, e.g. the ones containing remember picture/overlay

And here some comments on the resulting pgfmanual.pdf:

• [ ] the preamble stuff is e.g. shown as \usetikzlibrary {...} --> remove the space after the command
• [ ] the setup code,hidden isn't added/appended to the preamble stuff and thus doesn't show up. Is this intended? I would say sometimes this is useful, sometimes not. --> Add another option/key--value?

[hmenke]

1. No, that is not the only problem. BTW, setup code does the same thing in extract.lua as putting it into the pre key of every example. There are a lot more interdependencies though, e.g. this https://github.com/Mo-Gul/pgf/blob/f1b8890a941d5f89e36f7cd6ca98001faf4b998d/doc/generic/pgf/text-en/pgfmanual-en-dv-stylesheets.tex#L322-L329
2. Same problem as 1.
3. That's a tricky one, because animations only really work with dvisvgm. Please ignore these for now.
4. I guess it makes sense to ignore those entirely. I will add another line to extract.lua to check for remember picture (overlay alone should work though).
5. As I have mentioned countless times before, pretty printing is very low priority at this point.
6. If it is required to make the example, the hidden key should probably be removed and a sentence about the code should be added to the text.

[hmenke]

I've updated the extractor script to also work with the graphdrawing examples and to ignore remember picture. Here is an updated build script:

#!/bin/bash
mkdir -p mwe
texlua extract.lua text-en/ ../../../tex/generic/pgf/graphdrawing/lua/pgf/ mwe/
cd mwe

touch /tmp/pgfmanuallock
echo 1 > /tmp/pgfmanualcount
ls -1q *.tex */**/*.tex | sort | xargs -I @ -P `nproc` bash -c '
    d=`dirname "@"`;
    f=`basename "@"`;
    { flock -x 200;
        n=$(cat /tmp/pgfmanualcount);
        echo $((n+1)) > /tmp/pgfmanualcount;
    } 200> /tmp/pgfmanuallock
    str="Building $n/$(ls -1q *.tex */**/*.tex | wc -l) $d/$f"
    cd $d;
    if [ -f "${f%%.tex}.pdf" ]; then
        str="$str Skipped."
    else
        lualatex -interaction=batchmode -halt-on-error "$f" > /dev/null
        if [ "x$?" == "x1" ]; then
            str="$str Failed!"
        fi
    fi
    echo $str'

[ghost]

Hi Henri,

Leider bekomme ich

mratz@MacBook-Pro-2:~/Documents/GitHub/pgf/doc/generic/pgf> ./build.sh

Usage: texlua extract.lua

./build.sh: line 8: nproc: command not found

xargs: max. processes must be >0

ls: /**/.tex: No such file or directory

Was mache ich falsch?

Ciao,

Michael

On Thu, Jul 4, 2019 at 12:43 AM Henri Menke [email protected] wrote:

I've updated the extractor script to also work with the graphdrawing examples and to ignore remember picture. Here is an updated build script:

#!/bin/bash mkdir -p mwe texlua extract.lua text-en/ ../../../tex/generic/pgf/graphdrawing/lua/pgf/ mwe/cd mwe

touch /tmp/pgfmanuallockecho 1 > /tmp/pgfmanualcount ls -1q *.tex /**/.tex | sort | xargs -I @ -P nproc bash -c ' d=dirname "@"; f=basename "@"; { flock -x 200; n=$(cat /tmp/pgfmanualcount); echo $((n+1)) > /tmp/pgfmanualcount; } 200> /tmp/pgfmanuallock str="Building $n/$(ls -1q *.tex /**/.tex | wc -l) $d/$f" cd $d; if [ -f "${f%%.tex}.pdf" ]; then str="$str Skipped." else lualatex -interaction=batchmode -halt-on-error "$f" > /dev/null if [ "x$?" == "x1" ]; then str="$str Failed!" fi fi echo $str'

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pgf-tikz/pgf/issues/640?email_source=notifications&email_token=AISROXABUTDMQCGCQTSZNKTP5UTQRA5CNFSM4HD7XP72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZF3PYQ#issuecomment-508278754, or mute the thread https://github.com/notifications/unsubscribe-auth/AISROXHUTYNZXC5POIFKDXTP5UTQRANCNFSM4HD7XP7Q .

[hmenke]

I see, macOS doesn't seem to have nproc. You can replace it with sysctl -n hw.ncpu.

[Mo-Gul]

@hmenke, and for your information. Under Win7 flock isn't present. It also doesn't seem to be available in MinGW. Any ideas for a counter solution working under Windows?

That's why I am currently "just" running the solution with parallelization but without the counter (https://github.com/pgf-tikz/pgf/issues/640#issuecomment-487755301).

A "dummy" solution could be to clear help files (AUX, LOG), count all TEX files minus the number of PDF files to get the total number of files that need to be processed. Then counting the number of LOG files gives number of processed files ...

[Mo-Gul]

1. No, that is not the only problem. BTW, setup code does the same thing in extract.lua as putting it into the pre key of every example. There are a lot more interdependencies though, e.g. this https://github.com/Mo-Gul/pgf/blob/f1b8890a941d5f89e36f7cd6ca98001faf4b998d/doc/generic/pgf/text-en/pgfmanual-en-dv-stylesheets.tex#L322-L329

Maybe I should have mentioned that I had modified extract.lua thus that setup code is written in the preamble. I also did some other adjustments to get my stated results (see https://github.com/Mo-Gul/pgf/blob/PimpCodeexamples/doc/generic/pgf/extract.lua).

[Mo-Gul]

For the record: I have written a PowerShell script myself to have parallel processing of the extracted codeexamples and having a counter. It works with PowerShell v5.1 available here.

# -----------------------------------------------------------------------------
# adapted from <https://hkeylocalmachine.com/?p=612>
# (2do: have a look at 
#     <https://devblogs.microsoft.com/scripting/weekend-scripter-a-look-at-the-poshrsjob-module/>
#  maybe the module can help to simplify this script)
# -----------------------------------------------------------------------------

$DestFolder = "mwe"

### create `$DestFolder` if it doesn't exist already
If (-Not (Test-Path -Path $DestFolder -PathType Container )) {
    New-Item -Path "$PSScriptRoot\" -Name $DestFolder -ItemType Directory
}

### extract `codeexample`s  from the manual files
texlua extract.lua text-en/ ../../../tex/generic/pgf/graphdrawing/lua/pgf/ $DestFolder/

### TeX all extracted files in parallel
$Folder = $PSScriptRoot + "\$DestFolder"
$TexFiles = Get-ChildItem $Folder -Recurse -Include *.tex `
    | Where {-Not (Test-Path -Path "$($_.Directory)\$($_.BaseName).pdf" -Pathtype Leaf)}
     
# Create Runspace Pool with `$MaxRunspaces` threads
[int]$MaxRunspaces = [int]$env:NUMBER_OF_PROCESSORS
$Pool = [RunspaceFactory]::CreateRunspacePool(1, $MaxRunspaces)
$Pool.ApartmentState = "MTA"
$Pool.Open()
$Runspaces = @()
     
# The script you want run against each host
$ScriptBlock = {
    # Take a TeX file as a parameter
    Param ($File);
    
    $DirectoryName = $File.DirectoryName
    $FileName = $File.Name

    lualatex -interaction=batchmode -halt-on-error --output-directory="$DirectoryName" "$FileName"
}

# Loop through the files 
ForEach ($TexFile In $TexFiles) {
    $Runspace = [powershell]::Create()
    
    # Add script block to runspace (use $Null to avoid noise)
    $Null = $Runspace.AddScript($ScriptBlock)
    
    # Add IP address as an argument to the scriptblock (use $Null to avoid noise)
    $Null = $Runspace.AddArgument($TexFile)
    
    # Add/create new runspace
    $Runspace.RunspacePool = $Pool
    $Runspaces += [pscustomobject]@{Pipe=$Runspace; Status=$Runspace.BeginInvoke() }
}

# -------------------------------------------------------------------------
# Prepare the progress bar
$CurrentCount = 0;
$TotalCount = ($Runspaces | Measure-Object).Count;
 
# Pause until all runspaces have completed
While ($Runspaces.Status -ne $Null)
{
    $Completed = $Runspaces | Where { $_.Status.IsCompleted -eq $True };
    
    # Update progress bar
    $CurrentCount = $CurrentCount + ($Completed | Measure-Object).Count;
    $StatusText = "Completed $CurrentCount/$TotalCount"
    Write-Progress `
        -Activity "TeXing files..." `
        -Status $StatusText `
        -PercentComplete (([int]$CurrentCount/[int]$TotalCount)*100);
    
    # Clear completed runspaces
    ForEach ($Runspace In $Completed)
    {
        $Runspace.Pipe.EndInvoke($Runspace.Status)
        $Runspace.Status = $Null            
    }
}
# -------------------------------------------------------------------------

# Clean-up Runspace Pool
$Pool.Close();
$Pool.Dispose();

# recursively delete empty folders
# (from <https://www.powershelladmin.com/wiki/PowerShell_script_to_recursively_delete_empty_folders>)
Get-ChildItem -LiteralPath $Folder -Force -Recurse `
    | Where-Object {
        $_.PSIsContainer -and `
        @(Get-ChildItem -LiteralPath $_.Fullname -Force -Recurse `
            | Where { -not $_.PSIsContainer }).Count -eq 0 `
    } `
    | Remove-Item -Recurse

[Mo-Gul]

@hmenke, it seems that there are even more (code)examples in the (Lua) documentation. Unfortunately they have another syntax. Here an example.

https://github.com/pgf-tikz/pgf/blob/a927d72a9fbfe40a38e92331513e6c0b35c8a94f/tex/generic/pgf/graphdrawing/lua/pgf/gd/circular/doc.lua#L94-L101

I guess you will easily be able to adapt extract.lua again. But currently I don't have a clue how/where to add options to these (code) examples. Could you give me a hint for that?

[Mo-Gul]

@hmenke, I have worked through the codeexamples and example(s) in the Lua part of the documentation as well now. Regarding the examples which need to have a preamble option I have marked with % TODOsp: codeexamples: and stated what is needed there.

I also searched myself for more examples and didn't find any more (that are not commented), so I would say I am ready for a pull request. Shall I do a pull request for the branch or first merge it to the master and then do a pull request?

[ghost]

Sorry for stealing your time and disappointing you. After the events at texstackexchange last month my motivation to help the TeX community just dropped below zero. As long as Stefan Kottwitz is around there is no way I will be part of this. In any case, I cannot handle github so I was never of any use.

Best, M.

On Mon, Jul 15, 2019 at 9:25 AM Mo-Gul [email protected] wrote:

@hmenke https://github.com/hmenke, I have worked through the codeexamples and example(s) in the Lua part of the documentation as well now. Regarding the examples which need to have a preamble option I have marked with % TODOsp: codeexamples: and stated what is needed there.

I also searched myself for more examples and didn't find any more (that are not commented), so I would say I am ready for a pull request. Shall I do a pull request for the branch or first merge it to the master and then do a pull request?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pgf-tikz/pgf/issues/640?email_source=notifications&email_token=AISROXBSZNFTFAEVEM25RWDP7SQGZA5CNFSM4HD7XP72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ6HCMQ#issuecomment-511471922, or mute the thread https://github.com/notifications/unsubscribe-auth/AISROXHX5IFRN6RLTVEOQ2DP7SQGZANCNFSM4HD7XP7Q .

[hmenke]

@Mo-Gul I've put some stuff in place so that you can add options to the Lua examples.

A table of examples can be converted as such:

-  examples = {[["
-    \tikz \graph [spring electrical layout, horizontal=0 to 1]
-      { 0 [electric charge=1] -- subgraph C_n [n=10] };
-  "]],[["
-    \tikz \graph [spring electrical layout, horizontal=0 to 1]
-      { 0 [electric charge=5] -- subgraph C_n [n=10] };
-  "]],[["
-    \tikz \graph [spring electrical layout, horizontal=0 to 1]
-      { [clique] 1 [electric charge=5], 2, 3, 4 };
-  "]]
- }
+  examples = {
+    {
+      options = [["preamble={\usetikzlibrary{graphs,graphdrawing} \usegdlibrary{force}}"]],
+      code = [["
+        \tikz \graph [spring electrical layout, horizontal=0 to 1]
+          { 0 [electric charge=1] -- subgraph C_n [n=10] };
+      "]]
+    },{
+      code = [["
+        \tikz \graph [spring electrical layout, horizontal=0 to 1]
+          { 0 [electric charge=5] -- subgraph C_n [n=10] };
+      "]]
+    },{
+      code = [["
+        \tikz \graph [spring electrical layout, horizontal=0 to 1]
+          { [clique] 1 [electric charge=5], 2, 3, 4 };
+      "]]
+    }
+  }

Sometimes though there are single examples, which are given as a string. These have to be wrapped in two tables (note the double braces in the example):

-  examples = [["
-    \tikz \graph [spring electrical layout, horizontal=0 to 1]
-      { 0 [electric charge=1] -- subgraph C_n [n=10] };
-  "]]
+  examples = {{
+      options = [["preamble={\usetikzlibrary{graphs,graphdrawing} \usegdlibrary{force}}"]],
+      code = [["
+        \tikz \graph [spring electrical layout, horizontal=0 to 1]
+          { 0 [electric charge=1] -- subgraph C_n [n=10] };
+      "]]
+    }}

Please also disregard the code in tex/generic/pgf/graphdrawing/lua/pgf/gd/force/jedi. This doesn't seem to be interfaced and doesn't even compile.