Indents not being applied in noweave output

[RobertBaruch]

Test file t.nw (same as examples/test.nw):

% Copyright 1991 by Norman Ramsey.  All rights reserved.
% See file COPYRIGHT for more information.
<<*>>=
one <<two>> <<three>>	# uses two and three
@ %def one
<<two>>=
first of two
second of two
third of two
@ %def fish fowl duck
@ %def two
<<three>>=
first of three
 second of three
  third of three
@ %def three

$ noweave t.nw > t.tex
$ pdflatex t.tex
This is pdfTeX, Version 3.141592653-2.6-1.40.24 (TeX Live 2022) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode
(./t.tex
LaTeX2e <2022-06-01> patch level 2
L3 programming layer <2022-06-16>
(/usr/local/texlive/2022/texmf-dist/tex/latex/base/article.cls
Document Class: article 2021/10/04 v1.4n Standard LaTeX document class
(/usr/local/texlive/2022/texmf-dist/tex/latex/base/size10.clo))
(/usr/local/tex/inputs/noweb.sty)
(/usr/local/texlive/2022/texmf-dist/tex/latex/l3backend/l3backend-pdftex.def)
(./t.aux) [1{/usr/local/texlive/2022/texmf-var/fonts/map/pdftex/updmap/pdftex.m
ap}] (./t.aux) )</usr/local/texlive/2022/texmf-dist/fonts/type1/public/amsfonts
/cm/cmr10.pfb></usr/local/texlive/2022/texmf-dist/fonts/type1/public/amsfonts/c
m/cmsy10.pfb></usr/local/texlive/2022/texmf-dist/fonts/type1/public/amsfonts/cm
/cmti10.pfb></usr/local/texlive/2022/texmf-dist/fonts/type1/public/amsfonts/cm/
cmtt10.pfb>
Output written on t.pdf (1 page, 42739 bytes).
Transcript written on t.log.

Result is in t.pdf, which shows none of the expected indenting in codeblock <<three>>.

The problem doesn't seem to be with pdflatex since a \begin{verbatim} block retains indents.

[nrnrnr]

I can't duplicate the problem. (The indentation is fine for me.)

What's the checksum on your t.tex? I get

$ md5sum t.tex
51eda6cbefcb75ab1767aa5743850a11  t.tex

If yours is different, let's see it.

Details about your system, please?

I've got

This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020/Debian) (preloaded format=pdflatex)

[RobertBaruch]

I did get a different checksum. Here's t.tex:

\documentclass{article}\usepackage{noweb}\pagestyle{noweb}\noweboptions{}\begin{document}\nwfilename{t.nw}\nwbegindocs{0}\nwdocspar
% Copyright 1991 by Norman Ramsey.  All rights reserved.
% See file COPYRIGHT for more information.
\nwenddocs{}\nwbegincode{1}\moddef{*}\endmoddef
one \LA{}two\RA{} \LA{}three\RA{}   # uses two and three
\eatline
\nwendcode{}\nwbegincode{2}\moddef{two}\endmoddef
first of two
second of two
third of two
\eatline
\eatline
\nwendcode{}\nwbegincode{3}\moddef{three}\endmoddef
first of three
 second of three
  third of three
\eatline
\nwendcode{}\end{document}

And here's my tex version:

$ pdflatex --version
pdfTeX 3.141592653-2.6-1.40.24 (TeX Live 2022)
kpathsea version 6.3.4
Copyright 2022 Han The Thanh (pdfTeX) et al.
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Han The Thanh (pdfTeX) et al.
Compiled with libpng 1.6.37; using libpng 1.6.37
Compiled with zlib 1.2.11; using zlib 1.2.11
Compiled with xpdf version 4.03

[nrnrnr]

Your .tex file works correctly for me in pdflatex.

I'm not sure where your Noweb installation comes from, so I'm not sure what else we can do here.

[RobertBaruch]

Hmm, I cloned noweb from git. I can delete it all and try again...

[RobertBaruch]

I tried it in two different ways:

• apt install noweb
• git clone https://github.com/nrnrnr/noweb.git + compile & install

In both cases, indents were not applied.

This was all using TeX Live 2022.

I installed TeX Live 2019, and got the indents. So there is something that happened in TeX Live 2020, 2021, or 2022 that is causing whatever noweb uses to indent to not work anymore.

I don't really want to use 2019 because usually the latest version of TeX Live is recommended, and tlmgr to install packages tries to install the latest, and complains about 2019:

$ tlmgr repository list
(running on Debian, switching to user mode!)
List of repositories (with tags if set):
	http://mirror.ctan.org/systems/texlive/tlnet (main)

$ tlmgr --verify-repo=none install tikz
tlmgr: Local TeX Live (2019) is older than remote repository (2022). 
Cross release updates are only supported with
  update-tlmgr-latest(.sh/.exe) --update
See https://tug.org/texlive/upgrade.html for details.

And then that page has a big notice on it about how upgrading is dangerous and you should just install the latest version.

[nrnrnr]

OK, I can well believe it's a TeXlive issue. I've been bitten before.

I'm running Debian stable. That's TeXlive 2020. It looks like TeXlive 2022 is available on Debian unstable. But I can't anticipate setting up an unstable system any time soon---my plate is full right now, and it could be as long as a year.

Maybe there's an easier way of setting up a test environment? Or maybe you can provide me with access to a system on which to test? In that case I can likely track this issue down sometime between now and mid-August. (You could also try filing a bug report against Debian unstable, but given that it's manifestly an upstream issue, they might not be interested.)

[RobertBaruch]

I'm ok with having to use a stable version -- I can just apt install texlive-full which seems to load nearly all the packages in CTAN, versioned according to 2019 (or I guess 2020). I'm on Ubuntu 20.04.3 LTS, which pins 2019.

Probably best to leave this bug open and revisit when the major stable releases pin TeX Live to 2021 or 2022.

As for a test environment, you should be able to install 2022 in parallel with 2020, but you'd probably have to change some env variables like TEXINPUTS, PATH, and point the tlmgr repo to the right one. At least, that's all I had to do.

Then again, it's hard to know whether the installation is hermetic. Docker is probably the best option here.

[dbosk]

Works correctly on TeXLive 2021 (Ubuntu 22.04 LTS).

I can reproduce that it works incorrectly on TeXLive 2022:

$ docker run -it -v /tmp:/outside texlive/texlive:latest
root@e0fff1176e4e:/# apt update
[...]
root@e0fff1176e4e:/# apt install noweb
[...]
root@e0fff1176e4e:/# cp /outside/test.tex .
root@e0fff1176e4e:/# TEXMFHOME=/usr/share/texmf pdflatex test.tex
[...]

Yields:

• test.log
• test.pdf

[arcfide]

I've been bitten by this on MikTeX, too. I'm not sure what's going on. The verbatim environment works but not the noweb indentation, with or without tabs.

[arcfide]

Not just on Tex Live, but MikTeX also has this issue with its latest updates.

[arcfide]

I've made some progress on this. In support.nw, chunk 8b, the definition of \@begincode includes the following lines:

<<zap ligatures, fix spaces 10c>>
\nowebsize \setupcode

This appears to be where the problem occurs. It looks like there might be some clobbering of catcodes or the like. Chunk 10c includes the code that we want, namely, \@vobeyspaces. However, this doesn't seem to be in force by the time the actual code is rendered. I can add a line like \let =\ to the beginning of a code chunk and things work after that.

Strangely, I've found that if you swap the above two lines, then things work, that is, if you fix spaces after \nowebsize \setupcode are called. I also am not sure if we really need to be calling \setupcode two times, once in chunk 8b, and a second time in chunk 10c. At any rate, I have a minor workaround for the moment, but I'm not sure what the real fix should be.

[nrnrnr]

It's great to hear of some progress here, but I'm about to go on vacation for three weeks, then business travel. I'll try to remember to get back to this around the beginning of October.

[Mechtilde]

Hello i also stumpled about this problem with the Update to TexLive 2022 in Debian.

If some provide me a patched file I will be able to test it .

[meta-ed]

This problem, or one indistinguishable from the description above, is now seen with Noweb 2.12 on the Slackware 15.0 platform. There was a recent security patch to TeX Live by the Slackware 15.0 maintainers. The patch brought TeX Live forward from texlive-2021.210418 to texlive-2023.230322. With texlive-2023.230322, indentation is broken. Reverting to texlive-2021.210418 restores the correct indentation behavior.

Is the request still open for access to a system on which to test?

[Mechtilde]

Hello,

you can use the patch also to path your version of noweb under slackware as I did it for Debian

It will be released as stable next weekend.

I already use it regularly.

Kind regards

Am 06.06.23 um 14:43 schrieb meta-ed:

This problem, or one indistinguishable from the description above, is now seen with Noweb 2.12 on the Slackware 15.0 platform. There was a recent security patch to TeX Live by the Slackware 15.0 maintainers. The patch brought TeX Live forward from texlive-2021.210418 to texlive-2023.230322. With texlive-2023.230322, indentation is broken. Reverting to texlive-2021.210418 restores the correct indentation behavior.

Is the request still open for access to a system on which to test?

-- Mechtilde Stehmann

Debian Developer

PGP encryption welcome

F0E3 7F3D C87A 4998 2899 39E7 F287 7BBA 141A AD7F

[nrnrnr]

Fantastic. I will put it on my list for after the latest stable release this weekend.

[nrnrnr]

Fixed. If anybody wants to try it out, I'll take the "RC" off the version tag.

[meta-ed]

There is an unexpected dependency on the .git directory. When building from the tarball, probably there shouldn't be such a dependency.

make[1]: *** No rule to make target '../../.git/refs/heads/master', needed by 'gitversion.c'. Stop.

I do not find gitversion.c in 2.12, only 2.13rc, so I assume that's a dependency that crept in between the two point releases.

[nrnrnr]

Thanks for catching this bug. I've pushed v2_13rc3, which seems to resolve it.

[meta-ed]

I see rc2 but not rc3. I'll look again tonight. EDIT: I should add in case that's just a typo and it's rc2, I tried building rc2 and ran into the same issue.

[nrnrnr]

My bad. I pushed the commit but not the tag.

[meta-ed]

Making progress! The build now fails at: find: ‘/tmp/SBo/package-noweb-2.13rc3/usr/man’: No such file or directory Investigating, it appears the build writes the manpages to /usr/man instead of $DESTDIR/usr/man.

EDIT: I'm using the Slackware 15.0 build script that was developed for 2.12 to produce the binary package for Slackware 15. As I look deeper into it, apparently $DESTDIR is patched in by the build script maintainer exactly for this reason. So I need to go back and update that patch to work with 2.13.

[nrnrnr]

I accepted a PR that made several changes to the Makefiles, but I don't remember this quite being an issue. Can you please let me know which file it tried to deposit in the bad place?

[meta-ed]

A script by Kyle Guinn builds 2.12 for Slackware 15.0. That script applies a patch to 2.12 makefiles before running make install. The patch lets installed files be dropped in a temporary tree. The tree is then tarballed and becomes the binary distribution.

Going by Kyle's patch, I patched 2.13rc3, and successfully created a binary distribution of 2.13rc3.

Then I tested these four cases:

Texlive 2021, Noweb 2.12 -- works Texlive 2023, Noweb 2.12 -- indents don't work Texlive 2021, Noweb 2.13rc3 -- works Texlive 2023, Noweb 2.13rc3 -- works

Example of what the patch does:

-       mkdir -p $(BIN) $(LIB)
-       sed "s@|LIBDIR|@$(LIBNAME)@;s@|GITVERSION|@$(GITVERSION)@" shell/noweb > $(BIN)/noweb
-       chmod +x $(BIN)/noweb

becomes

+       mkdir -p $(DESTDIR)$(BIN) $(DESTDIR)$(LIB)
+       sed "s@|LIBDIR|@$(LIB)@;s@|GITVERSION|@$(GITVERSION)@" shell/noweb > $(DESTDIR)$(BIN)/noweb
+       chmod +x $(DESTDIR)$(BIN)/noweb

[nrnrnr]

Frankly, if I were writing such a patch, I'd be patching lines 22 to 28 and leaving the rest alone. That's the procedure suggested in step 1 of the INSTALL file.

An even better plan would be to create a shell script that invokes Make while setting those variables on the command line.

Obviously, Slackware can do as it likes.

[meta-ed]

I doubt I'll explain this well, but I'll try. And maybe you understand this already, and I'm not understanding the Noweb makefiles. If so I apologize for wasting your time.

The problem is not specific to Slackware. Any time an application is to be packaged as a binary tarball, the build procedure can't install the pieces to the running system library. The pieces need to go into an empty tree that can be tarballed and later extracted to target systems.

That's easily done by editing lines 22 to 28. I can set BIN=/tmp/build/usr/bin, LIB=/tmp/build/usr/libexec/noweb, MAN=/tmp/build/usr/man, etc. and that creates the empty tree perfectly.

Now the pathnames in the resulting tarball will not include the /tmp/build prefix. It will contain usr/bin/noweb usr/libexec/noweb/markup usr/man/noweb.1.gz etc. This lets the tarball extract into the target system's root directory structure.

But the change in pathname breaks Noweb at run-time. For example, /usr/libexec/noweb/markup exists, but Noweb scripts are still looking for it in /tmp/build/usr/libexec/noweb/markup.

[nrnrnr]

Ah, I think I see the problem. The expectation is as follows:

1. Noweb is installed into a temporary directory.

2. The directory is used to build a tarball containing relative pathnames.

3. The tarball is later to be unpacked into a final destination directory.

4. The files installed in the temporary directory should contain pathnames relative to the final destination directory.

Such functionality could be provided by altering src/Makefile so that the install targets place files into the temporary directory. The Debian build scripts have a patch like this as well. I'd consider a PR that would make this sort of change simpler, but I think I don't want to do it myself. The documentation could be gnarly, and it would be of interest only to package maintainers. Who, by and large, already know what they are doing.

@Mechtilde take note.

[meta-ed]

@Mechtilde can you, Kyle Guinn, and I maybe collaborate on a single pull request that would support package creation?

[turboencabulator]

A script by Kyle Guinn builds 2.12 for Slackware 15.0. That script applies a patch to 2.12 makefiles before running make install. The patch lets installed files be dropped in a temporary tree. The tree is then tarballed and becomes the binary distribution.

FYI the makefile patch mentioned here is largely PR #10, plus or minus some things, but the basic purpose was to make it easier to create a distro package. I think we deemed it too extensive of a change, so I've been trying to get it pulled in in smaller pieces, which has kind of stalled lately. I can restart that effort.

[Mechtilde]

Hello,

yes I'm willing to help as far as I can.

I opened also a bugreport https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1040171 and asking the former maintainer for help.

I had problems to adapt the Makefiles for Debian packages

Regards

Am 30.06.23 um 22:12 schrieb meta-ed:

@Mechtilde can you, Kyle Guinn, and I maybe collaborate on a single pull request that would support package creation?

-- Mechtilde Stehmann

Debian Developer

PGP encryption welcome

F0E3 7F3D C87A 4998 2899 39E7 F287 7BBA 141A AD7F