Canonical large-ish djot document (and related pandoc issues)

[clbarnes]

For benchmarking and testing, it would be helpful to have a large document covering a good spread of djot's features. I tried to convert the pandoc manual into djot for this purpose, using both djot.js and djot.lua:

First, getting the manual as pandoc AST:

wget https://pandoc.org/demo/MANUAL.txt -O - | pandoc -f markdown -t json > manual.json

djot.js, as documented here https://github.com/jgm/djot.js/tree/main#installing-the-command-line-utility

$ djot -f pandoc -t djot manual.json > manual.js.dj

Error: TypeError: Cannot read properties of undefined (reading 'length')
TypeError: Cannot read properties of undefined (reading 'length')
    at fromPandocAttr (/home/barnesc/code/djot-bench/node_modules/@djot/djot/lib/pandoc.js:399:18)
    at PandocParser.fromPandocBlock (/home/barnesc/code/djot-bench/node_modules/@djot/djot/lib/pandoc.js:746:30)
    at /home/barnesc/code/djot-bench/node_modules/@djot/djot/lib/pandoc.js:855:25
    at Array.map (<anonymous>)
    at PandocParser.fromPandoc (/home/barnesc/code/djot-bench/node_modules/@djot/djot/lib/pandoc.js:854:34)
    at fromPandoc (/home/barnesc/code/djot-bench/node_modules/@djot/djot/lib/pandoc.js:868:38)
    at Object.<anonymous> (/home/barnesc/code/djot-bench/node_modules/@djot/djot/lib/cli.js:175:43)
    at Module._compile (node:internal/modules/cjs/loader:1275:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1329:10)
    at Module.load (node:internal/modules/cjs/loader:1133:32)%

djot.lua as documented here https://github.com/jgm/djot.lua#readme

$ pandoc -f json -t djot-writer.lua > manual.lua.dj

pandoc: PandocLuaException "djot-writer.lua:8: attempt to index a nil value (local 'layout')\nstack traceback:\n\tdjot-writer.lua:8: in main chunk"

I'm not sure what the point of failure is here:

• pandoc's markdown-formatted manual may not make clean pandoc ast
• djot.lua may not be able to convert pandoc ast into djot
• djot.js may not be able to convert pandoc ast into djot
• PEBKAC (always a threat)

[jgm]

This is working fine for me:

curl  https://pandoc.org/demo/MANUAL.txt | pandoc -f markdown -t json | ~/.npm-packages/bin/djot -f pandoc -t djot

Have you installed the latest djot.js using npm install -g @djot/djot? Are you sure you're running that version and not another one?

[clbarnes]

I have just updated djot.js with the same results, although as a local install rather than global (npm i @djot/djot). I have tried accessing the binary using npx djot and node_modules/.bin/djot.

[jgm]

Since I can't reproduce it, you may have to track down the issue yourself. I'd suggest putting some tracing code in node_modules/@djot/djot/lib/pandoc.js:399:18 If we could see the value of pattr there it might give us a clue. More ambitiously you could add something that prints out block.t in fromPandocBlock, so we can keep track of which block is being processed and localize the issue.

[clbarnes]

Regardless of my possibly platform-specific issue, would it be possible to publish that generated djot file (or another one of significant size and feature-breadth) for test/benchmark purposes?

[bdarcus]

... would it be possible to publish that generated djot file (or another one of significant size and feature-breadth) for test/benchmark purposes?

@clbarnes - I'm attaching it here.

pandoc-manual.txt