Bibliography citations

[Omikhleia]

Bibliography citations are both supported by lunamark (in the native approach route) and Pandoc (with the Pandoc AST route) -- The latter possibly taking advantages of some "pandoc-filter".

~~SILE has a basic bibTeX bibliography support, but I haven't investigated yet if it could be used here.~~ See comment below...

[Omikhleia]

I have too little experience

• With bibTeX bibliography
• With (Pandoc-style) Markdown for citations

I have not idea how people may use this kind of things, and little time for investigation => Adding "Help wanted" label. Gentle readers, feel free to provide inputs regarding expectations on this topic, if you want to see it addressed.

[bdarcus]

Take a look at citeproc-lua, though I'm not sure how general it is (beyond LuaTeX) ATM. That could provide the citation and bibliography output processing analogous to citeproc in pandoc.

Also, I have a feeling you already saw the citation syntax will surely to be added to djot itself (not surprising since John himself is an academic), and will be an iteration on the pandoc one?

https://github.com/jgm/djot/issues/32

[Omikhleia]

@bdarcus Thanks for the head's up!

AFAIC, I'm too busy at the moment to consider bibliography citations properly -- It's not an excuse, but working on this package collection, besides working on my "resilient" collection of classes and packages for SILE, and also with a book in progress, all of this doesn't leave much "free" brain to assess the question in due form (esp. as I wrote above, I have little experience working with such bibliographies in my typesetting projects).

Anyhow, it's interesting food for thought; e.g. at a very quick glance, citeproc-lua seems to depend on other packages for XML parsing, i18n collations, etc. -- It could be interesting to check how deeply those are entangled in the engine or can be "hooked" to use the existing ways in SILE (which has some XML support natively, and can use ICU4C already, for instance) without reinventing the wheel. I'll certainly want to look at this at some point -- but other minds can also give it a try!

[bdarcus]

No worries; was just responding to your call for input, as I stumbled across SILE again.

It's a tricky problem, but will be made easier by a) libraries like these, and b) previous real world experience, most notably with pandoc, whose author is a philosopher.

Since you're also already supporting djot, you can also lean on that for the design there, which would avoid a lot of work here.

[Omikhleia]

I took some time to dabble into that topic.

There are two different things at stakes:

1. Support by the various lightweight-markup engines we use
2. Actual rendering

Regarding (1):

• Pandoc of course supports it, and our pandocast implementation could be improved to process the corresponding AST node (Cite)

  pandonc -t json -f markdown
  See @joe2011
  {"pandoc-api-version":[1,23,1],"meta":{},"blocks":[{"t":"Para","c":[{"t":"Str","c":"See"},{"t":"Space"},{"t":"Cite","c":[[{"citationId":"joe2011","citationPrefix":[],"citationSuffix":[],"citationMode":{"t":"AuthorInText"},"citationNoteNum":1,"citationHash":0}],[{"t":"Str","c":"@joe2011"}]]}]}]}
  

• Lunamark has support for something similar -- I haven't looked at the details, but it's in the code base, behind the citations option, so our markdown implementation could be extended in that direction: https://github.com/jgm/lunamark/blob/75f61d0a0fba61a09a12700bd826007f5af41dfb/lunamark/reader/markdown.lua#L1867-L1869
• Djot does not have it yet, though, as of now: https://github.com/jgm/djot/issues/32 -- So our djot support will still be lacking here.

With the lack of Djot support, looking for alternatives, I found this interesting idea (though I can recollect where): Even without syntax explicit support, we could "minimally" re-use the link syntax: [](@key) -- It would "sort of" be acceptable, as it's unlikely real "local" links start with an @. And while a "workaround", it would have a few advantages:

• No need to modify our readers (djot, markdown, pandoc AST), all the burden is moved to the renderer.
• Easy to fix with filters at a later stage... or merely a matter of replacing the (pseudo-)link by any appropriate syntax.
• (Not on topic here, but re-using links in "specific" ways is also a potential workaround for indexing entries (#https://github.com/Omikhleia/resilient.sile/issues/17), since neither Markdown nor Djot define a standard way to mark indexed sequences...)

Regarding (2), then, whether we eventually support some adequate syntax or stick to the above-mentioned work, the issue is then to resolve the key (from some bibliography file, e.g. in bibtex format) and format the appropriate text (in some "style", such as the Chicago-style reference style).

Here a few notes:

1. There's the CSL-enabled ¢iteproc-lua mentioned in previous exchanges.
2. There's the SILE way, with its bibtex package, although the latter is very rough and has lots of issues (e.g. https://github.com/sile-typesetter/sile/issues/2021, https://github.com/sile-typesetter/sile/issues/2022, https://github.com/sile-typesetter/sile/issues/2023, https://github.com/sile-typesetter/sile/issues/2024)
3. If we had to re-implement something in SILE, there could be other ways to consider, such as @zepinglee 's https://github.com/zepinglee/lua-bibtex-parser

Whatever route is taken, this would be beyond this package's scope -- Ideally, it boils down to leveraging the SILE way (2) with a more robust implementation...

Preliminary conclusion

• While waiting for a Djot (i.e. djot-lua[^1]) official solution, we could use a temporary workaround, at least for basic uses cases, relying on hacking the link syntax rendering (common to all our "inputters": Djot, Markdown, Pandoc AST). That would actually be a neat way to have something started with minimal effort, on the "parsing" side of things.
• For the "rendering" side of things, we should use SILE's bibtex package, however limited currently, and try to push it forward.

Does the above makes sense to potential readers here, or am I forgetting anything?

[^1]: Of course, we could also hack djot-lua... as I have already done for some extensions (conditionals, generalized captions)... but at some point this is not a good route to go... Even if such changes could go upstream eventually... which I wouldn't trust... I'm still waiting for an official 0.6 release of lunamark more than 1.5 years after I started contributing, so I am losing all faith there... The Lua ecosystem is weak, and one can only do what one can.

[Omikhleia]

Update on point (2) above: eventually, I started working on a CSL engine for SILE https://github.com/sile-typesetter/sile/pull/2082

The CSL 1.0.2 specification is.... oh well, say "unclear at places" not to offend anyone, but it's funny to code.