stabilize the renderer interface

[gdamore]

I've been looking a lot and the renderer backend for HTML.

Enough so that I'm going to be writing my own backend to produce ebooks -- basically forking the existing HTML backend for now. However, one of the challenges here is that the interface between the renderer and core (the "types" that are tokenized) isn't quite stable yet.

It would be good to get this to a point where it could be a little closer to stable so that we could begin to rely upon it.

I'm also thinking of trying to start a PDF renderer as well. It would be too hard to undertake that while the interface is unstable, I think.

And yes, the fact that I'm thinking about these, and want these, is a strong testament to the good work already done here. IMO, the go version of libasciidoc is superior to asciidoctor itself, although there are short-comings. I think this go based toolchain shows promise of actually surpassing asciidoctor. Stabilizing the APIs (or at least getting to the point when we can use semver) will be a major step towards that end.

[xcoulon]

hello @gdamore 👋

This is awesome news if you want to work on an epub3 renderer! And I appreciate very much your feedback about the library, although, I think there are still a lot of missing features compared to Asciidoctor.

That being said, regarding the stabilization of the types: are you referring to the current #573 and #574 issues? I have a PR ready for #573 but I'm not 100% sure it's worth it yet, and I may have time to work on #574 soon. Did you have other issues in mind?

[gdamore]

Actually I wasn't thinking of either of those specifically. I just noticed that as I'm working on a different backend renderer, if I pinned to v0.4.0 there were changes relative to tip that were breaking. In those cases I think it was fixes for the passthrough blocks and callouts. (The types got busted.)

[gdamore]

(By stable, I mean "interface is not changing"... interface-stability in other words. Semantic Versioning is what I am thinking about.)

[xcoulon]

ok, I see what you mean, and I understand your frustration. I'll see what I can do to avoid breaking the interfaces, or at least I'll try to let you know. I also plan to work on #558 and I know this one is going to be a lot of work, but hopefully it's not going to break the interfaces.

[gdamore]

I actually started to take a crack at #558 at as well. I set it aside, because its a lot of work indeed.

I would actually first recommend moving most of the globals in the renderers package into a single configuration or template object. Then those could be adjusted separately.

My thought was to have a common "templates" package, that had most of the code in the html5 package, but used some kind of interface for render, and also provided a concrete Templates object that would contain a bunch of strings (basically all the text template strings).

I'd also probably have all the templates get the same global funcMap, so that templates could make use of escaping, etc. as they need to.

Then the html5 (and probably a future xhtml5 package I was going to do) would just supply their own concrete templates, and a wrapper Render() method.

What's also cool is you could make a renderer that loaded all those templates from files in a directory -- allowing the entire thing to be driven by external resources, if some user decides.

I could probably knock this out in a day or two, but I wasn't really sure how open you were to the idea, and I don't want to step on toes.

[xcoulon]

I actually started to take a crack at #558 at as well. I set it aside, because its a lot of work indeed.

I would actually first recommend moving most of the globals in the renderers package into a single configuration or template object. Then those could be adjusted separately.

My thought was to have a common "templates" package, that had most of the code in the html5 package, but used some kind of interface for render, and also provided a concrete Templates object that would contain a bunch of strings (basically all the text template strings).

I'd also probably have all the templates get the same global funcMap, so that templates could make use of escaping, etc. as they need to.

Then the html5 (and probably a future xhtml5 package I was going to do) would just supply their own concrete templates, and a wrapper Render() method.

What's also cool is you could make a renderer that loaded all those templates from files in a directory -- allowing the entire thing to be driven by external resources, if some user decides.

I could probably knock this out in a day or two, but I wasn't really sure how open you were to the idea, and I don't want to step on toes.

Well, I initially thought that having the template vars in the same .go file as the renderXYZ() function that uses them would make the maintenance easier, because both would always together.

I agree for the same global funcmap for the escaping, etc. That could even be set in the newTextTemplate func, actually.

But I think that I see what you mean: decoupling the renderXYZ funcs from the templates, so we can use HTML5 or XHTML without having to reimplement (or just copy) the renderXYZ funcs?

In that case, maybe we could move all the renderXYZ funcs in the pkg/renderer package, and leave the templates and the tests in the pkg/renderer/html5 package? For now, I would be tempted to keep the templates in their own files, so we would have a direct mapping such as pkg/renderer/cross_reference.go -> pkg/renderer/html5/cross_reference.go + pkg/renderer/html5/cross_reference_test.go

[gdamore]

That seems pretty good to me. Splitting things this way will facilitate subclassing for other kinds of HTML-ish output (EPUB is mostly HTML, plus some other stuff.) But this would let us get to XHTML, and also semantic HTML (the html5s backend in asciidoctor). It might even be able to generate docbook since that's just SGML -- or other XML based online help systems, etc.

The man page generator might even be able to use templates this way to generate ROFF output (or the FreeBSD folks could do mandoc format output).

I doubt very much that we'd be able to use this templating stuff to get to PDF, unfortunately, but still even without it I think this can go quite a long ways.

[xcoulon]

That seems pretty good to me. Splitting things this way will facilitate subclassing for other kinds of HTML-ish output (EPUB is mostly HTML, plus some other stuff.) But this would let us get to XHTML, and also semantic HTML (the html5s backend in asciidoctor). It might even be able to generate docbook since that's just SGML -- or other XML based online help systems, etc.

indeed, all SGML-based renderers could use this solution. Each renderer would need to provide the templates for each kind of element (and in some cases, there may be empty templates)

The man page generator might even be able to use templates this way to generate ROFF output (or the FreeBSD folks could do mandoc format output).

I'm not familiar with ROFF, I would have to check it in details.

I doubt very much that we'd be able to use this templating stuff to get to PDF, unfortunately, but still even without it I think this can go quite a long ways.

Yes, PDF may be an exception :(

[gdamore]

ROFF (also known as troff or nroff - or for linux - groff -- output is a non-SGML/non-XML markup that is used to format online UNIX manual pages (ie. the things you get if you type "man ".)

I wrote a number of documents in this format in the past; some of my college papers were written in roff format with the ms macro package. (I'm old.)

[xcoulon]

ROFF (also known as troff or nroff - or for linux - groff -- output is a non-SGML/non-XML markup that is used to format online UNIX manual pages (ie. the things you get if you type "man ".)

Yes, I'm briefly seen that ROFF is used in man pages, but I'm not familiar with its syntax 😬

I wrote a number of documents in this format in the past; some of my college papers were written in roff format with the ms macro package. (I'm old.)

😄

[gdamore]

I'm about to start again on this.

I'd like to move the render functions to a sub package of the renderer (probably render/template) -- the reason for this is that I think the high-level render API may be something we want to support for other non-XML/SGML style outputs (think PDF).

So render/template provides the common framework for all the XML/SGML like backends, but the high-level render package can remain even more agnostic. The html5 package would contain the HTML templates, the tests for same, and would consume both the renderer and render/template package.

Does this sound reasonable?

[xcoulon]

I'm about to start again on this.

I'd like to move the render functions to a sub package of the renderer (probably render/template) -- the reason for this is that I think the high-level render API may be something we want to support for other non-XML/SGML style outputs (think PDF).

So render/template provides the common framework for all the XML/SGML like backends, but the high-level render package can remain even more agnostic. The html5 package would contain the HTML templates, the tests for same, and would consume both the renderer and render/template package.

Does this sound reasonable?

Yes, that looks ok to me but I believe that the pkg/renderer/template name is a bit confusing, because it's going to contain all the renderXYZ funcs, but not the go templates themselves, isn't it?

Also, your PR would be "only" refactoring, no new feature yet, wouldn't it?

[gdamore]

I started with render/templates -- I'm open to renaming this. In retrospect maybe "sgml" would work -- as all these formats (except for ROFF and mandoc, and I'm not sure we should worry ourselves about that) are just flavors of SGML.

So what I'm doing is creating a base renderer object in that package, which has something like this:

type renderer {
   Render(ctx, ...)
   // Used to set the template
   SetTemplate(name string, template string)
   // Used to set an entry in the FuncMap
   SetFunction(name string, function interface{})
}

Details may evolve as I work, but this is the general idea.

[gdamore]

In the html5 package there will be a sort of "default" object, which is intialized by init(), which sets those values.

[xcoulon]

started with render/templates -- I'm open to renaming this. In retrospect maybe "sgml" would work -- as all these formats (except for ROFF and mandoc, and I'm not sure we should worry ourselves about that) are just flavors of SGML.

yes, I was actually wondering if this pkg structure would make sense:

pkg
|- renderer
  |- sgml
    |- html5

with:

• pkg/renderer remaining unchanged for now
• pkg/renderer/sgml containing the .go files with the renderXYZ funcs
• pkg/renderer/sgml/html5 containing the go templates and the tests

in the future, we could have other SGML flavors (pkg/renderer/sgml/html5s, pkg/renderer/sgml/epub, etc.)

And when PDF support implementation lands, it would be in pkg/renderer/pdf What do you think about this?

[gdamore]

I like it. That's what I'll do.

[gdamore]

(Although technically... epub isn't sgml. Rather it has content documents that are sgml. So there might actually be two epub packages... we'll see.)

[xcoulon]

we could replace sgml with structured if we want to be more generic, but one may also argue that PDF and others are also structured ;)

[gdamore]

yeah, i'll stick with sgml for now.

[xcoulon]

we could replace sgml with structured if we want to be more generic, but one may also argue that PDF and others are also structured ;)

naming is hard 😅

[xcoulon]

yeah, i'll stick with sgml for now.

ok, sounds good to me 👍

[gdamore]

I don't think we'll fully stabilize this backend yet. I would like to hold this task given the optimizations and cleanups that I anticipate, and I think it makes sense to wait until we have a docbook backend too, since that might force changes.

Thus I'd recommend kicking this to backlog. I'm doing that for now.

[xcoulon]

ok, sounds good to me