Allow for fallback template inheritance

[plbt5]

Current operation

On specifying a template, currently it is possible to extend an existing template (extend: [<template_name>+]). This works very well when both the inherited and the inheriting templates are specified in the same configuration file; when the (to be) inherited template has not been specified in this file, an error is thrown:

/Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.5/lib/pandocomatic/configuration.rb:748:in block in extend_template: undefined local variable or method parent_template_name for #Pandocomatic::Configuration:0x00007fd7f89f15f0 (NameError)

Feature request

Include a default fallback operation that allows to inherit templates (that are not specified in the same yaml-file as the inheriting template) from the pandocomatic.yaml file that is available in pandoc's default directory: ~/.local/share/pandoc/. This allows for a more modular setup of pandocomatic configuration files, particular the availability of a baseline pandocomatic configuration that applies always (unless overwritten by a user-defined extending template, specified in a separate configuration file).

Suggested Operation

Assume the following excerpt of a user-defined local myInterestingPaperProject/pandocomatic.yaml:

# File: ~/Documents/publications/myInterestingPaperProject/pandocomatic.yaml 
# My pandocomatic YAML specification for my interesting paper
templates:
  author:
    metadata:
      author: Huub de Beer
   ...
  myrefs:
    extends: ['author', 'refs']
    pandoc:
      bibliography: "./myInterestingPaper.bib" #bibtex file, in the current project folder
      csl: "./apa.csl" #CSL style file, in the current project folder 
      ...
      author: Hans de Wolf

Consider the following excerpt from the current ~/.local/share/pandoc/pandocomatic.yaml:

...
#-----------------------------------------------------------------------------
  refs:
    pandoc:
      verbose: true # verbose by default
      citeproc: true
      bibliography: Core.json #JSON faster than BIB, symlinked within Pandoc data dir
      csl: csl/cell-doi.csl
      citation-abbreviations: cite-abbr.json # my journal abbreviations
      reference-links: true
    metadata:
      reference-section-title: Bibliography
      notes-after-punctuation: false
      link-citations: true 
      csl-hanging-indent: true
...

This configuration would have the following effect:

• author: Hans de Wolf # Overwriting the locally defined author with the new author's name
• bibliography: "./myInterestingPaper.bib" # Overwriting the generic json bib to this particular bib-file
• csl: "./apa.csl" # Overwriting the generic csl/cell-doi.csl reference style to the apa-style as requested by the journal

Caveat

On using identical template names for the (to be) inherited template between local and pandoc default specifications, i.e., one specified in the user-defined yaml-file and one with the exact same name specified in ~/.local/share/pandoc/pandocomatic.yaml, the user-defined local specification of the template shall apply. This implies that it is not possible to merge two templates with identical names.

I don't have a particular demand on excluding or allowing to merge values. I leave this to @htdebeer to decide how to bring this in accordance to the extension rules as defined in the documentation.

[htdebeer]

Thanks for your request.

Always taking into account the pandocomatic.yaml configuration file in user's pandoc configuration directory makes sense. As for potential conflicts, I suppose the natural order would be:

    internal template
>
    external template in config file
>  
    external template in pandocomatic.yaml in user's pandoc config directory
>
    external template in pandocomatic's default configuration

Although that last default configuration does not contain any templates.

PS The unreadable error you mention at the top has already been resolved with fac82c8b8f02cbaf699fd4947f671bb43cab721e. This fix will be included in the next release.

[htdebeer]

I think I've implemented global inheritance. Can you try pandocomatic-0.3.pre.alpha.gem and see if it behaves like you expect and want it to?

Also, @iandol, you might have a look at this to see if it doesn't break anything on your end. The change does not seem to have an effect on my test set, but I am using only simple templates and simple inheritance.

[iandol]

Using my standard pandoc setup and some fairly complex templates, I don't see any regressions. I haven't specifically tested this multi-file merging of metadata. How is it supposed to work, do you have to manually add both yaml files to the commandline, or do you specify the specific yaml file and the default one (or visa versa) is loaded "silently" ?

If the latter then this will change the behaviour of pandocomatic, and may lead to undesired side effects...

[htdebeer]

It is the latter: Now pandocomatic reads in first the "pandocomatic.yaml" configuration file in the user's pandoc configuration directory. Then "pandocomatic.yaml" in the data directory specified with option "--data-dir". Then the configuration file specified with option "--config".

I am not quite sure what the impact will be, but for my use cases and test cases this is not an issue: I define more specific templates in more specific configuration files and that behavior does not change. The advantage of loading all configuration files in the chain is that I can stop copying parts of templates from one configuration file to the next, but put them all in my global configuration files while still having the ability to specialize them for each project.

Anyway, because this is quite a change of behavior, conceptually speaking, I will introduce this for pandocomatic version 0.3 and later. I will not introduce this in the 0.2.x series.

[iandol]

I suppose it should be OK but does generate a possibly unobserved interaction. Would it be possible to have the config files used when pandocomatic runs shown in verbose mode, that way it would be easy to identify what config/metadata is utilised?

[htdebeer]

That is a good idea!

Showing a list of all configurations active, and their order is not a problem at all, of course. That would always be:

1. Source file to convert
2. Specific configuration file via --config option
3. Default pandocomatic configuration file in directory specified via --data-dir option
4. Default pandocomatic configuration file in pandoc's data directory (e.g., /home/you/.local/share/pandoc)

Where 2, 3, and 4 can be missing, and 1 changes for each conversion. Printing this list in verbose mode is no problem.

[plbt5]

Indeed a good idea. Since templates inhere from each other / are cascading to create the final configuration, it would be very helpful for the user to be informed about the order of cascading. As to your "natural order" and in reference to configurations 2-4 below, I'd expect 2 to overwrite 3, and 3 to overwrite 4, and the md-source internal yaml template to overwrite 2. But only for fields that they have in common, of course.

My 2 cts. Paul

On 24 Nov 2021, at 15:59, Huub de Beer @.***> wrote:

That is a good idea!

Showing a list of all configurations active, and their order is not a problem at all, of course. That would always be:

Source file to convert Specific configuration file via --config option Default pandocomatic configuration file in directory specified via --data-dir option Default pandocomatic configuration file in pandoc's data directory (e.g., /home/you/.local/share/pandoc) Where 2, 3, and 4 can be missing, and 1 changes for each conversion. Printing this list in verbose mode is no problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/htdebeer/pandocomatic/issues/96#issuecomment-977958101, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACAWGQOQEJYE7KTWXNR67A3UNT4VNANCNFSM5CDYMS4Q. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[plbt5]

More elaborate error messaging suggested.

An error occurrs when the pandcomatic.yaml configuration extends a template name that does not exist, as follows:

An unexpected error has occurred. You can report this bug via https://github.com/htdebeer/pandocomatic/issues/new.
Traceback (most recent call last):
	15: from /usr/local/bin/pandocomatic:23:in `<main>'
	14: from /usr/local/bin/pandocomatic:23:in `load'
	13: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/bin/pandocomatic:3:in `<top (required)>'
	12: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/pandocomatic.rb:58:in `run'
	11: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/cli.rb:42:in `parse'
	10: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/cli.rb:153:in `parse_options'
	 9: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/cli.rb:153:in `new'
	 8: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/configuration.rb:81:in `initialize'
	 7: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/configuration.rb:143:in `load'
	 6: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/configuration.rb:169:in `configure'
	 5: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/configuration.rb:169:in `each'
	 4: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/configuration.rb:181:in `block in configure'
	 3: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/configuration.rb:764:in `reset_template'
	 2: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/configuration.rb:743:in `extend_template'
	 1: from /Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/configuration.rb:743:in `each'
/Library/Ruby/Gems/2.6.0/gems/pandocomatic-0.2.7.6/lib/pandocomatic/configuration.rb:747:in `block in extend_template': undefined local variable or method `parent_template_name' for #<Pandocomatic::Configuration:0x00007f85d0100448> (NameError)

Naturally, the error should be thrown because of an invalid configuration. However, I couldn't figure out the source of the problem from the error message undefined local variable or method "parent_template_name". It took me some trial and error to figure out that the issue was located in the file pandocomatic.yaml. Next, it took me some time to identify what part of the file was wrong, and only then I quickly saw the light.

Given the more complicated template inheritance in the near future I'd suggest to improve this particular (and probably similar) error messages by including the name of the file, the name of the extending template and the name of the missing template that is being sought for extending.

[htdebeer]

Thanks for testing! Great you found some issues, because for a change like this I'd expect some, but was until now unable to find any. And that's scary.

I'll improve the diagnostic / error reporting. Any other requests regarding error reporting and diagnostics?

[htdebeer]

I've added the configuration hierarchy to the summary when running pandocomatic in verbose mode. Furthermore, when a template cannot be resolved, I give a warning with the list of templates I cannot find and the file the template is defined in. Both should help debugging template and inheritance issues.

This'll be in pandocomatic version 0.3. You can already test by checking out the latest version and running test/pandocomatic.rb.