Support platform specific import schemes

[lisael]

trafficstars

Standardize the way platform-specific imports are parsed, resolved and CBOR-encoded.

Use cases

Integration with the platform's packaging standards

A Dhall implementation may want to rely on language-specific semantics to locate dhall files. This permits users to use the language ecosystem's packaging standards to ship dhall files with an application or a library.

It's the use-case raised by Dhallj implementers:

let pkg = classpath://com.acme.myApp/package.dhall

With pydhall I plan to support pkg_resources data loading mechanism too.

Custom resolution routine

I'll show a pydhall example here, as it's actually the use case that triggered my reflexions. I don't want the discussion here to forget about this use-case :).

The pydhall API way to define a configuration schema is to define python classes:

from pydhall.schema import Schema, NaturalField
class MyConfig(Schema):
    a = NaturalField(default=42)

Then the end-user can import this schema in their dhall configuration:

let myApp = pydhall+schema://myApp.settings/MyConfig  -- load  MyConfig class from myApp.settings python module

This import resolves as:

{MyConfig = { Type = { a : Natural }, default = { a = 42 }}}

Proposed design

0. Custom imports are permitted by the standard
1. A custom imports MUST be a valid URI
2. Custom imports URI schemes SHOULD be namespaced with the name of the platform that first introduced them (pydhall+, dhallj+ ..)
3. The parser is modified to parse non-http URIs as custom import. The parser SHOULD split path components.
4. Custom imports are binary-encoded as imports, using the first unused scheme code (8 at time of writing): [24, <hash>, <import_type>, 8, <scheme_as_string>, <uri_authority>, <uri_path_element> ...]
5. Custom imports MAY be hash-protected
6. Resolution
   1. The implementation knows the scheme
      1. It's up to the implementation to resolve the import (including the handling of the import type (as Text,...) and of the chaining semantics)
      2. caching rules stay the same as those of standard import
         • same URI resolves as the same Dhall term durring an import pass
         • if hash protected, try to load the term from the cache and check the hash of the result
   2. The implementation doesn't know the scheme
      • the resolution fails
      • the implementation SHOULD raise a warning to the user

Notes

An implementation SHOULD provide a way to show the user the hash of an import. This permits the user to use standard Dhall tooling on dhall files that use custom imports:

let pkg = acme+loader://my.pkg/package.dhall ? missing sha256:deadbeef...
in ...

Is a correct input for dhall format, dhall normalize, dhall-to-json tools without modifying those, as long as the result of the import was cached beforehand.

[Gabriella439]

@lisael: Yeah, I think this is a good idea. The only issue I can think of is that dhall freeze or the language server's support for freezing imports wouldn't work on these custom imports unless we added support for plugin-based freezing

[lisael]

This is not easy to address :/. The best idea I got so far adds a lot of complexity, and I hope someone comes with something simpler:

• require implementations (that use custom imports) to provide a freeze command that reads an import statement from stdin and outputs the hash of the expression.
• add a $XDG_CONFIG_HOME/dhall.dhall that contains something like

let dhallcfg = https://raw.githubusercontent.com/dhall-lang/dhall-lang/v17.0.0/Prelude/Dhall/config.dhall
in dhallcfg::{ 
  , freezeCommand = toMap {
    , pydhall = "pydhall --freeze"
    , dhallj = "dhallj --freeze"
    }
  }

and change the tools that need to freeze imports so that they call the provided command according to the scheme namespaces (this requires that the implementations respect the namespace+scheme scheme form).

Note that the Prelude defaults could be pre-loaded with the configurations of the official implementations, and used if this file doesn't exist so most users won't have to create the configuration file as long as they use only official implementations).

This is a lot of work (on several projects) but it's the price to pay if we want to allow platform-specific import while not making platforms that chose to implement those second-class citizens regarding the tooling.

[lisael]

Of course the standardization and the implementation of this mechanism can be postponed after the standardization ot the original issue, that IMO adds value by itself, because at the moment none of the standard tooling can even parse dhall files that use custom imports.

[Gabriella439]

@lisael: Don't worry too much about it. I don't think it's a deal-breaker if dhall freeze doesn't support it