Allow loading of any Dhall value as Void

[blamario]

This is somewhat related to issues #505, #930, and #1480. The project I'm working on has a Haskell executable that can be given one of many available Dhall configuration files on its command line. The configuration files all have the same type, and they typically share much of their value, so they reuse each other a lot.

So far so good, a perfect use case for Dhall. Unfortunately, one component of each configuration file is a homogenous record with varying field names. Because the field names are not fixed, Haskell can't load the record. This I fixed by applying toMap to each record, and now the application happily loads them.

Did I say fixed? Well, yes insofar the loading works, but the original record was so much easier to reuse and adjust between configurations, like

let cfg1= "../otherConfiguration/cfg.dhall"
in cfg1 // {
    record= {
        foo= cfg1.record.foo,
        bar= cfg1.record.foo // {
            field= cfg1.record.foo.field ++ " modified"
           }
        }
   }

I've solved this problem as well, by splitting each configuration file into

• a reusable configuration file with the record, and
• a small wrapper file that imports the above reusable config and applies toMap to the record, ready to be loaded.

It all works now, but I'm not happy with the doubled file count. So I started thinking of solutions, and the easiest one I can think of would be to have both the record and the map as two fields in the same configuration file. The record would be there for reuse, the map for loading. The only thing that's missing is the ability to tell Haskell to ignore the record, because we don't know its type.

So here finally is the proposal: let Haskell load any Dhall value of any type if its expected Haskell type is Void. In other words,

input auto val :: IO Void

would always succeed, provided that val contains a valid Dhall expression. The point of this feature is obviously not to just verify the configuration and load nothing, it's to load the parts of the configuration that can be typed in Haskell while ignoring the rest.

[blamario]

I'm an idiot. Please read every occurrence of Void as () in the proposal above. Alternatively it could be a new data type Ignorable exported by Dhall.Import for the purpose.

[sjakobi]

Would you mind showing a worked example using this Ignorable type?

[blamario]

Haskell:

data ValueConfig = ValueConfig{
    interestingStuff :: Text
    }
data Config = Config{
  record :: Ignorable,
  mappings :: Dhall.Map.Map  Text ValueConfig,
  otherStuff :: Dhall.Natural
  }

Dhall:

let ValueConfig = {
   interestingStuff : Text
   }
let record= {
         foo = {
           interestingStuff ="Foo"
           },
         bar = {
           interestingStuff ="Bar"
           }
         }
in {
  record= record,
  mappings= toMap record,
  otherSuff= 4
  }

[Gabriella439]

Normally the solution I would suggest here is to add a FromDhall instance for Expr (analogous to the FromJSON instance for Value) that lets you marshal an arbitrary schema-free Dhall expression into Haskell. There's a catch, though, which is that it's not compatible with our current Decoder type which expects us to specify an expected Dhall type, so we would need to figure out a way to work around that.

[blamario]

The catch is exactly the problem.

[Gabriella439]

@blamario: The only thing I can think of at the moment is disabling the type-checking step for expressions that have an Expr somewhere inside of them. It's sub-optimal, though

[blamario]

What is the problem with the Ignorable solution? Is it not implementable?

[Gabriella439]

@blamario: I believe the Ignorable solution has the same limitation as the Expr-based solution

[sjakobi]

This is probably a dumb question, but I'm wondering why wrapping the input Text or Expr isn't good enough:

For example if we take the Dhall config example from above as t, you could simply form

"(" <> t <> ").{ mappings, otherStuff }"

to ignore the problematic record field.

If the input config doesn't contain the mappings field (which already feels like a workaround to me), you could have something like

"let config = " <> t <> " in config.{ otherStuff } // { mappings = toMap config.record }"

This might be more elegant on the Expr level, and I guess we could consider offering helpers for constructing those.

[Gabriella439]

@sjakobi: You also don't need to use text interpolation if the configuration is saved within another file:

"let config = ./config.dhall in …"

[blamario]

I did consider that class of solutions as well, but it felt ... tacky. The reason I'm switching to Dhall is not because I'm a fan of stringly typed programming. I think I prefer the existing two-file solution.

[blamario]

I want to add that a principled equivalent of Dhall.Ignorable type would be forall a.a. It would be perfectly sound to allow loading anything whatsoever with that type. The implementation may be a different matter.

Also I want to reply to @sjacobi 's

If the input config doesn't contain the mappings field (which already feels like a workaround to me)

Yes, the whole toMap business is a workaround for #1480 to start with. It turns out it's not a complete workaround for my purposes.

[Gabriella439]

@blamario: Using forall a . a would not fix the problem. The implementation would actually verify that the marshalled expression has that type (and would necessarily fail to do so)

I still maintain that if we go this route (selectively disabling the type-checker) the right solution is to add an Interpret instance for Expr rather than a new Ignorable type

[brendanhay]

I've literally run into this today - desiring an Interpret/FromDhall Expr instance to avoid actually having to do my day job.

The use case is as follows:

• (Very) large configurations using dhall-kubernetes.
• Partially-applied configuration is deserialised into Haskell in the form : Config -> ... -> [<KubernetesTypeUnion>].
• The above function is saturated and then each list item is submitted to the Kubernetes API via the requisite functions from the kubernetes-client Haskell package.

Except that mechanising To/FromDhall instances for the hundreds of involved Kubernetes types from an existing package such as kubernetes-client-core results in a myriad of subtle mismatches between naming (IP vs Ip, etc.), what is Optional vs required, how maps are represented, and so forth. I'd rather not deal with writing another OpenAPI generator just to needlessly deserialise these types when what I really care about is:

• Deserialise a partially-applied function of the form : Config -> ... -> [Expr s Void].
• Fully apply and call the equivalent of map dhall-to-json to get a list of Aeson Value.
• Send each Value merrily on its way. (In this case by checking the kind: ... JSON key and again calling kubernetes-client, FWIW.)

Being a responsible adult, having all my types and configuration defined in Dhall means you can expect me to run dhall for type checking etc. on the configuration to ensure it is well behaved, as the Dhall code is the source of truth. Haskell types and values are undesirable (and prohibitive) in this scenario. This morally seems to be the same as how dhall-json and dhall-yaml work, as an example.

[brendanhay]

@Gabriel439 Out of interest, is it possible to interpret a function a -> b -> Expr Src Void with the way the Encoder/Decoders are structured currently, even if it means hand-rolling the machinery?

[Gabriella439]

@brendanhay: It's not clear what you mean, but if all you need to do is to decode the Expr that the function returns you can use:

fmap (fmap (extract auto))
    :: FromDhall c => (a -> b -> Expr Src Void) -> (a -> b -> Extractor Src Void c)

[brendanhay]

@Gabriel439 In line with the thread, I want to preserve the Expr, not decode it, since there exists no Haskell data type representing the expected type - I merely want it to be well formed such that I can pass it to dhall-to-json, for example.

Some example desired Haskell representations:

newtype Config = Config (Foo -> Bar -> Expr Src Void)

data Foo
    = Bar (Expr Src Void)
    | Baz (Expr Src Void)

The question is - is it possible to decode these currently?

[Gabriella439]

@brendanhay: Currently it is not possible to decode Expr, but once we can decode Expr then we would be able to decode derived types that depend on Expr, such as your Config and Foo examples