Automated packaging (`dhall package`)

[SiriusStarr]

I was going to write a script to do this, but thinking about it, I'm wondering if it should be included in dhall-haskell.

The standard for the prelude (and several community packages it seems) is to have a file in each folder named package.dhall that consists of a record of frozen imports of all files in that folder and subfolders/etc. For the Prelude, at the very least, maintaining this is easy but inconvenient (and leads to a lot of failing builds).

As such, I'm (idly) proposing a new command, dhall package to generate such files. dhall package would be called on a folder, e.g.

Prelude
├── Bool
│   ├── and
│   └── show.dhall
├── Integer
│   ├── toDouble
│   └── toNatural

and generate

Prelude
├── Bool
│   ├── and
│   ├── package.dhall
│   └── show.dhall
├── Integer
│   ├── package.dhall
│   ├── toDouble
│   └── toNatural
├── package.dhall

Behavior:

Files Included: By default, *.dhall[b] and extensionless files. Packaging fails with an error message of any such files fail to typecheck/hash/etc. (e.g. are not valid Dhall files). (Perhaps there should be an ignore flag, so that a non-dhall, extensionless file could be ignored? Or a flag to only consume files with the explicit Dhall extensions?)

Extant package.dhall Files: Any new entries, removed entries, or changed hashes should be logged in the terminal output. Perhaps should ask for confirmation in such cases (with a flag to force)? Alternately, could include a format indicator comment at the start of package.dhall files so-packaged, e.g.

{- Packaged by dhall package vX.XX -}

and automatically overwrite such package.dhalls but abort if there's a package.dhall lacking that (since it was presumably made by hand then).

Anyways, as I said, somewhat idle thought.

[Gabriella439]

@SiriusStarr: I really like this idea!

[ari-becker]

in general, I'm interested in an idea like this. My main input is that I'd like there to be a flag which only freezes the imports in prior-existing files, rather than generating package.dhall files afresh each time.

We have a pretty common use-case whereby the package.dhall includes Kubernetes objects in the exported record, plus a field called objects which lists the names of the exported fields. It perhaps sounds redundant, but it's a great way to control the order by which jq processes the record (and to effectively disable specific objects in a pinch) and passes the contents to kubectl.

in other words... sometimes there are "smart" packages, and it would be nice if it were possible for this command to help with existing work (i.e. by focusing on freezing recursively).

[SiriusStarr]

@ari-becker

My main input is that I'd like there to be a flag which only freezes the imports in prior-existing files, rather than generating package.dhall files afresh each time.

I am all for something like this, but do you think it would be better as a flag for dhall package or a flag for dhall freeze? E.g. dhall freeze --recursive or something (which you'd just point at your manual package.dhall)?

[Gabriella439]

@SiriusStarr: I definitely think dhall freeze needs something like a --recursive flag, especially because if you freeze files in the wrong order some hashes can be incorrectly updated

For example, a common issue is:

• File A depends on B, and B depends on C
• C is changed
• A is refrozen, but picks up the old version of C (because B's reference to C was frozen)
• B is refrozen, which picks up the new version of C (but A still refers to the old C)

[ari-becker]

I agree - dhall freeze --recursive is the better pattern.

Indeed, I view dhall package as more of a reasonable default - for example, users may afterwards wish to decide to remove some of the files from the package since they're meant to share logic between other expressions in the same directory and aren't meant to be exported - renderInteger in the Prelude, which is shared between render and renderYAML but isn't really meant to be used downstream, being a good example (albeit conceding that it does indeed figure in the final package). If there is a desire for a dhall package --freeze flag (because discoverability?), internally it should probably just redirect to dhall freeze --recursive.