Allow passing defaults on the command line

[quasicomputational]

trafficstars

This is not a 100% baked idea, but hpack could accept defaults on the command line (and that could then be fed in by higher-level tools). Imagine if you have a monorepo with a few dozen Haskell packages sharing defaults: it might only be a line or two in each package.yaml, but that adds up across the whole project. If stack could learn to point hpack to a defaults file, specified in stack.yaml once, then that becomes a lot less boilerplatey.

[sol]

On the one hand I like this idea, on the other hand what ever we do here has to, at least in principal, also work for other build tools (specifically tinc).

[sol]

defaults aren't widely used yet. Before doing anything here, I would want to see defaults more widely used first. I'm closing this for now, but open to discuss it again in the future.

[srid]

@sol I'd like to see this happen as well. It will help projects using Nix. Because of reproducibility of nix, I can pass a hashed path (from store path) to hpack as part of the pre-commit-hook script.

Current approach

Add defaults: user/repo@branch to all package.yaml files

Downsides

• Repeat the defaults on all package.yaml files. If branch is to be changed, modify all those files again.
• branch is not reproducible, as HEAD revision can change. While you can specify a rev, there is no guarantee the hash may not change.

Proposed approach

Allow hpack --defaults-from=<some-path>, so package.yaml file need not be modified.

Upsides

• No need to modify N package.yaml files.
• Fully reproducible through flake.lock (the defaults repo is specified in flake.nix, and its rev/hash is locked in flake.lock)
• Integration with https://github.com/cachix/pre-commit-hooks.nix

[srid]

Failing this, I'd workaround this by creating a wrapper hpack script that injects the defaults on the given package.yaml before invoking the actual hpack. But that'd be pretty hacky.

[srid]

Oh, one more reason to accept defaults from the command line. hpack is sometimes invoked from nix flake checks, which run in a sandbox environment with no network access, leading build failures in the CI like this:

[1370](https://jenkins-nix-ci.betta-gray.ts.net/job/nammayatri/job/PR-2930/6/pipeline-console/?start-byte=478212&selected-node=52#log-1370)
error: builder for '/nix/store/yzkz0s0kk5jk3vrlj2hy9q8h380lgsk8-pre-commit-run.drv' failed with exit code 1;
[1371](https://jenkins-nix-ci.betta-gray.ts.net/job/nammayatri/job/PR-2930/6/pipeline-console/?start-byte=478212&selected-node=52#log-1371)
       last 10 log lines:
[1372](https://jenkins-nix-ci.betta-gray.ts.net/job/nammayatri/job/PR-2930/6/pipeline-console/?start-byte=478212&selected-node=52#log-1372)
       >   requestVersion       = HTTP/1.1
[1373](https://jenkins-nix-ci.betta-gray.ts.net/job/nammayatri/job/PR-2930/6/pipeline-console/?start-byte=478212&selected-node=52#log-1373)
       >   proxySecureMode      = ProxySecureWithConnect
[1374](https://jenkins-nix-ci.betta-gray.ts.net/job/nammayatri/job/PR-2930/6/pipeline-console/?start-byte=478212&selected-node=52#log-1374)
       > }
[1375](https://jenkins-nix-ci.betta-gray.ts.net/job/nammayatri/job/PR-2930/6/pipeline-console/?start-byte=478212&selected-node=52#log-1375)
       >  (ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 0, addrAddress = 0.0.0.0:0, addrCanonName = Nothing}, host name: Just "raw.githubusercontent.com", service name: Just "443"): does not exist (Temporary failure in name resolution))

[mpilgrem]

I was wondering: if a package is a 'unit of distribution', and a package includes its own specification, then is a package specification that relies on information outside of the package to specify the package, in a sense, malformed?

[srid]

@mpilgrem Valid concern ... which is somewhat alleviated, in our case, by generating (via pre-commit-hooks) and committing the .cabal file in git. Thus, the 'unit of distribution' includes the cabal file only (not hpack configuration).

What we want, in practice, is project-level defaults for all packages in a monorepo. Per-repo global package.yaml that gets imported in all local packages in that repo. How do we do this?

defaults.local works, but since the local packages can live at any level of nesting (see https://github.com/nammayatri/nammayatri), you are going to have to specify a different value for all the package.yaml in the sub directories.

[sol]

defaults.local works, but since the local packages can live at any level of nesting (see https://github.com/nammayatri/nammayatri), you are going to have to specify a different value for all the package.yaml in the sub directories.

I tend to deal with this by creating symlinks. It's still not 100% DRY as you have to create those symlinks, but it works reasonably well for me.

Also note that a local defaults file can itself contain a defaults statement. So you could put the revision into a local defaults file and then include that local defaults file everywhere else. If the revision changes, you only have to update it at one single place.

[sol]

Allow hpack --defaults-from=<some-path>, so package.yaml file need not be modified.

I am not ready to explicitly support defaults from the command line just yet.

However, GitHub defaults are cached under ~/.hpack/defaults/<user>/<repo>/<ref>/. If that path already exists, then a GitHub defaults statement essentially behaves the same way as a corresponding defaults.local statement.

Specifically

defaults: user/repo@ref

effectively desugar to

defaults: 
  local: ~/.hpack/defaults/user/repo/ref/.hpack/defaults.yaml

By controlling the file system, you can make hpack use whatever defaults you want + you can also prevent hpack from accessing the network.

Does this work for you?

We can make the base path ~/.hpack/defaults configurable. I imagine that way you have to jump through less hoops to make things work.

[sol]

@srid also keep in mind that defaults are somewhat limited, e.g. a top level defaults statement can only define top level fields, not per-section fields.

If you run into limitations then you might want to give YAML imports a try (instead of or in addition to local defaults): https://github.com/sol/hpack#not-repeating-yourself

[sol]

I was wondering: if a package is a 'unit of distribution', and a package includes its own specification, then is a package specification that relies on information outside of the package to specify the package, in a sense, malformed?

Yes, exactly, that's why I hesitate to add that feature.

[srid]

Does this work for you?

Certainly!

Is this a feature to be implemented, or does it already work?

[sol]

This should already work, except for that the base directory is not configurable yet (actually it kind of is, you could change it by setting HOME, I guess).