dhall-lang
dhall-lang copied to clipboard
dhall-lang
Octal numerals
The topic of octal notation has recently come up in https://github.com/dhall-lang/dhall-haskell/issues/1988 and https://github.com/dhall-lang/dhall-kubernetes/issues/139.
Particularly for file permissions, it would be useful if Naturals could be written in octal notation, e.g. 0o10 for 8.
For reference, Dhall already supports hexadecimal numerals: https://github.com/dhall-lang/dhall-lang/issues/858
To give another instance of not having octal numerals handled well in Dhall, that isn't related to k8s, I am trying to manage my Fedora CoreOS Configs which are YAML files with Dhall. The FCCT tool translates human readable Fedora CoreOS Configs (FCCs) into machine readable Ignition Configs and it expects mode directives to be written in octal too.
Take this for example:
variant: fcos
version: 1.1.0
storage:
files:
- path: /opt/file
contents:
inline: Hello, world!
mode: 0644
user:
id: 500
group:
id: 501
Converting it to dhall strips the leading zero:
cat fcos.yml | yaml-to-dhall
{ storage.files =
[ { contents.inline = "Hello, world!"
, group.id = 501
, mode = 644
, path = "/opt/file"
, user.id = 500
}
]
, variant = "fcos"
, version = "1.1.0"
}
If we add the zero back in:
{ storage.files =
[ { contents.inline = "Hello, world!"
, group.id = 501
, mode = 0644
, path = "/opt/file"
, user.id = 500
}
]
, variant = "fcos"
, version = "1.1.0"
}
It produces this error:
cat fcos.dhall |dhall-to-yaml
Error: Invalid input
(input):4:15:
|
4 | , mode = 0644
| ^
unexpected '6'
expecting ',', ->, :, keyword, operator, whitespace, or }
As a note, it would be pretty painless to add octal literals because they are currently a syntax error: #898
As a note, it would be pretty painless to add octal literals because they are currently a syntax error: #898
@philandstuff Is this an argument to use 0 for the octal notation prefix?
I originally wanted to propose the prefix 0o because that seems less ambiguous to me.
If we were designing in a vacuum without the baggage of history behind us, I’d agree that 0o is a better prefix. But there is a long tradition of using 0 as a prefix which will be familiar to our target market of sysadmin-type people. For example, I’ve seen lots of unix file permissions in 0755 form, I’ve never seen them in 0o755 form, and I feel this would be a speed bump to our target users.
Interestingly, even YAML has moved to using the 0o prefix in v1.2.
Currently, I feel that a minor speedbump is acceptable, if it helps make the notation more readable and less ambiguous, as in this case.
I agree with @philandstuff here. Any sysadmin is going to write modes as 755 or 0755 and keeping things as interchangeable as if they were using chmod on the CLI is most likely the better experience.
there is a long tradition of using 0 as a prefix
It seems to me to conflict with "Beginner-friendlyness", as it's ambiguous I like 0o better or to skip Octal (I use e.g. u+x for chmod, the friendly syntax).
My personal preference is 0o, mainly based on Wikipedia saying that 0o is what most new language implementations are using: https://en.wikipedia.org/wiki/Octal#In_computers
To give another instance of not having octal numerals handled well in Dhall mode: 0644
Converting it to dhall strips the leading zero:
If the leading zero is required, I would say it is semantically a string. Or else an oddity of output format.
If the output format supports both octal with leading zero and also decimal without, you could perhaps simply write the decimal number?
File modes, for example, are not a number and should not be represented as one in Dhall. They are a tuple of lists of known enums, represented as a bitfield in some APIs. So write the real type in Dhall, and if your API requires a particular format have a function or a dhall-to-format system that turns the type into the required format, depending on need.
If the leading zero is required, I would say it is semantically a string. Or else an oddity of output format. If the output format supports both octal with leading zero and also decimal without, you could perhaps simply write the decimal number?
It can't.
File modes, for example, are not a number and should not be represented as one in Dhall. They are a tuple of lists of known enums, represented as a bitfield in some APIs. So write the real type in Dhall, and if your API requires a particular format have a function or a dhall-to-format system that turns the type into the required format, depending on need.
Those of us who understand how file modes are implemented can probably agree that their literal data structure is too heavy-handed for Dhall - a user shouldn't have to jump through those hoops when they are writing mode: 0755 in YAML. The problem here is if Dhall is going to import YAML to Dhall and make changes without informing the user, it is going to lead to things not working. There are two examples of this case in this thread alone.
I have a PR up for the 0o syntax: https://github.com/dhall-lang/dhall-lang/pull/1062
@jdoss the linked issue appears to be asking to be allowed to write octal numbers as though they were decimal numbers. I'm saying that if the output format supports octal numbers as numbers, then likely just writing the decimal numbers as numbers will work (so for example for 0644 you would write 420 in the output).
On the contrary, I think anyone writing 0755 directly is probably just relying on memory or something. chmod certainly supports actual symbolic arguments these days, and I see no reason to prefer writing out a bitfield in octal of all things vs a more sensible enumeration, especially in a nicely typed language like Dhall.
Relatedly, it seems that @jdoss has found a bug in yaml-to-dhall: namely, that YAML treats leading zero as octal, but the resulting Dhall ends up containing the octal digits as if they were decimal, but with leading zero stripped. The octal should be converted to a Natural and written as either decimal or hexidecimal on the dhall side of the conversion.
@singpolyma: Just to clarify, the intention of #1062 wasn't to fix https://github.com/dhall-lang/dhall-haskell/issues/1988. The discussion beginning around here clarifies that they are disjoint issues like you note:
https://github.com/dhall-lang/dhall-haskell/issues/1988#issuecomment-672352319
I only created the PR to standardize octal support because @sjakobi created this issue and it seemed easy to do, but I'm not strongly attached to the feature.
@singpolyma: Actually, I forgot that there is a minor reason for having language support for octals, which is primarily for dhall-to-yaml, so that it can convert Dhall octal literals to YAML octal literals as noted here: https://github.com/dhall-lang/dhall-haskell/issues/1988#issuecomment-673535075
… but again, it's a minor thing that I'm not that attached to
I'm confused: wouldn't
...
, mode = "0644"
...
give the correct YAML output? And wouldn't an octal numeral be output as a completely different decimal number? Otherwise that means dhall has to track which numerals are in which base, which seems like too much work. So I don't see why octal numerals would help here.
I'm confused: wouldn't
... , mode = "0644" ...give the correct YAML output? And wouldn't an octal numeral be output as a completely different decimal number? Otherwise that means dhall has to track which numerals are in which base, which seems like too much work. So I don't see why octal numerals would help here.
Not in my use case. FCCT expects mode: to be an octal numeral and not a quoted string.
cat host.yml | yaml-to-dhall | dhall-to-yaml | podman run -i --rm quay.io/coreos/fcct:latest --pretty --strict
Error translating config: yaml: unmarshal errors:
line 14: cannot unmarshal !!str `0644` into int
line 19: cannot unmarshal !!str `0644` into int
line 24: cannot unmarshal !!str `0644` into int
line 29: cannot unmarshal !!str `0644` into int
line 37: cannot unmarshal !!str `0644` into int
line 43: cannot unmarshal !!str `0644` into int
line 48: cannot unmarshal !!str `0644` into int
@Nadrieril: For YAML specifically it wouldn't require any changes to the Haskell implementation's AST to keep track of octal. We already preserve the original source code for numeric literals and we use that when rendering them, which is why dhall format preserves hex literals, for example.
If we were designing in a vacuum without the baggage of history behind us, I’d agree that
0ois a better prefix. But there is a long tradition of using0as a prefix which will be familiar to our target market of sysadmin-type people. For example, I’ve seen lots of unix file permissions in0755form, I’ve never seen them in0o755form, and I feel this would be a speed bump to our target users.
It's more than just tradition, it came from the C language (standardized 1989)
@Nadrieril: For YAML specifically it wouldn't require any changes to the Haskell implementation's AST to keep track of octal. We already preserve the original source code for numeric literals and we use that when rendering them, which is why
dhall formatpreserves hex literals, for example.
I see. I imagine that means we can't do any operations on those Naturals, as this would lose track of the octalness. So maybe an octal/hex numeral could be a new opaque value, like Double?
It seems as if octal numerals have precisely one use case: file permissions. Maybe it’s worth reframing / renaming this issue to be around support for that specific use case. I don’t think anyone really wants octal for any other reason.
So maybe an octal/hex numeral could be a new opaque value, like Double?
I’d be open to this for octal (not for hex, that’d be a breaking change) although I anticipate people might want the equivalent of bitwise operations on it (eg 0700 & 0644 in C syntax, which evaluates to 0600). But we can standardise as opaque and add operations if it actually comes up. (I don’t have this need personally)
@philandstuff: Just to clarify: I assume that you mean that it would no longer an arbitrary octal number, but rather always one with 3 digits, specialized to the use case of permissions. Is that correct?
No, I’m just saying that although the headline issue is about octal numbers, the thing that people actually care about is file permissions. Octal numbers are one solution to the problem, but others have been proposed (in this thread and elsewhere). My hope is that by focussing on the problem to be solved, rather than the solution, we can have a more focussed discussion.
As an example, in the generic octal numbers case you’d want arithmetic operations, but this is simply not needed for file permissions. On the other hand, file permissions might want (the equivalent of) bitwise operations.
I don’t feel strongly about any of this, though.
