align retention.size format with IEC units

[Morriz]

It is now not possible to specify 5G or 5Gi, but we have to provide 5GB. This seems like an arbitrary decision to not adopt the k8s standard, but I am probably unaware of limitations in this matter. (Like, is it not possible to use i (as in Gi) to round to thousands?)

Proposal

Use case. Why is this important? To not have to learn multiple formats.

[roidelapluie]

it is unfortunate that the two projects use different notations. 5G or 5Gi misses the unit. To answer your secondary question, if you need to round up to thousands, you would need to use GiB (notice the extra B).

[Morriz]

Wouldn't it be beneficial to put this on the roadmap for alignment?

[roidelapluie]

Prometheus is independant for kubernetes. In this case, I think we do the right thing by enforcing a unit.

We also support time-based retention, which can be expressed in many units as well.

However, it could be a reasonable thing to propose to the Prometheus Operator, they could bridge the gap there.

[beorn7]

Prometheus uses Kingpin, and it is simply using the normal standard Kingpin ByteVar type here. Arguably, deriving from it would cause even more confusion (Kingpin conventions for all flags, except for bytes, where it uses K8s conventions…).

Please also note that the prefixes are always base-2. Both 10KB and 10KiB will result in 10240 bytes. (That's again how Kingpin decided to handle it.)

[roidelapluie]

Please also note that the prefixes are always base-2. Both 10KB and 10KiB will result in 10240 bytes. (That's again how Kingpin decided to handle it.)

To clarify, I think this only applies to "KB" and "KiB", not GB and GiB (https://github.com/alecthomas/units/blob/master/si.go).

[beorn7]

@roidelapluie I think this code is multiplying 1024 for all prefixes. The weird special-case handling at the bottom is just making sure that both kB and KB works for metric mode, while in binary mode, KiB and KB works but not kb.

[roidelapluie]

@roidelapluie I think this code is multiplying 1024 for all prefixes. The weird special-case handling at the bottom is just making sure that both kB and KB works for metric mode, while in binary mode, KiB and KB works but not kb.

ooh you are right.

[frittentheke]

Apart from the configuration not yet allowing the IEC (https://en.wikipedia.org/wiki/Binary_prefix#IEC_prefixes) prefixed to indicate binary units, ( as in GiB) there already seems to be a display inconsistency when looking at the flags at runtime:

The documentation at https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects clearly states that --storage.tsdb.retention.size is "Based on powers-of-2, so 1KB is 1024B. ".

I therefore set --storage.tsdb.retention.size=750GB, the only supported format for the Gigabyte unit. When looking at prometheus_tsdb_retention_limit_bytes being 805306368000 which is 75010241024*1024 this also seems accurate.

But when then looking at "http://localhost:9090/flags" is see that this is "converted" to --storage.tsdb.retention.size 750GiB using GiB. In any case, while potentially painful once when asking folks to change their config, it would be great to align any uses of units to avoid further confusion.

Adding / allowing scientific units might also be nice, sometimes that's what you got from your configuration management or whatever data source you use to set the tsdb.retention.size value ...

@Morriz maybe the title of this issue should better ask for the alignment with the standardized "IEC units" (https://en.wikipedia.org/wiki/Binary_prefix#IEC_prefixes) than pointing to "kubernetes sizes" which is just another piece of software using them.