protocolbuffers
Decimal Type?
It'd be amazing to have a Decimal type as a Well Known Type that can be leveraged across all the languages. I have no preference around the back-end implementation of it, as an end user I just think it'd greatly simplify codebases using Decimals
By "Decimal", I guess you mean something like java.math.BigDecimal in Java that can represent arbitrary precision. I think we should only introduce it if such a decimal type is supported as a built-in type or standard library type by all programming languages that we officially support. That probably won't happen any time soon though.
The java implementation could understand how to serialize BigDecimal or BigInteger to/from strings. I think this is a missed opportunity. Without it this causes developers to use floating pointing types instead, and use these type for calculation which may be bad. Leveraging protobuf to promote these types would help to use consistent and safe way to handle precision numbers.
I'm currently struggling with it myself (I'm using C#). I think you can't avoid having to map(convert) to decimal manually everytime you want to use decimal.
Some people pointed to https://github.com/googleapis/googleapis/blob/master/google/type/money.proto but there's no information on how to convert from nanos and to nanos.
And it gives you the same result as just storing it as a string.
I wanted to rewrite my code to use gRPC but the state of things with decimal is really putting me off.
Maybe a util method from protobuf's supported type (e.g., string) to the decimal type in the specific language?
@TeBoring that'd be the easier one. I'd rather have this than Avro's bytes method, as it destroys the user experience, but I'm sure their choices would be important to understand https://avro.apache.org/docs/1.8.1/spec.html#Decimal https://issues.apache.org/jira/browse/AVRO-1402
We are also using decimals quite heavily in our application and this is currently an issue for the migration to protobuf. Is there any update on this?
@TeBoring I would say this is not C# specific. We have:
- BigDecimal in Java https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/math/BigDecimal.html
- decimal in Python https://docs.python.org/3/library/decimal.html It is really important for all systems handling finance data. I also think a util function converting from string would be bad from a performance perspective. Better to use something like the money type, without currency, as input to the util function. Or look on Avros Logical type for decimal https://avro.apache.org/docs/1.8.0/spec.html#Decimal. Both ways would be more efficient than string.
https://avro.apache.org/docs/1.8.2/spec.html#Decimal When we talk about 'Decimal' most of us mean type that can be keep 'precision' and 'scale' value for float values. Avro is good example, they use unify serialisations of decimal to Bytes + keep two additional properties (scale, precision). Why its important: All use same serialisations process Previously some serialised it to string, some to Byte Array with Ascending Order, some With Descending, etc Now every one use same bytes, with same order, and keep metadata in same fields.
decimal would be very handy, I'm currently converting to a byte array base-256 digit, I guess is the most efficient way
"While suitable for many purposes, binary floating-point arithmetic should not be used for financial, commercial, and user-centric applications"
Yes, Google we need a Decimal type! This is an example of the issue and probably demonstrates the issue in any language: bool result = ((0.1 + 0.2) == 0.3); // result is false
The following link provides a detailed explanation and examples to explain the motivation for the "Decimal" data type, which is available in Python, Java and C#. http://speleotrove.com/decimal/
Update: I'm currently driving internal proposal for the "decimal" type and I published the proposal document externally, so that people can chime in as needed.
See https://github.com/protocolbuffers/protobuf/pull/7039
There's class BigDecimal in Ruby. So strong +1 to add Decimal to protobuf types 👍
I'd get rid of workaround for Google::Protobuf::TypeError: Expected number type for double field (given BigDecimal).
Decimals are exceptionally popular in C#, especially with database-backed business software. Almost all the "number" types in our application (and there is a lot of them) are decimals, there are only few floats here and there. Since float is not a safe replacement, we would end up mapping decimals to strings which is sort of embarrassing (and unsafe since not every string is a valid decimal).
Since it's a missing fundamental type, it should be in WellKnownTypes (as the suggested FixedDecimal). Alternatively, there should be another source of these "well known types" to avoid everyone copy-pasting the FixedDecimal to their codebases (in different namespaces)
This doesn't solve this in the WellKnownTypes land, but here we use:
message BigDecimal {
string value = 1; // feel free to substitute for a variant of precision/scale/unscaled
}
Then in the csharp land we can define some implicit/explicit casting operators or conversion methods via extension of the partial class:
public partial class BigDecimal
{
public static implicit operator BigDecimal(decimal value)
{
return new BigDecimal
{
Value = value.ToString(CultureInfo.InvariantCulture),
};
}
public static implicit operator decimal(BigDecimal value)
{
return decimal.Parse(value.Value, CultureInfo.InvariantCulture);
}
}
Are there any preview ports of this? Writing a finance app with millions of messages per second, non-native support and casting each time is a non-starter for gRPC currently without this
For those working with decimals, can I confirm that there are two approaches that we are left with while we await movement on #7039 :
- Use of string marshalling for decimals
- Use of a custom protobuf message with either units and nanos (ala Google's Money.proto example) or an coefficient, exponent and sign (ala big.js)
Then hand off the expected marshalling of such messages to internal implementations of each application.
If this is the case, can people chime in with their preferred approach to date?
For those working with decimals, can I confirm that there are two approaches that we are left with while we await movement on #7039 :
- Use of string marshalling for decimals
- Use of a custom protobuf message with either units and nanos (ala Google's Money.proto example) or an coefficient, exponent and sign (ala big.js)
Then hand off the expected marshalling of such messages to internal implementations of each application.
If this is the case, can people chime in with their preferred approach to date?
Many thanks! Personally, would prefer custom profobuf message (this is what i have been doing to date). If there are any one-time/optimised marshalling that can be implemented as part of the codegen to prevent repeated casting that would be my main concern
There's a reference to this and a workaround here for wcf/recode, but I can't believe this is still an open issue https://visualrecode.com/blog/csharp-decimals-in-grpc/
Likewise. Decimal is an absolutely critical type for handling money in C#. Please add this to WellKnownTypes.
Thank you!
Now that Google killed https://github.com/protocolbuffers/protobuf/pull/7039, what is going to happen with this ticket and/or C# decimal support?
My intention is to leave this ticket open. The desire is not invalid, it is simply the case that we are not actively pursuing it.
The signal is strongly (see @vivainio's comment above) that Google's team has placed any updates to support a decimal type on ice.
In the meantime, projects will need to DIY support for fixed decimal types.
Need support for Decimal128, so new fixed width wire type.
Workaround is to use with additional validation, but would prefer native scalar Decimal128.
Big Decimal can also be represented with bytes, or else string.
As noted above, we have no intention of pursuing this in the near term. The language specific aspects of this make it a very large amount of effort to implement across our entire surface. I recommend that users use a string field with appropriate parsing in their own code.
As noted above, we have no intention of pursuing this in the near term. The language specific aspects of this make it a very large amount of effort to implement across our entire surface. I recommend that users use a string field with appropriate parsing in their own code.
The decimal data type is not a language specific type. It is an IEEE standard -https://en.wikipedia.org/wiki/IEEE_754#Decimal In particular it is extensively used in SQL storage systems. The recommendation to use a string misunderstands the problem. Decimal is a bitwise encoding therefore one cannot transform from a string field to a correctly encoded decimal type.
Probably what needs to happen is for somebody to create a PR showing Google exactly how to implement a solution, since Google engineers appear to consistently misunderstand this issue.
While you are welcome to create a PR, the odds that it is adopted are low. As mentioned, any sort of increased precision datatype needs to be handled in the full set of languages supported consistently which makes this a substantially larger amount of effort then most anticipate. For every primitive datatype in protobuf that type must be reified into some language specific type for each target language.
Decimal as you have pointed to is a general encoding of a particular format for storing arbitrary precision numbers. It can in fact be stored as a string field (or rather a bytes field).