danielgtaylor
Timestamp bad parsing
During the investigation of this great library, me and my team faced a bug which is most likely coming from python special ability to fail in datetime covnverting to timestamp (and backward)
For example, we have a datetime object incoming: Friday, 12 January 1968, 20:06:43.200
In timestamp format it is -62135596800.0 which we use in our proto models
But when the code of this library is trying to convert it back to datetime, it fails with exit code 22:
return await self.unary_unary( File "\venv\lib\site-packages\betterproto_init.py", line 1124, in _unary_unary response = await stream.recv_message() File "\venv\lib\site-packages\grpclib\client.py", line 428, in recv_message message = await recv_message(self.stream, self.codec, File "\venv\lib\site-packages\grpclib\stream.py", line 32, in recv_message message = codec.decode(message_bin, message_type) File "\venv\lib\site-packages\grpclib\encoding\proto.py", line 54, in decode return message_type.FromString(data) File "\venv\lib\site-packages\betterproto_init.py", line 779, in FromString return cls().parse(data) File "\venv\lib\site-packages\betterproto_init.py", line 759, in parse value = self.postprocess_single( File "\venv\lib\site-packages\betterproto_init.py", line 718, in postprocess_single value = cls().parse(value) File "\venv\lib\site-packages\betterproto_init.py", line 759, in parse value = self.postprocess_single( File "\venv\lib\site-packages\betterproto_init.py", line 718, in postprocess_single value = cls().parse(value) File "\venv\lib\site-packages\betterproto_init.py", line 759, in parse value = self.postprocess_single( File "\venv\lib\site-packages\betterproto_init.py", line 710, in _postprocess_single value = Timestamp().parse(value).to_datetime() File "\venv\lib\site-packages\betterproto_init.py", line 973, in to_datetime return datetime.fromtimestamp(ts, tz=timezone.utc) OSError: [Errno 22] Invalid argument
After research we finally discovered that Python is unable to work with negative timestamps witthout timedelta (e.g. like here)
so actually the problem is over here
My team would appreciate a lot for fixing this issue!
What version of python are you on (I'm on 3.10.6), this seems to work fine for me if I turn the value into seconds
In [5]: datetime.datetime.fromtimestamp(-3739996800000 / 1000) # the thing the SO OP originally tried
Out[5]: datetime.datetime(1851, 6, 27, 0, 0)
In [6]: datetime.datetime.fromtimestamp(-62135596800.0)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Input In [6], in <cell line: 1>()
----> 1 datetime.datetime.fromtimestamp(-62135596800.0)
ValueError: year 0 is out of range
In [7]: datetime.datetime.fromtimestamp(-62135596800.0 / 1000)
Out[7]: datetime.datetime(1968, 1, 12, 20, 6, 43, 200000)
What version of python are you on (I'm on 3.10.6), this seems to work fine for me if I turn the value into seconds
In [5]: datetime.datetime.fromtimestamp(-3739996800000 / 1000) # the thing the SO OP originally tried Out[5]: datetime.datetime(1851, 6, 27, 0, 0) In [6]: datetime.datetime.fromtimestamp(-62135596800.0) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) Input In [6], in <cell line: 1>() ----> 1 datetime.datetime.fromtimestamp(-62135596800.0) ValueError: year 0 is out of range In [7]: datetime.datetime.fromtimestamp(-62135596800.0 / 1000) Out[7]: datetime.datetime(1968, 1, 12, 20, 6, 43, 200000)
The problem is that we cannot change code of the library to convert from timestamp like this The source is:
def to_datetime(self) -> datetime:
ts = self.seconds + (self.nanos / 1e9)
return datetime.fromtimestamp(ts, tz=timezone.utc)
So we cannot control the parsing process and ask the library to convert it like you mentioned
I think the issue here is that you are mistaking milliseconds for seconds in the datetime constructor or something everything here seems to work as I'd expect if I convert to seconds.
In [10]: @dataclass(repr=False)
...: class Foo(betterproto.Message):
...: bar: datetime.datetime = betterproto.message_field(1)
...:
In [11]: Foo(datetime.datetime.fromtimestamp(-62135596800.0 / 1000))
Out[11]: Foo(bar=datetime.datetime(1968, 1, 12, 20, 6, 43, 200000))
In [12]: Foo().parse(bytes(_))
Out[12]: Foo(bar=datetime.datetime(1968, 1, 12, 20, 6, 44, 200000, tzinfo=datetime.timezone.utc))
We have generated models from proto files
message ClassName {
GrpcInstrumentId Id = 1;
google.protobuf.Timestamp StartDate = 2;
google.protobuf.Timestamp EndDate = 3;
GrpcTimeInterval Period = 4;
}
In python:
@dataclass
class ClassName(betterproto.Message):
id: "GrpcInstrumentId" = betterproto.message_field(1)
start_date: datetime = betterproto.message_field(2)
end_date: datetime = betterproto.message_field(3)
period: "GrpcTimeInterval" = betterproto.enum_field(4)
And the coming responce from server is not converted correctly (as I have shown before) Generated code by betterproto works fine until the negative timestamp appearing
This isn't an issue with betterproto it's an issue with the server, there isn't anything we can do about this, I'm sorry.
It's not the server issue Betterproto does not work correctly on negative timestamp, while vanilla protobuf proceeds it correctly @Gobot1234
Python does not handle fromtimestamp calls with milliseconds, they handle it with seconds, please can you give the input bytes to reproduce this with because this to my knowledge is in line with what google does
In [23]: from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
In [24]: ts = google_dot_protobuf_dot_timestamp__pb2.Timestamp
In [25]: ts.FromSeconds(-62135596800)
In [26]: ts.ToDatetime()
Out[26]: datetime.datetime(1, 1, 1, 0, 0)
Maybe I can give a bit of clarification for this issue. We have a backend written in C#, which uses gRPC and a client written in python. In C#, there is a DateTime.MinValue property which will be equal to 1/1/0001 12:00:00 AM after converting to UTC. Then, we can convert this date to the C# Timestamp object from Google.Protobuf.WellKnownTypes, the result will be a Timestamp object with Seconds=-62135596800 and Nanos=0. So this is a valid Timestamp object which can be sent through gRPC protocol.
When the betterproto client receives this timestamp, it tries to convert this timestamp to python datetime object and fails. The major difference between gRPC default python package and betterproto is that betterproto tries to convert the values inside (which make the life of a developer much easier), but if it fails, then we can't read the message from server, while gRPC default python package doesn't perform such parsing, thus we can receive a message from server and process it. The given timestamp is a valid timestamp, pointing to the 1/1/0001 12:00:00 AM, so it should be parsed.
I am not that familiar with python, nevertheless consider following examples:
datetime.datetime(1970, 1, 1) + datetime.timedelta(seconds=(-62135596800)) - will result in correct date in python 3.10.6.
datetime.datetime.fromtimestamp(-62135596800, tz=datetime.timezone.utc) - will result in exception, while in C#, the Timestamp object can be converted to DateTime back.
This might just be a bug in Python then, I'm happy to ask on the python issue tracker if you aren't.
In [8]: datetime.datetime(1, 1, 1, 0, 0).timestamp()
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Input In [8], in <cell line: 1>()
----> 1 datetime.datetime(1, 1, 1, 0, 0).timestamp()
ValueError: year 0 is out of range
I'm very confident this is a bug
I've read documentation about fromtimestamp method and it says:
fromtimestamp() may raise OverflowError, if the timestamp is out of the range of values supported by the platform C localtime() or gmtime() functions, and OSError on localtime() or gmtime() failure. It’s common for this to be restricted to years in 1970 through 2038.
In our case it is OSError, so it is kind of documented thing. The documentation also states that:
To get an aware datetime object, call fromtimestamp():
datetime.fromtimestamp(timestamp, timezone.utc) On the POSIX compliant platforms, it is equivalent to the following expression:
datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(seconds=timestamp) except the latter formula always supports the full years range: between MINYEAR and MAXYEAR inclusive.
So maybe it possible to modify parsing algorithm to use this expression datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(seconds=timestamp) and if the tests will be fine mb it can be released.