rosbag2
rosbag2 copied to clipboard
ros2
Definition of duration in rosbag
I am trying to understand how the duration of a bag is defined in rosbag and why this definition has been chosen. I could not find anything on that matter in the design docs here or in ros_comm.
As far as I understand, the time range captured by a rosbag is internally defined by the closed interval [starting_time, ending_time], the values being the timestamps of the earliest and latest messages in the bag. Both rosbag2 and rosbag1 use these two timestamps to calculate the bag duration as ending_time - starting_time. Rosbag1 (at least on Python) does this on runtime, rosbag2 stores the duration in metadata.yaml.
With this definition a single message bag has a duration of 0, where I would expect a duration of 1ns. This makes boolean checks on the duration behave unexpectedly. Arguably the duration is always one nanosecond short.
Is this an intentional design decision? Would calculating the duration on a half-open interval be more appropriate?
Arguably the duration is always one nanosecond short.
I disagree that a single-message bag should have a duration of 1ns. Why this specific amount? The duration is defined as "the difference in timestamps between the first and last message". So, for a bag with 0 or 1 messages, this is well-specified as a 0 duration, or "no duration".
This makes boolean checks on the duration behave unexpectedly
Can you elaborate on this? What boolean checks behave unexpectedly? Is this a problem with the expectations, instead?
Arguably the duration is always one nanosecond short.
I disagree that a single-message bag should have a duration of 1ns. Why this specific amount? The duration is defined as "the difference in timestamps between the first and last message". So, for a bag with 0 or 1 messages, this is well-specified as a 0 duration, or "no duration".
I would choose 1ns because it is the resolution of our timestamps.
With the definition of duration above even a bag with 10, 100, ... messages with the same timestamp would have a duration of 0. It is likely that the time continuous events causing the messages are spread out over the discrete 1ns interval.
Take a sensor running sampling_frequency = 1 / (time_resolution + ϵ) as an example. The recorded stream will contain consecutive messages with identical timestamps, albeit their events are almost time resolution apart in continuous time. Putting two of those messages in a rosbag yields a duration of 0.
This makes boolean checks on the duration behave unexpectedly
Can you elaborate on this? What boolean checks behave unexpectedly? Is this a problem with the expectations, instead?
For me it is strange that duration(n events with same timestamp) == duration(no event).
The duration definition is odd regarding slicing of bags, take ten messages with consecutive timestamps as an example:
[42, 43, 44, 45, 46, 47, 48, 49, 50, 51]
9ns sum 9ns
[42, 43, 44, 45, 46] [47, 48, 49, 50, 51]
4ns 4ns sum 8ns
[42] [43] [44] [45] [46] [47] [48] [49] [50] [51]
0 0 0 0 0 0 0 0 0 0 sum 0ns
The duration definition is odd regarding slicing of bags, take ten messages with consecutive timestamps as an example:
I think the example you have given is misleading. Given your proposal:
[42] [50]
1 1 sum 2ns
However, the duration should actually be 8ns. Merging bags with this approach is not meaningful, the only meaningful piece of meta-information is
last_timestamp - first_timestamp
Which gives you 9ns for all cases.
With the definition of duration above even a bag with 10, 100, ... messages with the same timestamp would have a duration of 0
Yes, this is an intentional result of the definition of a duration. There is no time passage, it is an "instantaneous sample" if all timestamps are the same
Take a sensor running sampling_frequency = 1 / (time_resolution + ϵ) as an example.
Given ϵ < resolution/2 - would it not be more accurate to round down? Sub-nanosecond time differences could be 0 or 1, it's an arbitrary choice, and an unavoidable consequence of having discrete time. 0.1ns == 1ns is equally incorrect to 0.9ns == 0ns
For me it is strange that duration(n events with same timestamp) == duration(no event).
Ok, so you feel this - but I don't. It appears that this comes down to a matter of "feeling" and is not relevant to actual implementation, as long as it is well-defined. You can easily differentiate these cases by also taking into consideration the message count, which is readily available.
@emersonknapp You're raising another issue with storing sparse streams in rosbag and I think we should broaden the scope of this issue.
Pushing a sparse stream to the extreme: the absence of a failure message in a specific interval is significant information that so far cannot be stored in rosbag1/2.
Currently when splitting a bag, there is no way to tell whether messages are missing or whether the two bags are actually consecutive.
Both could be achieved by:
- decoupling the start/end time of a bag from the actual first/last message
- making the interval half open:
[start, end[
I think this information should be available from the bag or another file to be written by the recorder, because it's the recorder that knows during which interval it was ready to receive messages.
I agree with Emerson in this case. In the event that there is only a single message, or all messages have the same timestamp, I would expect the duration to be 0, because we have a well specified definition that the bag duration is the difference between the minimum and maximum message timestamps in a file. This is the same as what ROS1 bags did, and while we're not beholden to using the same definitions as ROS1, it's nice when it makes sense.
I think this is a reasonable definition, because if we start returning non-zero durations for these special cases, to me that would imply that the duration is somehow accounting for transmission or writing time, which is certainly not the case.
@chaoflow I'm not sure I understand your argument about failure messages. Are you talking about some kind of robotic system that is publishing some kind of periodic status message, and you want to know if that status message is in a bag file? If that's the case, I don't see why the absence of such a status message is not sufficient information to know that it was not published. I view bag files as strictly a means of recording message traffic on a ROS system, and attaching a particular meaning to the presence or absence of messages should be done as a separate analysis step.
I'll also add, this introduces a weird corner case where I have a bag with two messages that have timestamps 1ns apart. If we make the minimum duration 1ns, then this bag has the same duration as a bag with a single message, which is definitely not what I want.
@danthony06 my proposal is to decouple start and end time of message timestamps and define them as:
- start_time <= first message timestamp
- end_time >= last message timestamp + 1ns
- duration = end_time - start_time
That way the bag records during which time recording was happening independent of whether messages are published or not and when slicing such a bag: end_time1 = start_time2 independent of whether there are messages at these exact times, as is the case with the current definition.
my proposal is to decouple start and end time from message timestamps
This idea sounds pretty good - there is possibly some interesting data on times of "in recording state" vs the message data itself.
when slicing such a bag: end_time1 = start_time2 independent of whether there are messages at these exact times
This is a compelling case to me - I like it. It adds another layer of consistency between bags.
Would it be better to put the start and stop recording time, according to the system clock, in the metadata file? That would capture the idea that the actual bag recording duration is not necessarily reflected by the timestamps of the messages contained in it. Of course, we'd have to then think through a lot of things like what happens when you split a bag or manually insert data into it, because you could introduce weird cases of the duration of the messages in the bag file exceeding the duration the metadata file claims it ran.
@danthony06 I think we need to use the same clock source as rosbag record does for the message timestamp (not header timestamp), otherwise things get out hand, especially when recording messages while playing back and publishing /clock.
A bag that contains a timestamp x | x < start_time ∪ x >= end_time I'd consider broken.
In #819 we're discussing .db3 files to be self-contained. In that case we'd want the start/end time for each .db3 file stored in the respective .db3 file and have metadata.yaml potentially duplicate that information and additionally calculate and store the overall start/end time of the set. The overall time would not be in individual db3 files as these are not aware of being part of a set.