mavros icon indicating copy to clipboard operation
mavros copied to clipboard

where does the timestamp on mavros come from?

Open shubham-shahh opened this issue 3 years ago • 25 comments

Hi, Mavros uses time stamps that comes from the pixhawk or it considers the timestamp from the companion computer?

thanks

shubham-shahh avatar Nov 18 '21 06:11 shubham-shahh

When it is possible, sys_time does timesync between FCU and OBC. So mavlink messages with timestamp fields will be relatively close to FCU's time. You can also use ntpd_driver package to feed GPS time from FCU to ntpd/chrony.

vooon avatar Nov 19 '21 09:11 vooon

When it is possible, sys_time does timesync between FCU and OBC. So mavlink messages with timestamp fields will be relatively close to FCU's time. You can also use ntpd_driver package to feed GPS time from FCU to ntpd/chrony.

Hi, Thanks for the response, I want to have a time sync between mavros messages and the OBC system time

I am using Ardupilot and I tired these steps for the time sync

but I am getting the following errors after following those steps

[ INFO] [1637307795.853196064]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307795.906063936]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307795.906411360]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307796.123088320]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307796.229862976]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307796.230528736]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307796.416113216]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307796.416838336]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307796.806025120]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307796.806391872]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307796.932748160]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307797.225997184]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307797.226336640]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307797.314902368]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307797.392812064]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307797.472641792]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307797.473242464]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307797.668529632]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307797.669409984]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307797.856211776]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307797.857378816]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307797.929965856]: PR: got an unsolicited param value idx=1058, not resetting retries count 3 [ INFO] [1637307798.476274176]: PR: parameters list received [ WARN] [1637307804.634408000]: TM : RTT too high for timesync: 2442.02 ms. [ INFO] [1637307805.681991584]: HP: requesting home position [ WARN] [1637307809.105486528]: TM : RTT too high for timesync: 3612.68 ms. [ INFO] [1637307810.248981024]: FCU: Frame: QUAD [ INFO] [1637307810.444359552]: FCU: ArduCopter V4.0.5 (3f6b43e3) [ WARN] [1637307810.689017088]: CMD: Command 410 -- wait ack timeout [ INFO] [1637307811.440273888]: RC_CHANNELS message detected! [ WARN] [1637307811.925364128]: GP: No GPS fix [ WARN] [1637307812.166801600]: TM : RTT too high for timesync: 3374.38 ms. [ WARN] [1637307813.776849600]: TM : RTT too high for timesync: 3083.81 ms. [ INFO] [1637307814.237447936]: IMU: Raw IMU message used.

@vooon

shubham-shahh avatar Nov 19 '21 09:11 shubham-shahh

I do not know why, but sys_time reports that RTT is too high to operate correctly.

vooon avatar Nov 19 '21 09:11 vooon

I do not know why, but sys_time reports that RTT is too high to operate correctly.

okay, but according to you, are these steps correct?

and since I am using QGCS with other telem port along with mavros, could that be the reason for RTT too high?

and do I have to put sys_time plugin in apm_pluginlists.yaml's whitelist section?

shubham-shahh avatar Nov 19 '21 09:11 shubham-shahh

I'm not sure about setting RTC source (better to check docs), but overall instruction looks correct. RTT mostly depends on link speed and latency. Maybe id you are overloading cpu it might respond too slow (but it's unlikely, because in that case it'll be unflyable).

sys_* must be always in allow list.

vooon avatar Nov 19 '21 13:11 vooon

When it is possible, sys_time does timesync between FCU and OBC. So mavlink messages with timestamp fields will be relatively close to FCU's time. You can also use ntpd_driver package to feed GPS time from FCU to ntpd/chrony.

Hi @vooon can you please elaborate more upon the ntpd_driver approach you mentioned

My GPS is attached to the FCU

Thanks

shubham-shahh avatar Nov 03 '23 00:11 shubham-shahh

https://github.com/vooon/ntpd_driver

It simply forwards GPS time to NTP daemon, which do the rest of keeping time in sync.

vooon avatar Nov 03 '23 08:11 vooon

https://github.com/vooon/ntpd_driver

It simply forwards GPS time to NTP daemon, which do the rest of keeping time in sync.

Hi, I tried this package, and it's not working for the NTP for some reason, but it's working with chrony. I have a few questions,

  1. what if this node gets killed mid flight, in that case what happens to system time
  2. in your experience how badly the GPS reported time is impacted when the number of sats go low
  3. are there any catches one should take care of, while using this approach?

Thanks

shubham-shahh avatar Nov 03 '23 08:11 shubham-shahh

  1. It will get going with the same last adjusted pace (ntp adjusts frequency)
  2. I'm not an expert on clock sync, but since each sat have atomic clock onboard, and a whole constellation is synced, i think it's unnoticeable
  3. In your code you should be careful on which time source do you use for what reason. Its not specific to ros, just usual real vs monotonic timers.

Have you added lines from the readme to ntpd.conf? If my memory serves, ntpd opens furst two shm for root only, next two perhaps needs some permission adjustments. Also things like AppArmor or Selinux may block you.

vooon avatar Nov 03 '23 11:11 vooon

  1. It will get going with the same last adjusted pace (ntp adjusts frequency)
  2. I'm not an expert on clock sync, but since each sat have atomic clock onboard, and a whole constellation is synced, i think it's unnoticeable
  3. In your code you should be careful on which time source do you use for what reason. Its not specific to ros, just usual real vs monotonic timers.

Have you added lines from the readme to ntpd.conf? If my memory serves, ntpd opens furst two shm for root only, next two perhaps needs some permission adjustments. Also things like AppArmor or Selinux may block you.

Hi, thanks for the response, could you please elaborate more upon the first point? I am curious if the node is killed mid flight, will the time start drifting?

I'll read more about the 3rd point you mentioned. I am willing to ensure all the edge computers are synced with GPS such that, when they share time stamped data among each other, they are on the same page. If there are better approaches to do this, please share your opinions

I have added the lines mentioned in readme and I'm the root user so I doubt if it's the permissions and I don't have those apps installed on the machines

Thanks again for all the help

shubham-shahh avatar Nov 03 '23 15:11 shubham-shahh

That's always all about the requirements on how accurate time should be. Clock speed depends on many factors, e.g. temperature change. But actually with NTP it'll drift more. With "no sync" ntpd/chronyd just wont be doing anything.

Usually for networked systems it's just fine to have NTP on all peers. But again, that's up to the requirements.

vooon avatar Nov 03 '23 18:11 vooon

That's always all about the requirements on how accurate time should be. Clock speed depends on many factors, e.g. temperature change. But actually with NTP it'll drift more. With "no sync" ntpd/chronyd just wont be doing anything.

Usually for networked systems it's just fine to have NTP on all peers. But again, that's up to the requirements.

Hi, If I am willing to use the chrony approach we discussed above, and since I have n agents in my system among which I am willing to establish a time sync, what are your thoughts on NTP peers? If I set each agent as peer of another?

thanks

shubham-shahh avatar Nov 27 '23 06:11 shubham-shahh

I'm not sure if it is feasible to use all 16 sources to sync your computer time (while i think is possible). But maybe you can use few FCUs to source accurate time.

Note that you don't sync to FCU's time, you sync to GNSS time, which should be about the same for all drones. Timesync of each UAS to FCU is a separate thing.

vooon avatar Nov 28 '23 11:11 vooon

I'm not sure if it is feasible to use all 16 sources to sync your computer time (while i think is possible). But maybe you can use few FCUs to source accurate time.

Note that you don't sync to FCU's time, you sync to GNSS time, which should be about the same for all drones. Timesync of each UAS to FCU is a separate thing.

Yes, I agree it is a separate thing, If one agent loses GPS but it is connected to other agents and is mentioned as a NTP peer then it should be able to get time from the peers, that's what I was wondering.

one more question was, does monotonic clock work in case of NTP, I had this thought, If I sync my clock once and then the on board crystal can take over and since a UAV won't fly for more than an hour it should not drift much right?

shubham-shahh avatar Nov 28 '23 12:11 shubham-shahh

Well, that's not something i can be sure, so i've googled some:

  • https://stackoverflow.com/questions/74621192/how-does-clock-monotonic-handle-ntp-changes-to-the-system-clock
  • https://inelpandzic.com/articles/clock-synchronization-and-monotonic-clocks/
  • https://ebrary.net/64822/computer_science/monotonic_clocks

So looks like frequency adjustments may affect monotonic time, but delta cannot be negative. Anyway are you sure, that this timesync really matters? I think it's just a few milliseconds.

vooon avatar Nov 28 '23 15:11 vooon

Well, that's not something i can be sure, so i've googled some:

  • https://stackoverflow.com/questions/74621192/how-does-clock-monotonic-handle-ntp-changes-to-the-system-clock
  • https://inelpandzic.com/articles/clock-synchronization-and-monotonic-clocks/
  • https://ebrary.net/64822/computer_science/monotonic_clocks

So looks like frequency adjustments may affect monotonic time, but delta cannot be negative. Anyway are you sure, that this timesync really matters? I think it's just a few milliseconds.

Hi, Thanks for the resources, I'll go through them. when agents share timestamped data among each other such as trajectory data in case of a swarm, I'd want to ensure the times are as close as possible, I was hoping to add NTP peers only to ensure that, incase one of the agents loses clock from GPS it can sync it with its peer agents. Also I tried a approach where I time sync all the agents once and then disconnect them from the internet, in that case there's a huge drift

shubham-shahh avatar Nov 29 '23 07:11 shubham-shahh

Hi, @vooon are there any ways to do the OBC time sync with GPS without ROS? since I am willing to have updated time before we start ROS

shubham-shahh avatar Feb 29 '24 07:02 shubham-shahh

@shubham-shahh only if you wrote something like ntpd_driver, but which will directly parse mavlink. Then you should add a mavlink proxy (router) to separate FCU stream for several users (e.g. mavros and your time reader).

vooon avatar Feb 29 '24 08:02 vooon

@shubham-shahh only if you wrote something like ntpd_driver, but which will directly parse mavlink. Then you should add a mavlink proxy (router) to separate FCU stream for several users (e.g. mavros and your time reader).

Thanks for the response, I was thinking to keep the ntpd_driver code as is and in place of the message callback, I write a separate function that listens directly to the mavlink message and rest of the things can remain the same, please correct me if there are any gaps

Thanks

shubham-shahh avatar Feb 29 '24 08:02 shubham-shahh

@shubham-shahh that would work only for ROS2. On ROS1 you had to have running ros core before start any of the nodes. Since ROS2 is masterless, DDS can reconnect the node after the start. But take into account that this just my guesses based on theory.

Also i don't really see any value to mix same message processes trough mavros and direct mavlink decoding. NTPd shared memory interface is pretty simple. MAVConn library potentially can be used outside of ROS.

vooon avatar Feb 29 '24 12:02 vooon

@shubham-shahh that would work only for ROS2. On ROS1 you had to have running ros core before start any of the nodes. Since ROS2 is masterless, DDS can reconnect the node after the start. But take into account that this just my guesses based on theory.

Also i don't really see any value to mix same message processes trough mavros and direct mavlink decoding. NTPd shared memory interface is pretty simple. MAVConn library potentially can be used outside of ROS.

yes, I was thinking something similar, to use mavproxy to create separate FCU streams, and Cpp mavlink library to listen to the mavlink message decode it and use the ntpd_driver's (all ros code stripped) shm logic to set the time

thanks

shubham-shahh avatar Feb 29 '24 12:02 shubham-shahh

@shubham-shahh well, that's part not even (entirely) mine, tribute to gpsd.

vooon avatar Feb 29 '24 12:02 vooon

@shubham-shahh well, that's part not even (entirely) mine, tribute to gpsd.

I am thankful to mavros and gpsd for the novel open source work, what are your thoughts? would the approach mentioned above work?

thanks

shubham-shahh avatar Feb 29 '24 12:02 shubham-shahh

@shubham-shahh yes, i think it should work. Just split the stream using mavlink-router, then use mavconn to receive that stream and process SYSTEM_TIME.

Your approach actually should be faster and with better jitter as you eliminating network hop.

vooon avatar Feb 29 '24 13:02 vooon

@shubham-shahh yes, i think it should work. Just split the stream using mavlink-router, then use mavconn to receive that stream and process SYSTEM_TIME.

Your approach actually should be faster and with better jitter as you eliminating network hop.

Hi, Thank you so much for the info, I'll keep updating my progress here

shubham-shahh avatar Mar 04 '24 06:03 shubham-shahh