Unable to parse dates/times in CSV with more than millisecond precision

[logos-electromechanical]

Thanks for contributing to PlotJuggler. You are great!

Problem description

I'm trying to plot some csv data with timestamps with the following format: 2023-02-10 21:36:43.000000

The CSV loader then claims to be unable to parse this time stamp with the following format specification: yyyy-MM-dd hh:mm:ss.zzz Experimentally, I tried yyyy-MM-dd hh:mm:ss.zzzzzz with similar results.

The error window does not allow me to copy and paste the error into this issue.

Steps to reproduce (important)

-- System: Windows 10 running the 3.6.0 version (latest available binary) -- Any timestamp of the given format with more than three digits to the right of the decimal point triggers the bug, -- I edited the CSV file by deleting the trailing three digits on the timestamp of the first several lines and confirmed that it moved the point in the file where the error occurred to the first occurrence of a timestamp with six digits to the right of the decimal point.

[zdavkeos]

Hi @logos-electromechanical

It appears Qt is limited to 3 digits worth of milliseconds. Your format parses for me if I use this format string: yyyy-MM-dd hh:mm:ss.zzz000. It matches, but throws out, the three least significant digits (as long as they are all zero).

See here for reference: https://stackoverflow.com/questions/8229658/how-do-i-get-milliseconds-in-a-qdatetime-with-qsqlquery-in-qt-c

[logos-electromechanical]

Turns out if I use timestamp values rather than date/times in the inputs it works... but 'can only have zeros past the milliseconds place' is still a serious bug in code that ingests CSVs, and one that really needs to be fixed.

[Bartimaeus-]

I've run into this limitation before and looked in to it last summer to try and fix it. @zdavkeos is correct that qdatetime only supports time out to milliseconds. My assessment at the time was that it would be a pretty involved change because qdatetime is used pretty extensively within Plotjuggler.

To expand on what @zdavkeosposted, you can provide a format string that uses other digits besides zero provided that those match what's in the file.

ex: I have used yyyy-MM-dd hh:mm:ss.zzz123 for data that had sub-millisecond precision but was sampled at a constant interval above 1ms (so all of the timestamps ended in "123"). This works for allowing Plotjuggler to parse the data, but time resolution below a millisecond is lost.

One potential workaround that could be implemented is support for a wildcard character in the format string that could be used to strip characters from the datetime before converting to qdatetime, which wouldn't get rid of the millisecond limitation of qdatetime but would allow Plotjuggler to import data and discard resolution beyond what can be handled.

As another note, you'll want to check that importing time as timestamp values instead of datetime doesn't lose the sub-millisecond resolution for the same reason. I believe that if you try exporting that data from Plotjuggler to a csv it may not have the original resolution