Error parsing datetime from CSV files

[sa3eed3ed]

Error parsing datetime while uploading CSV files.

Date time formatted as in 2022-08-09T05:29:39-05:00, the errors is:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/tasks.py", line 735, in run_csv_jsonl
    for event in read_and_validate(file_handle):
  File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/utils.py", line 188, in read_and_validate_csv
    chunk["timestamp"] = chunk["datetime"].dt.strftime("%s%f").astype(int)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 5135, in __getattr__
    return object.__getattribute__(self, name)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/accessor.py", line 187, in __get__
    accessor_obj = self._accessor(obj)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/accessors.py", line 480, in __new__
    raise AttributeError("Can only use .dt accessor with datetimelike values")
AttributeError: Can only use .dt accessor with datetimelike values

did some testing, the pandas.to_datetime will return a timestamp object (pandas._libs.tslibs.timestamps.Timestamp) in this case which has no dt attribute: https://github.com/google/timesketch/blob/9f035cfdae4b2d561f583d3e7213a19dadbb0867/timesketch/lib/utils.py#L292

Possible solution:

• Add some type based branches to parse the object returned from pandas.to_datetime
• Convert non UTC timestamps to UTC

[LX5321]

Hey @sa3eed3ed @jaegeral would like to work on this issue. Do assign it to me if possible.

Thanks!

[jaegeral]

Done, a few ideas how to tackle it:

• [ ] consider creating new unit tests for the method failing this parsing
• [ ] consider create a new e2e test for this input
• [ ] fix the issue in the utils.py class.

[bwhelan212]

I'm working on this