Default timestamp in CSV files should be Unix Time

[zhenyun]

currently after naarad processing, the timestamp is epoch time (e.g.,, 132455555), we should use unix time (e.g., 2014-03-02 22:11:11).

If we really want epoch time in some cases, we can use config file to specify.

[richardhsu]

This sounds reasonable to me so that CSV files are more readable. This will add some overhead on write as all timestamps need to be translated.

This will affect both the generation of CSV files and also must look into the JavaScript code that touches the download of the CSV file as that I think converts timestamps as well.

[feng-tao]

In naarad, every timestamp will translate into epoch time in naarad.utils.get_standardized_timestamp. We could change this function to use unix time instead of epoch time. But the trickiest thing is that plot library(e.g matplotlib) uses the csv file to plot graph and it only recognizes epoch time. It will trigger exception if the csv time series is not epoch time. Any advise for the fix?

[zhenyun]

The goal of the "filed issue" is to store Unix time in *.csv file (for easy human reading afterwards). In naarad, the unix time needs to be translated to epoch time for plotting.

---

From: feng-tao [[email protected]] Sent: Thursday, March 26, 2015 6:08 PM To: linkedin/naarad Cc: Zhenyun Zhuang Subject: Re: [naarad] Default timestamp in CSV files should be Unix Time (#310)

In naarad, every timestamp will translate into epoch time in naarad.utils.get_standardized_timestamp. We could change this function to use unix time instead of epoch time. But the trickiest thing is that plot library(e.g matplotlib) uses the csv file to plot graph and it only recognizes epoch time. It will trigger exception if the csv time series is not epoch time. Any advise for the fix?

— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/naarad/issues/310#issuecomment-86779393.

[richardhsu]

Matplotlib can handle unix timestamps, do you have documentation when it cannot? I believe, I tried to just have the CSV files have the same timestamps as logs and it seemed to graph correctly. I can try to dig that up another time.

I think the biggest issue, the last time I looked at this, was that there are timezone details to deal with as well.

[feng-tao]

Figure out the issue. I just push the initial fix which works on my example sar configure file. So now the csv file will show unix time and graph is generated correctly. I dont think the time zone is an issue as it shows up correctly in my test. But I may be wrong.

[feng-tao]

BTW, the fix combines for both bug #277 and bug #310 . Do you think I need to separate them?

[feng-tao]

Push a second fix(#328) per Richard's comment