csv-to-influxdb icon indicating copy to clipboard operation
csv-to-influxdb copied to clipboard

Published 20 hours ago • verified publisher icon jpillora

parse timestamp column as integer timestamp value like influxdb would

Open N-Coder opened this issue 8 years ago • 5 comments

If the time format is any of (h, m, s, ms, u, ns) the timestamp column will be parsed as integer with the given precission (instead of a formatted string), mimicking the behaviour of the influxdb line write API using the given precission. (please don't hurt me if there is anything wrong with my code, these are the first lines of Go that I ever wrote)

N-Coder avatar Nov 12 '16 00:11 N-Coder

Oh, hello.

It seems we both have implemented a similar feature (see https://github.com/jpillora/csv-to-influxdb/pull/11).

I really wish for this feature to be merged in the trunk (whatever the implementation is).

I just have the following question concerning the behavior of your version. Does your version allows parsing any timestamp, whatever its precision is ?

The passed precision would then, I assume, be only be used for overriding it (going into a lower precision as InfluxDB line protocol only accepts nanosecond precision).

p3r7 avatar Nov 28 '16 17:11 p3r7

Oh, yeah, it seems we had pretty much the same idea. :sweat_smile:

I added the legal InfluxDB epoch specifiers (n, u ms, s, m ,h) as further possible values for TimestampFormat: https://github.com/jpillora/csv-to-influxdb/pull/12/files#diff-7ddfb3e035b42cd70649cc33393fe32cR238 If you pass in one of those instead of a go date format, the timestamps from the csv file will be parsed as int and scaled to the right unit (https://github.com/jpillora/csv-to-influxdb/pull/12/files#diff-7ddfb3e035b42cd70649cc33393fe32cR257). I don't think that auto-detecting the unit of a timestamp would work reliably, as that would break for values close to 1970.

N-Coder avatar Nov 29 '16 18:11 N-Coder

Hey guys, sorry I've been swamped. Usually PRs are only 5 line changes, so I can generally merge via mobile and I haven't had time to properly review and test. I might also refactor this tool into multiple files soon since it's slowly outgrowing main.go.

RE: time magnitude: I think having it as a separate option might be wise and default to s since that's most common (I think?). Agreed auto-detect might be a bit too magical.

I'll review both PRs and most likely copy-paste sections into the newly refactored version.

jpillora avatar Nov 29 '16 22:11 jpillora

Sounds great to me.

Indeed, I tried implementing auto-detect but didn't thought of timestamps close to unix epoch :expressionless:

I'm also ok with a separate option for precision :thumbsup:

p3r7 avatar Nov 30 '16 13:11 p3r7

Hello,

Will you have time to progress on this?

Could we give a hand if needed?

p3r7 avatar Jan 02 '17 17:01 p3r7