Enhancement: Support writing to elasticsearch

[gavmckee80]

@toni-moreno

How practical do you think it would be to write an extension to write to Elastic search. Its one of the attractive things about telegraf , given its support for a output to Elastic.

This project is awesome - I think this would make it even better :D

[sbengo]

Hi @gavmckee80 , we have plans to add more data backends, but is not on our short plans...

In order to send data to ES, you can use Telegraf to convert data to ES format:

SNMPCollector --- [Influx Format] ---> Telegraf --- [ES Format] ---> Elastic

[gavmckee80]

so I do the following

[[inputs.socket_listener]]
  ## URL to listen on
  service_address = "tcp://1.1.1.1:65000"
  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
   data_format = "influx"

Before telegraf writes to ES, the timestamp looks a bit funky:

interfaceStats,host=xxxx,hostname=xxxxx,ifIndex=3,ifName=eth1,vendor=UNKNOWN ifOutMulticastPkts=0,ifHCInOctets=17898222416,ifInErrors=0,ifOutBroadcastPkts=0,ifOutDiscards=0,ifHCInUcastPkts=278328483,ifHCOutMulticastPkts=0,ifILastChange=17696257,ifInUnknownProtos=0,ifAdminStatus=1,ifHCOutBroadcastPkts=0,ifInBroadcastPkts=0,ifInDiscards=0,ifHCOutOctets=815182852,ifHCOutUcastPkts=13586377,ifHighSpeed=1000,ifInMulticastPkts=0,TotalOctets=18713405268,ifDesc="eth1",ifHCInBroadcastPkts=0,ifHCInMulticastPkts=0,ifOutErrors=0,ifOutQLen=0 1511994960
interfaceStats,ifIndex=1,ifName=lo,vendor=UNKNOWN,host=xxxxx,hostname=xxxxx ifHighSpeed=10,ifInMulticastPkts=0,ifOutBroadcastPkts=0,ifOutDiscards=0,ifOutErrors=0,ifAdminStatus=1,ifILastChange=0,ifInDiscards=0,ifOutQLen=0,ifOutMulticastPkts=0,ifHCInBroadcastPkts=0,ifHCInUcastPkts=91770,ifHCOutOctets=7729548,ifHCOutUcastPkts=91770,ifInUnknownProtos=0,ifHCOutMulticastPkts=0,ifInBroadcastPkts=0,ifInErrors=0,TotalOctets=15459096,ifDesc="lo",ifHCInMulticastPkts=0,ifHCInOctets=7729548,ifHCOutBroadcastPkts=0 1511994960
interfaceStats,ifIndex=2,ifName=eth0,vendor=UNKNOWN,host=xxxxxx,hostname=aur-cme-pnt-con02 ifOutDiscards=0,ifDesc="eth0",ifHCOutMulticastPkts=0,ifILastChange=17696095,ifInBroadcastPkts=0,ifInUnknownProtos=0,ifOutQLen=0,TotalOctets=19487501722,ifHCInMulticastPkts=0,ifHCInOctets=18139844571,ifHCOutUcastPkts=15936623,ifOutErrors=0,ifHCInUcastPkts=280085453,ifHCOutBroadcastPkts=0,ifHCOutOctets=1347657151,ifHighSpeed=1000,ifOutMulticastPkts=0,ifOutBroadcastPkts=0,ifAdminStatus=1,ifHCInBroadcastPkts=0,ifInDiscards=0,ifInErrors=0,ifInMulticastPkts=0 1511994960

1511994960 = Wednesday, November 29, 2017 4:36:00 PM GMT-06:00

@sbengo @toni-moreno - can you provide any insights on the question ? Is this timestamp coming from snmpcollector or from telegraf ?

[gavmckee80]

@toni-moreno A little more information here: I changed the timestamp precision within the Influx DB configuration to nanosecond - this updates Elastic with the proper date/time. But every single timestamp received from snmpcollector is the same

record 1

interfaceStats,hostname=YYYYYY,ifIndex=436359168,ifName=Ethernet1/38,host=XXXXX ifHighSpeed=10000,ifInErrors=0,ifOutMulticastPkts=0,TotalOctets=0,dot3StatsFCSErrors=0,dot3StatsInternalMacReceiveErrors=0,ifHCInUcastPkts=0,dot3StatsAlignmentErrors=0,ifOutBroadcastPkts=0,ifInDiscards=0,dot3StatsDuplexStatus=3,dot3StatsInternalMacTransmitErrors=0,ifHCInBroadcastPkts=0,ifHCOutBroadcastPkts=0,ifHCOutOctets=0,ifHCOutUcastPkts=0,dot3StatsFrameTooLongs=0,ifHCOutMulticastPkts=0,ifOutErrors=0,ifAdminStatus=1,ifInUnknownProtos=0,ifInMulticastPkts=0,ifLinkUpDownTrapEnable=1,ifDesc="Ethernet1/38",ifHCInMulticastPkts=0,ifHCInOctets=0,ifILastChange=0,ifInBroadcastPkts=0,ifOutDiscards=0,ifOutQLen=0 1513777440000864245

record 2

interfaceStats,hostname=YYYYYY,ifIndex=436359168,ifName=Ethernet1/38,host=XXXXXX dot3StatsInternalMacTransmitErrors=0,ifAdminStatus=1,ifHCInBroadcastPkts=0,ifHCInOctets=0,dot3StatsDuplexStatus=3,ifInDiscards=0,ifOutErrors=0,ifHCOutUcastPkts=0,ifHCOutBroadcastPkts=0,ifInBroadcastPkts=0,ifInErrors=0,ifInMulticastPkts=0,ifLinkUpDownTrapEnable=1,ifOutQLen=0,dot3StatsFCSErrors=0,dot3StatsInternalMacReceiveErrors=0,ifHCOutMulticastPkts=0,ifHighSpeed=10000,TotalOctets=0,ifHCInMulticastPkts=0,ifDesc="Ethernet1/38",ifHCInUcastPkts=0,ifOutBroadcastPkts=0,ifOutMulticastPkts=0,dot3StatsAlignmentErrors=0,ifHCOutOctets=0,ifOutDiscards=0,dot3StatsFrameTooLongs=0,ifInUnknownProtos=0,ifILastChange=0 1513777440000864245