InfluxDB 2.0 Support

[lucaderi]

In InfluxDB 2.0 HTTP requests need to use a token generated inside influxDB itself. This means that ntopng is unable to talk with the database

2020-04-15T12:01:04.082762Z info Unauthorized {"log_id": "0MBRG8IW000", "error": "token required"} 2020-04-15T12:01:04.200689Z info Unauthorized {"log_id": "0MBRG8IW000", "error": "token required"}

Evaluate the use of /metric scraper that can simplify many things

[victorhooi]

Is there any update on when ntopNG will support InfluxDB 2.x?

Thanks!

[simonemainardi]

The discussion is still ongoing internally and cannot provide an ETA.

[victorhooi]

@simonemainardi Was there any update on the internal discussions, around InfluxDB 2.0 support in ntopng?

[simonemainardi]

Hi. This is not currently among the prioritized activities. Cannot provide an ETA.

[laurentduru]

Hello @simonemainardi , as I understand there is no more developpement on the ntop datasource for grafana. You promote the usage of the influxdb datasource. https://www.ntop.org/ntop/ntopng-influxdb-and-grafana-a-step-by-step-guide-to-create-dashboards/

Supporting influxdb2.0 will enhance grafana/ntop intercactions.

One of the need we have is "selective result" to get top n application or IP and not the full list, for example. From our point of view, it might be a good idea to reprioritize this evolution ;)

Best regards,

[simonemainardi]

did you consider using the REST API of ntopng?

[laurentduru]

did you consider using the REST API of ntopng?

@simonemainardi , do you mean using REST API from GRAFANA with the JSON PLUGIN : https://grafana.com/grafana/plugins/simpod-json-datasource/ ?

Not for the moment but it might change ;) do you have examples of implementation ?

[victorhooi]

@simonemainardi Thanks for getting back, and the clear info about the roadmap.

Over time, InfluxDB 1.0 will be going away - it would be awesome if ntopNG can be ported to keep up with the current version of InfluxDB.

Furthermore, in terms of a time series database - it has been great having InfluxDB support in ntopng, and supports a lot of great functionality built on top of the ntopng data. (RRD in ntopng has already gone away, right?)

[simonemainardi]

Furthermore, in terms of a time series database - it has been great having InfluxDB support in ntopng, and supports a lot of great functionality built on top of the ntopng data.

Right. Support will be added, insofar as possible.

RRD in ntopng has already gone away, right?

RRD still there, supported and present.

[tigattack]

For the time being, could you modify or disable the version check in the InfluxDB 1 module?

InfluxDB 2 has a v1 compatability API, but it's not possible to use it with ntopng due to the following error: [prefs.lua:104] [influxdb.lua:1682] ERROR Incompatible InfluxDB version: required "1.5.1", found "nil". Please install an updated version from the official site.

[daymickcorr]

Influx seems to stop wanting collecting metrics directly from the server, graphite an opentsdb were dropped from influxdb 2.0 and are now in telegraf, on the long run it would be better for ntopng to write to one of the input plugins for telegraf.

now the way it works let's say you have netdata which can output in graphite or prometheus prometheus will be sent to /metrics graphite will be sent on a listening port let's say 5000 to telegraf and then sent in an output format usualy influxdb line to influxdb

prometheus has been added as an input to telegraf already.

Since influxdb cloud does not support scrapeable endpoints and influxdb server exports their own data in prometheus i have the impression scrapeable endpoints were only added to interconnect the servers or for microservices.

For the future the most desirable way to support influxdb2.0 would probably be to export in prometheus to telegraf also with telegraf ntopng could export data to other servers like servicenow, splunk, graphite, influxdb cloud.

Also at the moment the ntopng rest api answers in json it should be possible to collect metrics with telegraf using the http telegraf input in json and sending it to influxdb line to influxdb just by configuring the telegraf agent without having to write code https://github.com/influxdata/telegraf/blob/release-1.18/plugins/inputs/http/README.md https://github.com/influxdata/telegraf/tree/release-1.18/plugins/parsers/json https://www.ntop.org/guides/ntopng/api/rest/index.html

if you are using ntopng on pfsense telegraf is already an available package

[Ryananana]

Hi,

It seems that influxdb 2.0 still provides api for writing and querying data and the usage of these api has not changed much from influxdb 1.8, while the 1.x compatibility API is provided in influxdb 2.0.

https://github.com/ntop/ntopng/blob/89c765a325a1c3dedad91f4083accf5628553bed/scripts/lua/modules/timeseries/drivers/influxdb.lua#L738 https://github.com/ntop/ntopng/blob/89c765a325a1c3dedad91f4083accf5628553bed/scripts/lua/modules/timeseries/drivers/influxdb.lua#L916 It looks like these two lines of code are using the /write and /query APIs to communicate with influxdb, where the username&password method of authentication has been replaced by username&token based on V1 compatible endpoints, and most of the other parameters have not changed much.

[jmilleriec]

Just tried this with Influx Cloud and got this: Incompatible InfluxDB version: required "1.5.1", found "2.0". Please install an updated version from the official site.

Would be great to get rid of the version check if that's the thing that is preventing this, at least in the short term until a longer term solution can be found.

[victorhooi]

@lucaderi Is there any word from the development team - either for disabling the version check, or making the changes to be compatible with InfluxDB 2.0?

[jufy111]

I'm in the same vote, my InfluxedDB (2.x) is set up to handle the V1 API, but I get the "Incompatible InfluxDB version: required "1.5.1", found "2.3.0". Please install an updated version from the official site." in the logs.

It appears that it doesn't try unless the major version of influxdb is 1.x.x. I believe I have isolated the function that occurs (see below), but I'm well aware that it may not be as easy as just changing this to allow for 2.x.

https://github.com/ntop/ntopng/blob/8ade112b819217a3f84f5f084b4332a02a837a63/scripts/lua/modules/timeseries/drivers/influxdb.lua#L29 https://github.com/ntop/ntopng/blob/8ade112b819217a3f84f5f084b4332a02a837a63/scripts/lua/modules/timeseries/drivers/influxdb.lua#L1481-L1492

It would be awesome if it is as simple as changing this function.

edit

I've tested today and modified the influxdb.lua script that running on my pfsense box. For those who are interested the files are directory is:
/usr/local/share/ntopng/scripts/lua/modules/timeseries/drivers/influxdb.lua

It appears to now be writing successfully to the bucket that I set up the v1 API for.
I am however getting a bunch of errors in the ntopng log, but it means I can now utilise client traffic in Grafana.

[prop-physics]

Would very much like to see support for 2.0. I am aware that this might still not be a development priority, but I do hope that we could at least leverage the 1.x compatibility layer as a stop-gap measure.

Thanks for posting your findings @jufy111, you've saved me some digging. Will see if I can get it working myself.

[zombielinux]

I'd also like to lodge a vote in favor of influxdb2 native support, or removal of the version check.

[DoubleStrike]

+1 here. A year and a half after this was brought up, this is still open?

[pr0peller]

2.x support would be great +1 here

[victorhooi]

@simonemainardi @lucaderi Is there any update from the internal ntop discussions you mentioned in comment above?

Would ntop be able to use the stopgap measure (removing the compatibility check) suggested above here, whilst full InfluxDB 2.0 support is being worked on?

[djstarfox]

I am testing nTopNG against InfluxDB 2.7.1 but using the v1 compatibility API. Even if full v2 support is not developed, could we at least get v1 API support?

Seems to work but there are errors in the logs: 05/Jun/2023 16:11:53 [get_influxdb_info.lua:22] [influxdb.lua:364] ERROR database not found: _internal 05/Jun/2023 16:11:53 [get_influxdb_info.lua:21] [influxdb.lua:364] ERROR database not found: _internal 05/Jun/2023 16:11:53 [get_influxdb_info.lua:20] [influxdb.lua:364] ERROR database not found: _internal 05/Jun/2023 16:11:15 [get_influxdb_info.lua:22] [influxdb.lua:364] ERROR database not found: _internal 05/Jun/2023 16:11:15 [get_influxdb_info.lua:21] [influxdb.lua:364] ERROR database not found: _internal 05/Jun/2023 16:11:15 [get_influxdb_info.lua:20] [influxdb.lua:364] ERROR database not found: _internal 05/Jun/2023 16:10:02 [export_influxdb_stats.lua:21] [influxdb.lua:364] ERROR database not found: _internal 05/Jun/2023 16:05:02 [export_influxdb_stats.lua:21] [influxdb.lua:364] ERROR database not found: _internal

This may be of help too.
https://docs.influxdata.com/influxdb/v2.7/upgrade/v1-to-v2/automatic-upgrade/

The stats page for InfluxDB hasn't worked in Ntop for several versions, so fixing the driver seems more of priority to fix multiple issues... And the assumptions made in the code (e.g., scripts/lua/modules/timeseries/drivers/influxdb.lua) are now false. Perhaps a ntop config item specifying the bucket where internal metrics are stored (or disable internal metrics if blank)?

function driver:exists(schema, tags_filter, wildcard_tags)
  if schema.options.influx_internal_query then
    -- internal metrics always exist
    return(true)
  end

[djstarfox]

@jufy111 How did you get influx v1 compatibility mode working? There is no support for continuous queries in v2, so the "InfluxDB setup failed" message repeats in the ntop logs.

[jufy111]

@djstarfox My solution didn't provide full support, it just got ntopng writing data to my influxDB instance. I was get likely getting errors in the logs too. I had to enable v1 support in my influxdb manually over the CLI.

All I did was disable the version check shown in my post above.

[jmonline86]

I too would like the ability for ntopng to write directly to InfluxDB v2.0. It's the last remaining application in my stack which isn't v2.0 compatible.

[djstarfox]

The lack of attention on this issue is troubling.

[jonz-secops]

Are there any pulls related to this request that people have submitted? This is something that would be very positive to have addressed.

[djstarfox]

This issue is approaching 4 years old now... The way I see it, the easiest path forward would be to refactor code to work with InfluxDB 2.0 with version 1 compatibility API. Afterwards, it will still work with native v1.

So, what will it take to get this issue addressed sooner?

[MatteoBiscosi]

we decided currently to not support influxdb 2.0 due to huge lack of performances from the influx side and we are not planning on adding the support.

[jufy111]

I can understand the decision not to support influxdb v2, however I do see benefit in altering the version check, so the those who wish to use the v1 API with influxdb v2+ can do so.