review query_factory and smartquery

[peekjef72]

About

I've tried to use query methods from grafana-client info grafana-snapshot-tool. For that I need to perform some adjustement to api interface.

• extend query_factory to be able to set more parameters for each "known" datasource according to their parameters. Goal it to use parameters set in the panel mixed with datasource requirements and format to generate a request object (method, uri fragments, data to Post)
• review examples datasource-smartquery.py and datasource-health-check.py to ensure they still work.

Discussion

it seems to me that :

• datasource accesses change with grafana version: Prometheus in v 7 use format specified in function query() and query_range(), but now use a standard format for the queries (type or range) and results.
• the format of the results changes from one datasource to anothers, maybe for grafana vesion too.
• some of your tests (Loki ) seems not working for me : format my have change... not sure.

I'm not sure that the grafana_client must implement a generic function smartquery() that permit to users to query and version to Grafana server and any data source in a unified way. What is your opinion ?

[peekjef72]

I've tried to use query methods from grafana-client info grafana-snapshot-tool. For that I need to perform some adjustement to api interface:

• extend query_factory to be able to set more parameters for each "known" datasource according to their parameters. Goal it to use parameters set in the panel mixed with datasource requirements and format to generate a request object (method, uri fragments, data to Post)
• review examples datasource-smartquery.py and datasource-health-check.py to ensure they still work.

Discussion: it seems to me that :

• datasource accesses change with grafana version: Prometheus in v 7 use format specified in function query() and query_range(), but now use a standard format for the queries (type or range) and results.
• the format of the results changes from one datasource to anothers, maybe for grafana vesion too.
• some of your tests (Loki ) seems not working for me : format my have change... not sure.

I'm not sure that the grafana_client must implement a generic function smartquery() that permit to users to query and version to Grafana server and any data source in a unified way. What is your opinion ?

[amotl]

Dear Jean-Francois,

thanks a stack for your excellent contribution and for bringing up the corresponding discussion topics.

I've tried to use query methods from grafana-client in grafana-snapshot-tool. For that I need to perform some adjustments to the API interface.

grafana-client should definitively be able to support grafana-snapshot-tool in the way you are proposing. It is very sweet to see you have been able to build upon my recent additions at grafana_client.elements.datasource, where you certainly improved some important details in order to make it more generic. This is the place where I had to stop on the last iteration, so I am exceptionally happy about it.

I will try to dedicate some time for a detailed review of your patch soon. I think your recent updates, especially removing smartquery_old, makes the overall appearance much more clear now.

With kind regards, Andreas.

[amotl]

Dear Jean-Francois,

having to apologize again for the long delay, I finally found some time to review your patch in detail. I really like it, and would like to wrap it up, to satisfy the linter and the software tests, and also squash the commits, in order to finally integrate it.

~I would start pushing directly to your feature branch~, After rebasing your branch, I just pushed it to GH-112 already. Do you have any objections about this? If you want, and have a few spare minutes, feel free to add your comments there if you think I made a mistake anywhere.

datasource accesses change with grafana version: Prometheus in v 7 use format specified in function query() and query_range(), but now use a standard format for the queries (type or range) and results.

It is sad, right. I was well aware about those details when starting this experiment, that things will be in flux for times to come. Thank you for addressing any anomalies with your patch.

the format of the results changes from one datasource to anothers, maybe for grafana vesion too.

C'est la vie, I would say. Thank you so much for making the best out of it, by introducing corresponding version switching.

some of your tests (Loki ) seems not working for me : format my have change... not sure.

Thanks, I will look into it when starting to work on this.

I'm not sure that the grafana_client must implement a generic function smartquery() that permit to users to query and version to Grafana server and any data source in a unified way. What is your opinion?

I am absolutely not sure about it, but I definitively wanted to give it a try, because I thought it would make a nice feature. I know there will always be anomalies wrt. to a completely unified interface, that is natural. I am open to any discussions about this, and to hear back about what you think, how this topic could be approached better.

Still, I think the outcome is quite nice: After learning a bit of query parameters, how to use smartquery() for your purposes, I think it becomes very much usable for scripting purposes and for automating and monitoring Grafana, and the data sources behind. At least, this is what I was originally humbly aiming at, in order to expand what monitoring-check-grafana can do [^1].

With kind regards, Andreas.

[^1]: Currently, it is only a Shell program, and just supports InfluxDB 1.x, so it was clear to use Python for the next iteration.

[amotl]

Closing this in favor of GH-112, where I added some adjustments to make the tests work again. Your improvements have been included into the 3.8.0 release, and I will be happy to receive corresponding feedback or bugfix patches, in case I made any mistakes. Thank you again, for both your patch and your patience.