grok_exporter
grok_exporter copied to clipboard
add pushgateway related operations
Add pushgateway related operations. User can choose to push metrics to the pushgateway endpoint provided by config file as well as to delete metrics from it.
Added attributes of config.yml
Global section:
pushgateway_addr
: pushgateway address, only support
Metric section:
pushgateway
: specify if this metric needs to be pushed to pushgateway, default is false
.
job_name
: specify job name when pushing metric to pushgateway, default is grok_exporter
.
delete_match
: similar to match attribute, logs which match this pattern will trigger a delete() operation from pushgateway.
grouping_key
: specify the grouping key when pushing metric to pushgateway, default is nil
.
All of the added attributes are not required, thus the original config.yml still works.
@fstab The CI test for file tailer has failed because of timeout, but I did it locally and there was no such failure. Also I think this has no effect for the pushgateway functions. Please take a look at this PR, thanks in advance.
Thanks for the PR! I am currently on holidays with very limited access to the Internet, but I will get back to it next week.
@fstab Sorry for the late reply. I appreciate your response.
I reviewed the changes and have a couple of questions to understand the code better:
- Could you provide a few example log lines and a config file where all pushgateway-related options are used? Having a real live example would be helpful to me.
- What would you say is the intended usage scenario? I can imagine the pushgateway is useful if
grok_exporter
runs behind a NAT firewall. Is this the intended use? - As the pushgateway support is optional, I would like to move the code out of the core implementation into its own package. It would be easy to move the new functions in
grok_exporter.go
to a new package, but I am not sure what to do with the extensions tometrics.go
. Maybe a few of the new fields inmetrics.go
are not really needed:- The
pushgateway
flag was introduced to push only specific metrics to the pushgateway. I don't see a usage scenario where it makes sense to collect metrics ingrok_exporter
, but not push them to the pushgateway. What do you think of removing thepushgateway
flag and pushing all metrics to the gateway as soon as the pushgateway is configured? - The
metricsVec
was introduced to callm.MetricVec().DeleteLabelValues(labelValues...)
inpushMetrics()
ingrok_exporter.go
. What do you think of skipping this and keeping the labels? If the metric is a Summary or Histogram, it might be useful to collect more than one value with the same labels. - What is the difference between the grouping key and other labels? Can we use the existing labels for grouping instead of the grouping key?
- I guess the
delete_regex
would be useful to delete metrics if some kind of shutdown message is found in a log file indicating that a service has stopped. This is an interesting idea, but maybe it is unrelated to the pushgateway and would be useful to the regulargrok_ exporter
as well. Is there any reason that metrics are deleted from the pushgateway but not from the regulargrok_exporeter
?
- The
Thanks a lot! Fabian
@fstab Thanks for the response.
-
Here is the example log file and config file. log file: test.log config file: config.yml
-
The usage scenario for us is to monitor task waiting time cross node in large scale cluster. Sometimes a task can be created in one node and arranged to be executed in another node. The time interval between its creation time and start-execution time is an important metric to monitor. We can use
time() - creation time
in Prometheus to get the actual waiting time. Since we have to monitor every task and use a some kind "ID" label to specify it, and this will overload the storage of Prometheus in a long time. so we have to delete the metric as soon as the task is no longer in a "waiting" state. Pushgateway provides delete() API to do so. -
For the new fields:
-
pushgateway
: The push() and delete() operations provided bypushgateway
are somehow HTTP requests. It may cause a huge overhead if we push all of the metrics defined ingrok_exporter
topushgateway
. In fact, we just want to focus some special metrics and this flag can provide an option to users to determine whether to push the metric or not. -
m.MetricVec().DeleteLabelValues(labelValues...)
is actually not deleting labels but deleting metrics with the given label-value sets. This is actually the main found for me inmetrics_test.go
. If we can implement somedelete()
function ingrok_exporter
, thepushgateway
is no longer needed I think. -
grouping key is used when pushing metric to pushgateway. By default, only
job_name
is required when doing push() operation. In packageclient_golang
provided by Prometheus, grouping key can not be a subset of metric labels and it can be some user-defined values for separating the pushed metrics into different groups. I re-implement the push() API, and put the code indoRequest()
ingrok_exporter.go
. Here grouping key is also a subset of labels. The value or template it used is determined when regularmatch
attribute has been matched. -
delete_regex
is doing exactly what you mentioned. For deleting metrics inpushgateway
but not ingrok_exporter
, as I mentioned before, if we can implement some kinddelete()
function ingrok_exporter
, thepushgateway
is no longer needed.
-
I am not so good at Go
and I apologize for making the code difficult to understand. Hopefully the comment above can explain the code clearly.
Thanks.
Thanks a lot for your response. I understand your usage scenario much better now.
I really like the idea to be able to delete metrics. This should be a general feature in grok_exporter
, not only for the pushgateway. As a general feature, it might be more straightforward to rename the grouping_key
to something like delete_labels
, because the term "grouping" does not occur outside of the pushgatwasy world:
delete_match: 'DELETE %{USERNAME:user}'
delete_labels:
user: '{{.user}}'
As you said, there is no simple delete()
function in the client_golang
for deleting all metrics matching the delete_labels
. However, it should be straightforward to implement this. There should be a relatively small amount of different labels. In the worst case, we could simply keep a map of all label values and call DeleteLabelValues()
individually for each value.
I think the best way to proceed is to implement the generic delete functionality first, and then migrate your pushgateway code to the new version of grok_exporter
. I will try to provide a delete implementation within the next few days.
Thanks four your contribution, it is always good to see different usage scenarios and get some fresh ideas. I'll get back to this issue when the delete functionality is done.
Glad to know this idea is accepted by the author @fstab . I really appreciate your response. Thanks a lot!.
I pushed a delete_match
and delete_labels
implementation. The new function metric.ProcessDelete(line)
returns a map with the matching delete_labels
and their values. I hope the return value of metric.ProcessDelete(line)
is useful for calling delete on the pushgateway
. Let me know if this makes sense for you.
@fstab Thanks, I'll check it out and give you a response later.
@fstab Sorry for late response, I was engaged in some other projects in our company in the last a few weeks, I will try to use this new functions in my pushgateway branch and give you a detailed reply in the next weeks. Thanks.