support mapping logs to programs to avoid unnecessary work

[jaqx0r]

trafficstars

https://groups.google.com/d/topic/mtail-users/hdXIyiluDxc/discussion talks about mapping logs to programs and points out that running the apache program over the nginx logs is a waste of CPU. It really is.

This bug acknowledges the feature request, and hopefully comes up with a solution that avoids the issues I'm worried about:

• increasing maintenance cost of this mapping as the number of programs increases
• ensuring distribution and OS independence across log filename patterns

[mattbostock]

@jaqx0r In the mailing list thread, you said:

One reason I haven't tried to implement this is because I want to try to offer a standard library of mtail programs for common applications, to encourage reuse, and I haven't thought of how to do that in a platform independent way -- every distribution and OS keeps their logs in different filename paths ;-)

I think that creating a library of mtail programs is a great goal to have. I actually think that including the path to the filename in the mtail program could help, rather than hinder that goal.

For example, the example mtail programs could include the file paths for all of the common Linux/BSD distributions, commented out. This would make it easier for someone to re-use the example programs, since they wouldn't have to hunt for the path for their distribution - just confirm that the path is correct for their distro/release and remove the paths for the other distributions.

Does that make sense? Happy to give an example if I haven't explained the idea well.

[jaqx0r]

I think I understand but if you show me what you think it should look like that would be helpful, thanks!

On Mon, 18 Jul 2016, 02:57 Matt Bostock [email protected] wrote:

@jaqx0r https://github.com/jaqx0r In the mailing list thread, you said:

One reason I haven't tried to implement this is because I want to try to offer a standard library of mtail programs for common applications, to encourage reuse, and I haven't thought of how to do that in a platform independent way -- every distribution and OS keeps their logs in different filename paths ;-)

I think that creating a library of mtail programs is a great goal to have. I actually think that including the path to the filename in the mtail program could help, rather than hinder that goal.

For example, the example mtail programs could include the file paths for all of the common Linux/BSD distributions, commented out. This would make it easier for someone to re-use the example programs, since they wouldn't have to hunt for the path for their distribution - just confirm that the path is correct for their distro/release and remove the paths for the other distributions.

Does that make sense? Happy to give an example if I haven't explained the idea well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mtail/issues/35#issuecomment-233191734, or mute the thread https://github.com/notifications/unsubscribe-auth/AC5b-1bSGxxu5pIBMqs1CPnUZIe78RFiks5qWl7-gaJpZM4JN6Al .

[jaqx0r]

That suggestion also makes me think that multiple uncommented rules should allow the same program to be shared, unmodified, between OSes.

On Mon, 18 Jul 2016, 09:56 Jamie Wilkinson [email protected] wrote:

I think I understand but if you show me what you think it should look like that would be helpful, thanks!

On Mon, 18 Jul 2016, 02:57 Matt Bostock [email protected] wrote:

@jaqx0r https://github.com/jaqx0r In the mailing list thread, you said:

One reason I haven't tried to implement this is because I want to try to offer a standard library of mtail programs for common applications, to encourage reuse, and I haven't thought of how to do that in a platform independent way -- every distribution and OS keeps their logs in different filename paths ;-)

I think that creating a library of mtail programs is a great goal to have. I actually think that including the path to the filename in the mtail program could help, rather than hinder that goal.

For example, the example mtail programs could include the file paths for all of the common Linux/BSD distributions, commented out. This would make it easier for someone to re-use the example programs, since they wouldn't have to hunt for the path for their distribution - just confirm that the path is correct for their distro/release and remove the paths for the other distributions.

Does that make sense? Happy to give an example if I haven't explained the idea well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mtail/issues/35#issuecomment-233191734, or mute the thread https://github.com/notifications/unsubscribe-auth/AC5b-1bSGxxu5pIBMqs1CPnUZIe78RFiks5qWl7-gaJpZM4JN6Al .

[NightTsarina]

Quite a coincidence. I am working right now on packaging mtail for Debian, and while testing it out, I realised I wanted this feature.

I think what @mattbostock suggests is great for me, as I would just patch the sample programs to point to standard Debian paths, and users get a ready-to use library of programs adapted to their distribution. Most other distributions will probably think the same.

[yuvipanda]

@TheTincho great to hear you're working on packaging it! Let me know if there's anywhere I can test it to provide feedback :)

[NightTsarina]

@yuvipanda, sorry for the late response, I had missed the notification!

mtail is already in debian testing, and will be present in the next release which is due in a few months: https://packages.debian.org/source/testing/mtail

I will try to package a last snapshot before the release, and if you see any problems there, please let me know!

[jaqx0r]

I can't think of a nice way to do this that doesn't introduce a new configuration file to mtail. I'd rather not use a configuration file because there are already the programs themselves, and adding a configuration file would mean a second language to have to use to get it working.

On the other hand, it would mean not requiring a convoluted commandline, for example: mtail --progs /etc/mtail/programs --log /var/log/apache/error.log=apache.mtail --log /var/log/rsyncd.log=rsyncd.mtail and so on.

[jaqx0r]

A configuration file might be:

• one fragment per program or
• one single config file

containing sections like:

[apache.mtail]
files_to_monitor = /var/log/apache/*.log

[rsync.mtail]
files_to_monitor = /var/log/rsyncd.log

or being named /etc/mtail/configs/apache and only containing the config key=value pairs.

[mattbostock]

@jaqx0r: How about each program having a files_to_monitor (I think that's what you mean by 'one fragment per program')?

[jaqx0r]

You mean putting the config in the program itself?

I think that is a good idea for keeping the total configuration space small, but I don't like it for the maintainability aspect.

I'm discarding it because it means the program text becomes bound to a particular distribution or operating system's configuration. I.e., apache logs aren't in /var/log/apache on Red Hat (IIRC) so hardcoding it there is not correct. I'd like the program to be "change averse" (i.e rarely changed) but the config be customisable by the distro or local operator.

Putting every distro's log location in the program could work in practice, but I imagine there'd be a lot of variance that would need to be supported there, and then the text is noisy with mostly-unused code, irrelevant to almost every user. mtail would end up with lots of fsnotify watchers on paths that would never exist.

To combat that, it could have a parameter that descirbes the distro/os it's running on and select only the ones that match. Now we have yet another config knob and yet more code (and thus bugs.)

So, complexity and maintenance costs make me want to use the external config file, that mean less code in mtail itself, and less program text and configuration for users as well.

On 4 July 2017 at 22:31, Matt Bostock [email protected] wrote:

@jaqx0r https://github.com/jaqx0r: How about each program having a files_to_monitor (I think that's what you mean by 'one fragment per program')?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mtail/issues/35#issuecomment-312948909, or mute the thread https://github.com/notifications/unsubscribe-auth/AC5b-1Irletpksq2cuNxGOWRtNLyG0bxks5sKqETgaJpZM4JN6Al .

[mattbostock]

You mean putting the config in the program itself?

Yep.

I imagine there'd be a lot of variance that would need to be supported there

This assumes thats most mtail users will more commonly use the examples (versus than their own custom programs), which I think is reasonable and a desirable goal.

That said, in the case of distributions packaging mtail, the distribution can set the paths accordingly (see above).

As for people copying the example programs from GitHub, you could either:

• allow all possibilities, and comment them out by default (this would be a burden to maintain)
• allow for all major distributions (again, some maintenance burden, and complications arising from supporting multiple versions)
• just leave the field empty, to be filled in by the user

I suspect that most people are likely to either use the example from their distribution's packaging, or copy the programs into configuration management and they're going to want to template the log location.

So I propose that we put the config in the program itself, leave the field empty, and leave it to users/distribution package maintainers to set it appropriately.

The above said, I'm not a package maintainer.

[NightTsarina]

As a package maintainer (see above, this would not be a problem, I would just patch the examples to point to the right location in Debian.

[jaqx0r]

OK, all we need now is a syntax. Changing the language (by having a new config file) made it easy because the config directs the behaviour of the mtail file watcher. The program code doesn't have a notion of file watching (yet.)

I was thinking of things like C's #pragma, and emacs' comment blocks with embedded lisp, but all make me queasy.

On 7 July 2017 at 18:02, Martín Ferrari [email protected] wrote:

As a package maintainer (see above <#m_7211267015025400118_issuecomment-233216223>, this would not be a problem, I would just patch the examples to point to the right location in Debian.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mtail/issues/35#issuecomment-313723397, or mute the thread https://github.com/notifications/unsubscribe-auth/AC5b-2HLDpNUTQ3pvG08zZ6cZGjLGrnNks5sLlaxgaJpZM4JN6Al .

[jaqx0r]

I just went back to tool at the earlier version, the internal Google mtail, which has a method "IsHandled()" optionally defined in the plugins (aka programs) themselves, so that's a precedent I missed.

Still figuring out how to do this in the new syntax...

[jaqx0r]

I'm looking at AWK again, and note that it uses its global variables to let the programmer modify it's behavoiur. Notably, gawk lets you do things like set BINMODE and LINT:

LINT # When this variable is true (nonzero or non-null), gawk behaves as if the --lint command-line option is in effect (see Options). With a value of "fatal", lint warnings become fatal errors. With a value of "invalid", only warnings about things that are actually invalid are issued. (This is not fully implemented yet.) Any other true value prints nonfatal warnings. Assigning a false value to LINT turns off the lint warnings.

[jaqx0r]

The log filename is now available to the Loader in which the line dispatch happens, so the loader now needs to know if it should filter log lines to VMs.

[jnovack]

If we run one instance of mtail per program, our limiting factor becomes ports at that time, right? CPU addition is effectively insignificant, but is memory also insignificant?

A single instance of mtail monitoring 1 file (that wasn't moving) was only 1.934MiB. Surely we can spare 20MB of memory to run 10 instances?

Is it better to run (a) 1 mtail with 2 programs/2 files, or (b) 2 mtails with 1 program/1 file?

[NightTsarina]

I think there will be use cases for both scenarios. For example, if I am scraping logs for apache, I might want use a different port, to expose the mtail instance as an apache exporter in prometheus, with job=apache, etc.

Currently, I have very convoluted relabelling rules in prometheus to separate mtail timeseries coming from different files into different jobs.

[jaqx0r]

I think it's best to run one mtail per isolation domain, i.e. one per machine, but I'm old school, and perhaps it's better to run one per application as a sidecar container (if isolation domain means namespace/container). I totally recommend one mtail instance per mtail program as a workaround if the log per program fanout cost is affecting anyone today.

Martin, can you spare an example of the relabelling rules you're using?

On 20 April 2018 at 11:45, Martín Ferrari [email protected] wrote:

I think there will be use cases for both scenarios. For example, if I am scraping logs for apache, I might want use a different port, to expose the mtail instance as an apache exporter in prometheus, with job=apache, etc.

Currently, I have very convoluted relabelling rules in prometheus to separate mtail timeseries coming from different files into different jobs.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mtail/issues/35#issuecomment-382938573, or mute the thread https://github.com/notifications/unsubscribe-auth/AC5b-2VlSDZ1D_T8QPelnrA0kT289sS8ks5tqT3SgaJpZM4JN6Al .

[NightTsarina]

Sure, these are the ones I use for apache and postfix. They overwrite job and instance with useful values that make it look as it I had one mtail per service:

  - job_name: 'mtail'
    static_configs:
      - targets: [...]
    metric_relabel_configs:
      - source_labels: [prog, server_port]
        regex: 'apache_metrics.mtail;(.*)'
        target_label: instance
        replacement: ${1}
      - source_labels: [prog, server_port]
        regex: 'apache_metrics.mtail;.*'
        target_label: job
        replacement: apache
      - source_labels: [prog]
        regex: 'apache_metrics.mtail'
        target_label: server_port
        replacement: ''
      - source_labels: [prog]
        regex: 'apache_metrics.mtail'
        target_label: prog
        replacement: ''

      - source_labels: [prog]
        regex: 'postfix.mtail'
        target_label: job
        replacement: 'postfix'
      - source_labels: [prog, instance]
        regex: 'postfix.mtail;(.*):3903'
        target_label: instance
        replacement: '$1:25'
      - source_labels: [prog]
        regex: 'postfix.mtail'
        target_label: prog
        replacement: ''
      - regex: 'exported_instance'
        action: labeldrop

[jnovack]

that make it look as it I had one mtail per service

Hmm, that's mildly confusing, the bread can be sliced two ways. Do you mean "per service" as in:

1. one mtail instance for apache, one mtail instance for postfix (service = program), or
2. one mtail instance for domain1.com for apache/postfix, one mtail instance for domain2.com for apache/postfix (service = domain)

[NightTsarina]

@jnovack , these rules are a bit different for postfix and apache.

For postfix, I just make it look like there is a postfix exporter at {job="postfix'", instance="host:25"}. For apache, I use vhost info extracted from the logs to make it look like {job="apache'", instance="vhost:vport"}. An alternative would have been to present that as {job="apache'", instance="host:80", vhost="vhost:vport"}

[jaqx0r]

I've been hacking up some more programs, and see this pattern:

getfilename() =~ "apache.access.log" {
  # matches testdata/apache_access.log and /var/log/apache/access.log
...
}

It doesn't remove the cost inside the program loader for duplicating lines for every program, but it does allow for early exit from the program if a log source doesn't match.

I should add an 'exit' command so the sense of the test can be inverted and avoid a lot of nesting, i.e.:

getfilename() !~ "apache.access.log" {
  exit  # terminates the program on this line
}

...

[mattpr]

I know this is quite old but as it is still open and top of my mind, I thought I would add some ideas.

We are investing in using mtail for more things and was wondering today if there was a better way to do this (managing which logs are processed by which mtail programs).

Currently we use our provisioning system to just spawn multiple mtail instances. One mtail instance for a given set of logs/programs (usually one set of logs and one program). Downside is that our monitoring then needs to know about the additional ports to crawl and so there is more complicated "service discovery" involved over the recommended "one mtail instance per server".

We have avoided using the Avoiding unnecessary work pattern so we don't have to hard-code log file paths into the mtail program. The provisioning system may be configuring the app and mtail's log path at the same time and we would like to only have to update this in one place.

getfilename() !~ /apache.access.?log/ {
  stop
}

command line flags

Another idea on the command line flags option would be to accept --logs and --progs multiple times and have the order matter (ala find or ffmpeg for example)

ie all consecutive input/--logs would be grouped into a single channel/stream until the first output/--progs

One downside is this breaks your current API (someone doing mtail --logs /foo/bar --progs /bar/foo would be fine but mtail --progs /bar/foo --logs /foo/bar would break).

mtail \
    --logs /var/log/nginx/* \       # input/log-channel (1)
    --progs /opt/mtail/nginx \      # output/programs for log-channel (1)
    --logs /var/log/apache/* \      # input/log-channel (2)
    --logs /var/log/apache2/* \     # more input/log-channel (2)
    --progs /opt/mtail/apache \     # output/programs for log-channel (2)
    --logs /var/log/cron/* \        # input/log-channel (3)
    --progs /opt/mtail/cron \       # output/programs for log-channel (3)
    --progs /opt/mtail/cron2        # more output/programs for log-channel (3)

config file thoughts

If you go down this route, it would be great if mtail could support a conf.d paradigm (e.g. /etc/mtail/mtail.d) as this makes things much easier for auto-provisioning different mtails configs together in different ways on different servers. Having a single config file means that either we have to maintain a version of that config file for every server deployment or we have to mess around with templating (like editing /etc/nginx/nginx.conf rather than dropping a new .conf in /etc/nginx/conf.d/).

You could then have a simplified config file format:

[log_paths]
/var/log/apache/*.log
/var/log/apache2/*.log

[program_dirs]
/opt/mtail/apache

A single config file then maps log paths and mtail program paths together.

The problem here is that you could end up with overlapping configs, ie if you had a second config with a second set of programs that want to monitor some (but not all) of the log files another group of mtail programs are already monitoring.

[log_paths]
/var/log/apache2/*.log

[program_dirs]
/opt/mtail/apache2progs

(both configs monitor /var/log/apache2/*.log with different programs)

I think the solution is that we can't assume a single config implies a 1 to 1 mapping because different configs can overlap in inputs and/or outputs. I think we would need to evaluate all configs to build the final mappings of log inputs to programs/consumers and then group the channels (inputs/outputs) on that basis. This is only really a penalty at setup time (config parse, log rotation, new log file discovery).

On the plus side it would give users a lot of flexibility to tell mtail that progX wants to monitor logA and logB and later progY may also want to monitor logB and logC and this doesn't create a mess on the provisioning/config side of things because progX and progY don't need to know about each other but mtail can still route the right logs to the right programs.