Multiple regexes

[Pehesi97]

My fluentd config has two regexes, but only the first one will match.

This is my .conf file.

<source>
  @type http
  port 8888
</source>

## live debugging agent
<source>
  @type debug_agent
  bind 127.0.0.1
  port 24230
</source>

<source>
	@type tail
	path /Users/pedro/.pm2/logs/fv-back-out*.log
	pos_file /Users/pedro/.pm2/logs/fv-back-response.log.pos
	tag fv-back
	<parse>
		@type multi_format
		<pattern>
			format regexp
			expression /(?<time>((\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})\.(\d{3})) (\-|\+)(\d{2}):(\d{2})), \[(?<tags>(.)*response(.)*)\] data: (?<method>[^ ]+) (?<endpoint>[^ ]+) (?<query>[^ ]+) (?<payload>[^ ]+) (?<statusCode>[^ ]+) \((?<responseTime>[^ ]+)\) (?<credentials>[^ ]+)/
			time_format %Y-%m-%d %H:%M:%S.%L
			types time:time,tags:array,statusCode:integer
		</pattern>
		<pattern>
			format regexp
			expression /(?<time>((\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})\.(\d{3})) (\-|\+)(\d{2}):(\d{2})), \[(?<tags>((.)*error(.)*))\] data: \[(?<statusCode>.*)\] (?<type>.*): (?<message>.*)/
			time_format %Y-%m-%d %H-%M-%S.%L
                	types time:time,tags:array,statusCode:integer
		</pattern>
	</parse>
</source>

<match fv-back*>
	@type copy

	<store>
		@type elasticsearch
		host localhost
		port 9200
		logstash_format true
	</store>

	<store>
		@type s3
		aws_key_id AKIAI3BKNP4PWLJW5PMQ
		aws_sec_key hoWhK+KzAKOwKuPszBkS8099Nuu8WZKJKo516XUY
		s3_bucket softruck.pm2.logs
		s3_region us-east-1
		path /

		<buffer time>
			@type file
			path /var/log/td-agent/s3_buffer
			timekey 3600
			timekey wait 10m
			timekey_use_utc true
			chunk_limit_size 256m
		</buffer>
	</store>
</match>

<match **>
	@type null
</match>

And this 2018-08-08 16:45:05 -0300 [warn]: #0 pattern not match: "2018-08-08 16:45:04.869 -03:00, [log,api,error] data: [500] SequelizeDatabaseError: relation \"daily_courses\" does not exist" is my Fluentd log.

Rubular says my regex should pass. Is there anything I'm doing wrong?

[ZhangSIming-blyq]

same issue here. wait for an answer.

[ggpaue]

same issue here, can anyone help?

[rsilva-rs]

You could try this plugin: https://github.com/sesame/fluent-plugin-regexp_multi

[davelosert]

We are having the same issue - is there a chance this will get fixed?

[ankit1mg]

We are having the same issue - is there a chance this will get fixed?

+1

[asdfII]

We are having the same issue - is there a chance this will get fixed?

+1

[cm0s]

Same issue. As anyone found a solution? @rsilva-rs plugin doesn't seem to exist anymore.

[repeatedly]

I think this is not a bug. Issue author's problem is the mismatch between log body and time format.

Second expression matches 2018-08-08 16:45:04.869 -03:00, [log,api,error] data: [500] SequelizeDatabaseError: relation "daily_courses" does not exist but time field, 2018-08-08 16:45:04.869, is %Y-%m-%d %H:%M:%S.%L format, not %Y-%m-%d %H-%M-%S.%L. Put more <pattern> fixes this problem. If you have similar pattern not match log, check your expression/time_format combination for actual logs.