fluentd icon indicating copy to clipboard operation
fluentd copied to clipboard

Published 20 hours ago • verified publisher icon fluent

td-agent is pushing partial data from log file using defined multiple pattern after upgrading td-agent version2 to td-agent4

Open eavdyad opened this issue 1 year ago • 0 comments

Describe the bug

Hi Team,

We have upgraded td-agent version 2.2 to version 4.3 After upgrade, td-agent is pushing partial data to mongodb.

To Reproduce

please help to verify and validate the conf file.

Expected behavior

complete data must be push into mongoDB from log file which match the patter defined in td-agent conf file

Your Environment

- Fluentd version: 4.3
- TD Agent version: 4.3
- Operating system: Red Hat Enterprise Linux 8.0 (Ootpa)
- Kernel version: Linux 4.18.0-80.el8.x86_64

Your Configuration

####
## Output descriptions:
##

# Treasure Data (http://www.treasure-data.com/) provides cloud based data
# analytics platform, which easily stores and processes data from td-agent.
# FREE plan is also provided.
# @see http://docs.fluentd.org/articles/http-to-td
#
# This section matches events whose tag is td.DATABASE.TABLE
<match td.*.*>
  @type tdlog
  apikey YOUR_API_KEY

  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td
</match>

## match tag=debug.** and dump to console
<match debug.**>
  @type stdout
</match>

####
## Source descriptions:
##

## built-in TCP input
## @see http://docs.fluentd.org/articles/in_forward
<source>
  @type forward
</source>

## built-in UNIX socket input
#<source>
#  type unix
#</source>

# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
# POST http://localhost:8888/td.myapp.login?json={"user"%3A"me"}
# @see http://docs.fluentd.org/articles/in_http
<source>
  @type http
  port 8888
</source>

## live debugging agent
<source>
  @type debug_agent
  bind 127.0.0.1
  port 24230
</source>

####
## Examples:
##

## File input
## read apache logs continuously and tags td.apache.access

<source>
  @type tail
  #format multi_format
  
  <parse>
@type multi_format
  
#  <pattern>
# format /^(?<DATE>.*) \| INFO  (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ACTN:(?<ACTN>.*)\|CH:(?<CH>.*)\|CHCODE:(?<CHCODE>.*)\|TP:(?<TP>.*)\|PRODUC:(?<PRODUC>.*)\|PID:(?<PID>.*)\|PNAME:(?<PNAME>.*)\|PCNAME:(?<PCNAME>.*)\|PTP:(?<PTP>.*)\|PUSNAME:(?<PUSNAME>.*)\|PSMSNAME:(?<PSMSNAME>.*)\|PWBNAME:(?<PWBNAME>.*)\|PCNSTRNT:(?<PCNSTRNT>.*)\|DNAME:(?<DNAME>.*)\|SUBNO:(?<SUBNO>.*)\|DIRECT:(?<DIRECT>.*)$/
#  </pattern>

# <pattern>
#format /^(?<DATE>.*) \| INFO  (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ACTN:(?<ACTN>.*)\|CH:(?<CH>.*)\|CHCODE:(?<CHCODE>.*)\|TP:(?<TP>.*)\|PRODUC:(?<PRODUC>.*)\|PID:(?<PID>.*)\|PNAME:(?<PNAME>.*)\|PCNAME:(?<PCNAME>.*)\|PTP:(?<PTP>.*)\|PUSNAME:(?<PUSNAME>.*)\|PSMSNAME:(?<PSMSNAME>.*)\|PWBNAME:(?<PWBNAME>.*)\|PCNSTRNT:(?<PCNSTRNT>.*)\|DNAME:(?<DNAME>.*)\|MSISDN:(?<MSISDN>.*)\|DIRECT:(?<DIRECT>.*)$/
  #</pattern>



  #<pattern>
#format /^(?<DATE>.*) \| INFO  (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ACTN:(?<ACTN>.*)\|CHCODE:(?<CHCODE>.*)\|TP:(?<TP>.*)\|PRODUC:(?<PRODUC>.*)\|PID:(?<PID>.*)\|PNAME:(?<PNAME>.*)\|PCNAME:(?<PCNAME>.*)\|PTP:(?<PTP>.*)\|PUSNAME:(?<PUSNAME>.*)\|PSMSNAME:(?<PSMSNAME>.*)\|PWBNAME:(?<PWBNAME>.*)\|PCNSTRNT:(?<PCNSTRNT>.*)\|DNAME:(?<DNAME>.*)\|SUBNO:(?<SUBNO>.*)\|DIRECT:(?<DIRECT>.*)$/
# </pattern>

  <pattern>
format /^(?<DATE>.*) \| INFO  (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ACTN:(?<ACTN>.*)\|CH:(?<CH>.*)\|CHCODE:(?<CHCODE>.*)\|TP:(?<TP>.*)\|PRODUC:(?<PRODUC>.*)\|PID:(?<PID>.*)\|PNAME:(?<PNAME>.*)\|PCNAME:(?<PCNAME>.*)\|PTP:(?<PTP>.*)\|PUSNAME:(?<PUSNAME>.*)\|PSMSNAME:(?<PSMSNAME>.*)\|PWBNAME:(?<PWBNAME>.*)\|PCNSTRNT:(?<PCNSTRNT>.*)\|DNAME:(?<DNAME>.*)\|MSISDN:(?<MSISDN>.*)\|DIRECT:(?<DIRECT>.*)\|MODE:(?<MODE>.*)\|VMIP:(?<VMIP>.*)\|RPRODID:(?<RPRODID>.*)\|OOPID:(?<OOPID>.*)\|TCODE:(?<TCODE>.*)\|CORRID:(?<CORRID>.*)\|ME2U:(?<ME2U>.*)\|ME2UOP:(?<ME2UOP>.*)\|FNFOP:(?<FNFOP>.*)\|CMDGET:(?<CMDGET>.*)\|CMDSET:(?<CMDSET>.*)\|CMDUA:(?<CMDUA>.*)\|RBCK:(?<RBCK>.*)$/
  </pattern>


  <pattern>
format /^(?<DATE>.*) \| INFO  (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ACTN:(?<ACTN>.*)\|CHCODE:(?<CHCODE>.*)\|TP:(?<TP>.*)\|PRODUC:(?<PRODUC>.*)\|PID:(?<PID>.*)\|PNAME:(?<PNAME>.*)\|PCNAME:(?<PCNAME>.*)\|PTP:(?<PTP>.*)\|PUSNAME:(?<PUSNAME>.*)\|PSMSNAME:(?<PSMSNAME>.*)\|PWBNAME:(?<PWBNAME>.*)\|PCNSTRNT:(?<PCNSTRNT>.*)\|DNAME:(?<DNAME>.*)\|MSISDN:(?<MSISDN>.*)\|DIRECT:(?<DIRECT>.*)\|MODE:(?<MODE>.*)\|VMIP:(?<VMIP>.*)\|RPRODID:(?<RPRODID>.*)\|OOPID:(?<OOPID>.*)\|TCODE::(?<TCODE>.*)\|CORRID:(?<CORRID>.*)\|ME2U:(?<ME2U>.*)\|ME2UOP:(?<ME2UOP>.*)\|FNFOP:(?<FNFOP>.*)\|CMDGET:(?<CMDGET>.*)\|CMDSET:(?<CMDSET>.*)\|CMDUA:(?<CMDUA>.*)\|RBCK:(?<RBCK>.*)$/
  </pattern>


  <pattern>
format /^(?<DATE>.*) \| INFO  (?<INFO>.*) \| RID:(?<RID>.*)\|ACTN:(?<ACTN>.*)\|CH:(?<CH>.*)\|CHCODE:(?<CHCODE>.*)\|TP:(?<TP>.*)\|DNAME:(?<DNAME>.*)\|MSISDN:(?<MSISDN>.*)\|DIRECT:(?<DIRECT>.*)\|MODE:(?<MODE>.*)\|VMIP:(?<VMIP>.*)\|RPRODID:(?<RPRODID>.*)\|OOPID:(?<OOPID>.*)\|TCODE:(?<TCODE>.*)\|CORRID:(?<CORRID>.*)\|ME2U:(?<ME2U>.*)\|ME2UOP:(?<ME2UOP>.*)\|FNFOP:(?<FNFOP>.*)\|CMDGET:(?<CMDGET>.*)\|CMDSET:(?<CMDSET>.*)\|CMDUA:(?<CMDUA>.*)\|RBCK:(?<RBCK>.*)$/
  </pattern>

  #<pattern>
  #format /^(?<DATE>.*) \| INFO  (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ACTN:(?<ACTN>.*)\|CHCODE:(?<CHCODE>.*)\|TP:(?<TP>.*)\|PRODUC:(?<PRODUC>.*)\|PID:(?<PID>.*)\|PNAME:(?<PNAME>.*)\|PCNAME:(?<PCNAME>.*)\|PTP:(?<PTP>.*)\|PUSNAME:(?<PUSNAME>.*)\|PSMSNAME:(?<PSMSNAME>.*)\|PWBNAME:(?<PWBNAME>.*)\|PCNSTRNT:(?<PCNSTRNT>.*)$/
  #</pattern>
  #<pattern>
  #format /^(?<DATE>.*) \| INFO  (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ACTN:(?<ACTN>.[^ ]+)\|CHCODE:(?<CHCODE>.*)\|TP:(?<TP>.*)\|PRODUC:(?<PRODUC>.*)\|PID:(?<PID>.*)\|PNAME:(?<PNAME>.*)\|PCNAME:(?<PCNAME>.*)\|PTP:(?<PTP>.*)\|PUSNAME:(?<PUSNAME>.*)\|PSMSNAME:(?<PSMSNAME>.*)\|PWBNAME:(?<PWBNAME>.*)\|PCNSTRNT:(?<PCNSTRNT>.*)$/
  #</pattern>



  #<pattern>
  #format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ACTN:(?<ACTN>.*)\|CHCODE:(?<CHCODE>.*)\|TP:(?<TP>.*)\|DNAME:(?<DNAME>.*)\|SUBNO:(?<SUBNO>.*)\|DIRECT:(?<DIRECT>.*)$/
  #</pattern>

  #<pattern>
  #format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ACTN:(?<ACTN>.*)\|INIP:(?<INIP>.*)\|CH:(?<CH>.*)\|SUBNO:(?<SUBNO>.*)$/
 #</pattern>


  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|SID:(?<SID>.*)$/
  </pattern>

  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|AMTCHG:(?<AMTCHG>.*)\|CHND:(?<CHND>.*)$/
  </pattern>

  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ISLOWBAL:(?<ISLOWBAL>.*)$/
  </pattern>
  
  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|LOANPURCHASE:(?<LOANPURCHASE>.*)$/
  </pattern>

  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|LOANAMT:(?<LOANAMT>.*)$/
  </pattern>

  
  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|LOANELIGIBLE:(?<LOANELIGIBLE>.*)$/
  </pattern>

  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|CHRGRSLT:(?<CHRGRSLT>.*)$/
  </pattern>

  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|RSN:(?<RSN>.*)$/
  </pattern>

  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|PROVRSLT:(?<PROVRSLT>.[^ ]+)\|PROVRSN:(?<PROVRSN>.*)\|UA99=(?<UA99>.*) \|UA159=(?<UA159>.*) $/
  </pattern>
  
  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|PROVRSLT:(?<PROVRSLT>.[^ ]+)\|UA99=(?<UA99>.*) \|UA159=(?<UA159>.*) $/
  </pattern>
  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|PROVRSLT:(?<PROVRSLT>.[^ ]+)\|PROVRSN:(?<PROVRSN>.*)$/
  </pattern>

  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|PROVRSLT:(?<PROVRSLT>.[^ ]+)$/
  </pattern>



  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|NOTIF:(?<NOTIF>.*)\|TID:(?<TID>.*)\|MSISDN:(?<MSISDN>.*)$/
  </pattern>  

  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|NOTIF:(?<NOTIF>.*)\|TID:(?<TID>.*)\|MSISDN:(?<MSISDN>.*)\|PID:(<PID>.*)$/
  </pattern>  
  
  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|NOTIF:(?<NOTIF>.*)\|TID:(?<TID>.*)$/
  </pattern>
  
  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|NOTIF:(?<NOTIF>.*)$/
  </pattern>


  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|PROVCMD:(?<PROVCMD>.*)\|PARMMOD:(?<PARMMOD>.*)$/
  </pattern>

  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|OGIP:(?<OGIP>.[^ ]+)\|LNAME:(?<LNAME>.[^ ]+)\|CH:(?<CH>.[^ ]+)\|MSISDN:(?<MSISDN>.[^ ]+)\|ORESPRSLT:(?<ORESPRSLT>.[^ ]+)$/
  </pattern>

  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|OGIP:(?<OGIP>.[^ ]+)\|LNAME:(?<LNAME>.[^ ]+)\|CH:(?<CH>.[^ ]+)\|MSISDN:(?<MSISDN>.[^ ]+)\|OREQMODE:(?<OREQMODE>.[^ ]+)$/
  </pattern>

  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|INIP:(?<INIP>.*)\|LNAME:(?<LNAME>.*)\|CH:(?<CH>.*)\|MSISDN:(?<MSISDN>.*)\|PID:(?<PID>.*)$/
  </pattern>



  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ACTN:(?<ACTN>.*)\|INIP:(?<INIP>.*)\|LNAME:(?<LNAME>.*)\|CH:(?<CH>.*)\|MSISDN:(?<MSISDN>.*)$/
  </pattern>


  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|IFIP:(?<IFIP>.[^ ]+)\|IFTP:(?<IFTP>.[^ ]+)\|IFND:(?<IFND>.[^ ]+)\|IFRQMODE:(?<IFRQMODE>.[^ ]+)$/
  </pattern>

  #<pattern>
  #format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ACTN:(?<ACTN>.[^ ]+)\| CHCODE:(?<CHCODE>.[^ ]+)\|TP:(?<TP>.[^ ]+)$/
  #</pattern>

  #<pattern>
  #format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|ACTN:(?<ACTN>.*)\|CHCODE:(?<CHCODE>.*)\|TP:(?<TP>.[^ ]+)$/
  #</pattern>

  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|PROVMOD:(?<PROVMOD>.[^ ]+)\|PROVACT:(?<PROVACT>.[^ ]+)$/
  </pattern>
  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|IFRESMODE:(?<IFRESMODE>.[^ ]+)\|CHND:(?<CHND>.*)$/
  </pattern>
  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|INRESULT:(?<INRESULT>.[^ ]+)$/
  </pattern>
  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|CMDREQ:(?<CMDREQ>.[^ ]+)$/
  </pattern>
  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|PPFLOW:(?<PPFLOW>.[^ ]+)$/
  </pattern>
   
  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|TBH:(?<TBH>.[^ ]+)$/
  </pattern>
  
  <pattern>
  format /^(?<DATE>.*) \| INFO (?<INFO>.*) \| RID:(?<RID>.[^ ]+)\|RequestId:(?<RequestId>.*)$/
  </pattern>

  <pattern>
  format /^(?<DATE>.*) \| ERROR (?<ERROR>.*) \| RID:(?<RID>.[^ ]+)\|ECODE:(?<ECODE>.*)\|EDESC:(?<EDESC:>.*)$/
  </pattern>

  <pattern>
  format /^(?<DATE>.*) \| ERROR (?<ERROR>.*) \| RID:(?<RID>.[^ ]+)\|execution status:(?<execution_status>.*)$/
  </pattern>
</parse>
  path /data/fdp/logs/defaultCircle/requestLogs.log
  pos_file /var/log/log_pos.log
  tag access.test
</source>



## File output
## match tag=local.** and write to file
<match access.test>
  @type mongo
  database logs
  collection GRZMB
  host mongovip
  port 27017
  capped
  capped_size 4974m
  flush_interval 10s
</match>

## Forwarding
## match tag=system.** and forward to another td-agent server
#<match system.**>
#  type forward
#  host 
#  # secondary host is optional
#  <secondary>
#    host 
#  </secondary>
#</match>

## Multiple output
## match tag=td.*.* and output to Treasure Data AND file
#<match td.*.*>
#  type copy
#  <store>
#    type tdlog
#    apikey API_KEY
#    auto_create_table
#    buffer_type file
#    buffer_path /var/log/td-agent/buffer/td
#  </store>
#  <store>
#    type file
#    path /var/log/td-agent/td-%Y-%m-%d/%H.log
#  </store>
#</match>

Your Error Log

Partial data is getting pushed from td-agent to mongoDB

Additional context

No response

eavdyad avatar Sep 16 '22 05:09 eavdyad