paStash icon indicating copy to clipboard operation
paStash copied to clipboard

Published 20 hours ago • verified publisher icon sipcapture

Cisco UCM CDR filter

Open astrakid opened this issue 4 years ago • 18 comments

Hi, I want to heplify Cisco UCM CDRs (currently version 12.5, but afaik the structure didn't change in the last years). The plain-text-files have the following structure:

column-name | type | example

cdrRecordType | INTEGER | 1 globalCallID_callManagerId | INTEGER | 2 globalCallID_callId | INTEGER | 28377 origLegCallIdentifier | INTEGER | 36951964 dateTimeOrigination | INTEGER | 1593547211 origNodeId | INTEGER | 2 origSpan | INTEGER | 36951964 origIpAddr | INTEGER | 152200202 callingPartyNumber | VARCHAR(50) | +49123456789 callingPartyUnicodeLoginUserID | VARCHAR(128) |   origCause_location | INTEGER | 0 origCause_value | INTEGER | 16 origPrecedenceLevel | INTEGER | 4 origMediaTransportAddress_IP | INTEGER | 152200202 origMediaTransportAddress_Port | INTEGER | 23618 origMediaCap_payloadCapability | INTEGER | 2 origMediaCap_maxFramesPerPacket | INTEGER | 20 origMediaCap_g723BitRate | INTEGER | 0 origVideoCap_Codec | INTEGER | 0 origVideoCap_Bandwidth | INTEGER | 0 origVideoCap_Resolution | INTEGER | 0 origVideoTransportAddress_IP | INTEGER | 0 origVideoTransportAddress_Port | INTEGER | 0 origRSVPAudioStat | VARCHAR(64) | 0 origRSVPVideoStat | VARCHAR(64) | 0 destLegIdentifier | INTEGER | 36951967 destNodeId | INTEGER | 2 destSpan | INTEGER | 36951967 destIpAddr | INTEGER | 152200202 originalCalledPartyNumber | VARCHAR(50) | +4923456789 finalCalledPartyNumber | VARCHAR(50) | +4923456789 finalCalledPartyUnicodeLoginUserID | VARCHAR(128) |   destCause_location | INTEGER | 0 destCause_value | INTEGER | 0 destPrecedenceLevel | INTEGER | 4 destMediaTransportAddress_IP | INTEGER | 152200202 destMediaTransportAddress_Port | INTEGER | 23320 destMediaCap_payloadCapability | INTEGER | 2 destMediaCap_maxFramesPerPacket | INTEGER | 20 destMediaCap_g723BitRate | INTEGER | 0 destVideoCap_Codec | INTEGER | 0 destVideoCap_Bandwidth | INTEGER | 0 destVideoCap_Resolution | INTEGER | 0 destVideoTransportAddress_IP | INTEGER | 0 destVideoTransportAddress_Port | INTEGER | 0 destRSVPAudioStat | VARCHAR(64) | 0 destRSVPVideoStat | VARCHAR(64) | 0 dateTimeConnect | INTEGER | 1593547211 dateTimeDisconnect | INTEGER | 1593547987 lastRedirectDn | VARCHAR(50) | +49258147369 pkid | UNIQUEIDENTIFIER | dbfc3109-c95c-4a46-96fe-d94d89e20cde originalCalledPartyNumberPartition | VARCHAR(50) | PSTN_OUT callingPartyNumberPartition | VARCHAR(50) |   finalCalledPartyNumberPartition | VARCHAR(50) | PSTN_OUT lastRedirectDnPartition | VARCHAR(50) |   duration | INTEGER | 776 origDeviceName | VARCHAR(129) | CUBE destDeviceName | VARCHAR(129) | CUBE origCallTerminationOnBehalfOf | INTEGER | 12 destCallTerminationOnBehalfOf | INTEGER | 17 origCalledPartyRedirectOnBehalfOf | INTEGER | 0 lastRedirectRedirectOnBehalfOf | INTEGER | 18 origCalledPartyRedirectReason | INTEGER | 0 lastRedirectRedirectReason | INTEGER | 146 destConversationId | INTEGER | 0 globalCallId_ClusterID | VARCHAR(50) | StandAloneCluster joinOnBehalfOf | INTEGER | 18 comment | VARCHAR(2048) |   authCodeDescription | VARCHAR(50) |   authorizationLevel | INTEGER | 0 clientMatterCode | VARCHAR(32) |   origDTMFMethod | INTEGER | 2 destDTMFMethod | INTEGER | 2 callSecuredStatus | INTEGER | 0 origConversationId | INTEGER | 0 origMediaCap_Bandwidth | INTEGER | 64 destMediaCap_Bandwidth | INTEGER | 64 authorizationCodeValue | VARCHAR(32) |   outpulsedCallingPartyNumber | VARCHAR(50) |   outpulsedCalledPartyNumber | VARCHAR(50) |   origIpv4v6Addr | VARCHAR(64) | 10.10.10.99 destIpv4v6Addr | VARCHAR(64) | 10.100.10.99 origVideoCap_Codec_Channel2 | INTEGER | 0 origVideoCap_Bandwidth_Channel2 | INTEGER | 0 origVideoCap_Resolution_Channel2 | INTEGER | 0 origVideoTransportAddress_IP_Channel2 | INTEGER | 0 origVideoTransportAddress_Port_Channel2 | INTEGER | 0 origVideoChannel_Role_Channel2 | INTEGER | 0 destVideoCap_Codec_Channel2 | INTEGER | 0 destVideoCap_Bandwidth_Channel2 | INTEGER | 0 destVideoCap_Resolution_Channel2 | INTEGER | 0 destVideoTransportAddress_IP_Channel2 | INTEGER | 0 destVideoTransportAddress_Port_Channel2 | INTEGER | 0 destVideoChannel_Role_Channel2 | INTEGER | 0 IncomingProtocolID | INTEGER | 1 IncomingProtocolCallRef | VARCHAR(32) | 216DF07ABA4311EAB627B490D23A7537 OutgoingProtocolID | INTEGER | 1 OutgoingProtocolCallRef | VARCHAR(32) | 4AF15980000100000009151F0212640A currentRoutingReason | INTEGER | 0 origRoutingReason | INTEGER | 0 lastRedirectingRoutingReason | INTEGER | 0 huntPilotPartition | VARCHAR(50) |   huntPilotDN | VARCHAR(50) |   calledPartyPatternUsage | INTEGER | 5 IncomingICID | VARCHAR(50) |   IncomingOrigIOI | VARCHAR(50) |   IncomingTermIOI | VARCHAR(50) |   OutgoingICID | VARCHAR(50) |   OutgoingOrigIOI | VARCHAR(50) |   OutgoingTermIOI | VARCHAR(50) |   outpulsedOriginalCalledPartyNumber | VARCHAR(50) |   outpulsedLastRedirectingNumber | VARCHAR(50) |   wasCallQueued | INTEGER | 0 totalWaitTimeInQueue | INTEGER | 0 callingPartyNumber_uri | VARCHAR(255) |   originalCalledPartyNumber_uri | VARCHAR(255) |   finalCalledPartyNumber_uri | VARCHAR(255) |   lastRedirectDn_uri | VARCHAR(255) |   mobileCallingPartyNumber | VARCHAR(50) |   finalMobileCalledPartyNumber | VARCHAR(50) |   origMobileDeviceName | VARCHAR(129) |   destMobileDeviceName | VARCHAR(129) |   origMobileCallDuration | INTEGER | 0 destMobileCallDuration | INTEGER | 0 mobileCallType | INTEGER | 0 originalCalledPartyPattern | VARCHAR(50) | +! finalCalledPartyPattern | VARCHAR(50) | +! lastRedirectingPartyPattern | VARCHAR(50) |   huntPilotPattern | VARCHAR(50) |   origDeviceType | VARCHAR(100) |   destDeviceType | VARCHAR(100) |   origDeviceSessionID | VARCHAR(128) | fe17729285175238945d2e0c49e9929c destDeviceSessionID | VARCHAR(128) | b78e7565083b5488b59e1cdc3ba7f58c

astrakid avatar Nov 19 '20 12:11 astrakid

Hi Andre - Are those materialized as individual files?

lmangani avatar Nov 19 '20 12:11 lmangani

he file is simple csv, that means the example given looks in csv like this:

1,2,28377,36951964,1593547211,2,36951964,152200202,"+4912345678","",0,16,4,152200202,23618,2,20,0,0,0,0,0,0,"0","0",36951967,2,36951967,152200202,"+4923456789","+4923456789","",0,0,4,152200202,23320,2,20,0,0,0,0,0,0,"0","0",1593547211,1593547987,"+4912345678","dbfc3109-c95c-4a46-96fe-d94d89e20cde","PSTN_OUT","","PSTN_OUT","",776,"CUBE","CUBE",12,17,0,18,0,146,0,"StandAloneCluster",18,"","",0,"",2,2,0,0,64,64,"","","","10.100.10.99","10.100.10.99",0,0,0,0,0,0,0,0,0,0,0,0,1,"216DF07ABA4311EAB627B490D23A7537",1,"4AF15980000100000009151F0212640A",0,0,0,"","",5,"","","","","","","","",0,0,"","","","","","","","",0,0,0,"+!","+!","","","","","fe17729285175238945d2e0c49e9929c","b78e7565083b5488b59e1cdc3ba7f58c"

astrakid avatar Nov 19 '20 12:11 astrakid

first line of each csv contains column names, second line column types, line 3 ff contains data.

astrakid avatar Nov 19 '20 12:11 astrakid

Hi Andre - Are those materialized as individual files?

no, datasets are grouped by time - in our setup each hour a new file is created containing all CDRs / CMRs for that timeframe (one file for CDRs, one for CMRs).

astrakid avatar Nov 19 '20 12:11 astrakid

wait up - i'm confused. if its a CSV file each hour, how can the values be spread on multiple lines? it seems unlikely. from the example, it seems there must be a fixed header on the first line of the file, and each cdr to be one full line after.

Could you please provide the first 3-4 lines of any such csv file?

lmangani avatar Nov 19 '20 12:11 lmangani

sorry for confusion. here the first 4 lines of a cdr file of 2020 june 30th 21:13:

"cdrRecordType","globalCallID_callManagerId","globalCallID_callId","origLegCallIdentifier","dateTimeOrigination","origNodeId","origSpan","origIpAddr","callingPartyNumber","callingPartyUnicodeLoginUserID","origCause_location","origCause_value","origPrecedenceLevel","origMediaTransportAddress_IP","origMediaTransportAddress_Port","origMediaCap_payloadCapability","origMediaCap_maxFramesPerPacket","origMediaCap_g723BitRate","origVideoCap_Codec","origVideoCap_Bandwidth","origVideoCap_Resolution","origVideoTransportAddress_IP","origVideoTransportAddress_Port","origRSVPAudioStat","origRSVPVideoStat","destLegIdentifier","destNodeId","destSpan","destIpAddr","originalCalledPartyNumber","finalCalledPartyNumber","finalCalledPartyUnicodeLoginUserID","destCause_location","destCause_value","destPrecedenceLevel","destMediaTransportAddress_IP","destMediaTransportAddress_Port","destMediaCap_payloadCapability","destMediaCap_maxFramesPerPacket","destMediaCap_g723BitRate","destVideoCap_Codec","destVideoCap_Bandwidth","destVideoCap_Resolution","destVideoTransportAddress_IP","destVideoTransportAddress_Port","destRSVPAudioStat","destRSVPVideoStat","dateTimeConnect","dateTimeDisconnect","lastRedirectDn","pkid","originalCalledPartyNumberPartition","callingPartyNumberPartition","finalCalledPartyNumberPartition","lastRedirectDnPartition","duration","origDeviceName","destDeviceName","origCallTerminationOnBehalfOf","destCallTerminationOnBehalfOf","origCalledPartyRedirectOnBehalfOf","lastRedirectRedirectOnBehalfOf","origCalledPartyRedirectReason","lastRedirectRedirectReason","destConversationId","globalCallId_ClusterID","joinOnBehalfOf","comment","authCodeDescription","authorizationLevel","clientMatterCode","origDTMFMethod","destDTMFMethod","callSecuredStatus","origConversationId","origMediaCap_Bandwidth","destMediaCap_Bandwidth","authorizationCodeValue","outpulsedCallingPartyNumber","outpulsedCalledPartyNumber","origIpv4v6Addr","destIpv4v6Addr","origVideoCap_Codec_Channel2","origVideoCap_Bandwidth_Channel2","origVideoCap_Resolution_Channel2","origVideoTransportAddress_IP_Channel2","origVideoTransportAddress_Port_Channel2","origVideoChannel_Role_Channel2","destVideoCap_Codec_Channel2","destVideoCap_Bandwidth_Channel2","destVideoCap_Resolution_Channel2","destVideoTransportAddress_IP_Channel2","destVideoTransportAddress_Port_Channel2","destVideoChannel_Role_Channel2","IncomingProtocolID","IncomingProtocolCallRef","OutgoingProtocolID","OutgoingProtocolCallRef","currentRoutingReason","origRoutingReason","lastRedirectingRoutingReason","huntPilotPartition","huntPilotDN","calledPartyPatternUsage","IncomingICID","IncomingOrigIOI","IncomingTermIOI","OutgoingICID","OutgoingOrigIOI","OutgoingTermIOI","outpulsedOriginalCalledPartyNumber","outpulsedLastRedirectingNumber","wasCallQueued","totalWaitTimeInQueue","callingPartyNumber_uri","originalCalledPartyNumber_uri","finalCalledPartyNumber_uri","lastRedirectDn_uri","mobileCallingPartyNumber","finalMobileCalledPartyNumber","origMobileDeviceName","destMobileDeviceName","origMobileCallDuration","destMobileCallDuration","mobileCallType","originalCalledPartyPattern","finalCalledPartyPattern","lastRedirectingPartyPattern","huntPilotPattern","origDeviceType","destDeviceType","origDeviceSessionID","destDeviceSessionID" INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,VARCHAR(50),VARCHAR(128),INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,VARCHAR(64),VARCHAR(64),INTEGER,INTEGER,INTEGER,INTEGER,VARCHAR(50),VARCHAR(50),VARCHAR(128),INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,VARCHAR(64),VARCHAR(64),INTEGER,INTEGER,VARCHAR(50),UNIQUEIDENTIFIER,VARCHAR(50),VARCHAR(50),VARCHAR(50),VARCHAR(50),INTEGER,VARCHAR(129),VARCHAR(129),INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,VARCHAR(50),INTEGER,VARCHAR(2048),VARCHAR(50),INTEGER,VARCHAR(32),INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,VARCHAR(32),VARCHAR(50),VARCHAR(50),VARCHAR(64),VARCHAR(64),INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,INTEGER,VARCHAR(32),INTEGER,VARCHAR(32),INTEGER,INTEGER,INTEGER,VARCHAR(50),VARCHAR(50),INTEGER,VARCHAR(50),VARCHAR(50),VARCHAR(50),VARCHAR(50),VARCHAR(50),VARCHAR(50),VARCHAR(50),VARCHAR(50),INTEGER,INTEGER,VARCHAR(255),VARCHAR(255),VARCHAR(255),VARCHAR(255),VARCHAR(50),VARCHAR(50),VARCHAR(129),VARCHAR(129),INTEGER,INTEGER,INTEGER,VARCHAR(50),VARCHAR(50),VARCHAR(50),VARCHAR(50),VARCHAR(100),VARCHAR(100),VARCHAR(128),VARCHAR(128) 1,2,28377,36951964,1593547211,2,36951964,152200202,"+49123456780","",0,16,4,152200202,23618,2,20,0,0,0,0,0,0,"0","0",36951967,2,36951967,152200202,"+492222222225","+492222222225","",0,0,4,152200202,23320,2,20,0,0,0,0,0,0,"0","0",1593547211,1593547987,"+49222222220","dbfc3109-c95c-4a46-96fe-d94d89e20cde","PSTN_OUT","","PSTN_OUT","",776,"CUBE","CUBE",12,17,0,18,0,146,0,"StandAloneCluster",18,"","",0,"",2,2,0,0,64,64,"","","","10.10.10.9","10.10.10.9",0,0,0,0,0,0,0,0,0,0,0,0,1,"216DF07ABA4311EAB627B490D23A7537",1,"4AF15980000100000009151F0212640A",0,0,0,"","",5,"","","","","","","","",0,0,"","","","","","","","",0,0,0,"+!","+!","","","","","fe17729285175238945d2e0c49e9929c","b78e7565083b5488b59e1cdc3ba7f58c" 1,2,28381,36951974,1593547949,2,36951974,152200202,"+49811111186","",0,0,4,152200202,25650,2,20,0,0,0,0,0,0,"0","0",36951975,2,36951975,152200202,"+49333333333336","+493333333333336","",0,16,4,152200202,25118,2,20,0,0,0,0,0,0,"0","0",1593547949,1593548010,"+4933333336","c6de9862-b162-4b0b-9e62-81dcdc6d3d3b","PSTN_OUT","","PSTN_OUT","PSTN_OUT",61,"CUBE","CUBE",0,12,0,0,0,0,0,"StandAloneCluster",0,"","",0,"",2,2,0,0,64,64,"","","","10.10.10.9","10.10.10.9",0,0,0,0,0,0,0,0,0,0,0,0,1,"DB6E689EBA4411EAB68BB490D23A7537",1,"049CFA0000010000000915230212640A",0,0,0,"","",5,"","","","","","","","",0,0,"","","","","","","","",0,0,0,"+!","+!","+!","","","","111bc3de1f13551090ee3b07a8b368b9","887ee37864ae553ea014984224b04b77"

astrakid avatar Nov 19 '20 12:11 astrakid

each new file contains x datasets, where each row contains one dataset (call detail record). except the first two rows of each files are containing headers. hope to calrify it. ;)

astrakid avatar Nov 19 '20 12:11 astrakid

overall, the first step is pretty easy - pastash already has a CSV parser - we just skip the first 2 lines and teach it what headers. From there we have a JSON object we can use to form a HEP packet of various types.

input {
  file {
    path => '/opt/cisco.cdr'
    start_index => 2
  }
}


filter {
  csv {
    headers => ["cdrRecordType","globalCallID_callManagerId","globalCallID_callId","origLegCallIdentifier","dateTimeOrigination","origNodeId","origSpan","origIpAddr","callingPartyNumber","callingPartyUnicodeLoginUserID","origCause_location","origCause_value","origPrecedenceLevel","origMediaTransportAddress_IP","origMediaTransportAddress_Port","origMediaCap_payloadCapability","origMediaCap_maxFramesPerPacket","origMediaCap_g723BitRate","origVideoCap_Codec","origVideoCap_Bandwidth","origVideoCap_Resolution","origVideoTransportAddress_IP","origVideoTransportAddress_Port","origRSVPAudioStat","origRSVPVideoStat","destLegIdentifier","destNodeId","destSpan","destIpAddr","originalCalledPartyNumber","finalCalledPartyNumber","finalCalledPartyUnicodeLoginUserID","destCause_location","destCause_value","destPrecedenceLevel","destMediaTransportAddress_IP","destMediaTransportAddress_Port","destMediaCap_payloadCapability","destMediaCap_maxFramesPerPacket","destMediaCap_g723BitRate","destVideoCap_Codec","destVideoCap_Bandwidth","destVideoCap_Resolution","destVideoTransportAddress_IP","destVideoTransportAddress_Port","destRSVPAudioStat","destRSVPVideoStat","dateTimeConnect","dateTimeDisconnect","lastRedirectDn","pkid","originalCalledPartyNumberPartition","callingPartyNumberPartition","finalCalledPartyNumberPartition","lastRedirectDnPartition","duration","origDeviceName","destDeviceName","origCallTerminationOnBehalfOf","destCallTerminationOnBehalfOf","origCalledPartyRedirectOnBehalfOf","lastRedirectRedirectOnBehalfOf","origCalledPartyRedirectReason","lastRedirectRedirectReason","destConversationId","globalCallId_ClusterID","joinOnBehalfOf","comment","authCodeDescription","authorizationLevel","clientMatterCode","origDTMFMethod","destDTMFMethod","callSecuredStatus","origConversationId","origMediaCap_Bandwidth","destMediaCap_Bandwidth","authorizationCodeValue","outpulsedCallingPartyNumber","outpulsedCalledPartyNumber","origIpv4v6Addr","destIpv4v6Addr","origVideoCap_Codec_Channel2","origVideoCap_Bandwidth_Channel2","origVideoCap_Resolution_Channel2","origVideoTransportAddress_IP_Channel2","origVideoTransportAddress_Port_Channel2","origVideoChannel_Role_Channel2","destVideoCap_Codec_Channel2","destVideoCap_Bandwidth_Channel2","destVideoCap_Resolution_Channel2","destVideoTransportAddress_IP_Channel2","destVideoTransportAddress_Port_Channel2","destVideoChannel_Role_Channel2","IncomingProtocolID","IncomingProtocolCallRef","OutgoingProtocolID","OutgoingProtocolCallRef","currentRoutingReason","origRoutingReason","lastRedirectingRoutingReason","huntPilotPartition","huntPilotDN","calledPartyPatternUsage","IncomingICID","IncomingOrigIOI","IncomingTermIOI","OutgoingICID","OutgoingOrigIOI","OutgoingTermIOI","outpulsedOriginalCalledPartyNumber","outpulsedLastRedirectingNumber","wasCallQueued","totalWaitTimeInQueue","callingPartyNumber_uri","originalCalledPartyNumber_uri","finalCalledPartyNumber_uri","lastRedirectDn_uri","mobileCallingPartyNumber","finalMobileCalledPartyNumber","origMobileDeviceName","destMobileDeviceName","origMobileCallDuration","destMobileCallDuration","mobileCallType","originalCalledPartyPattern","finalCalledPartyPattern","lastRedirectingPartyPattern","huntPilotPattern","origDeviceType","destDeviceType","origDeviceSessionID","destDeviceSessionID"]
  }
  compute_field {
    field => correlation_id
    value => "#{globalCallID_callId}"
  }
}

output {
  stdout {}
}

From here its on you to tell us what could/should be done with all those columns. If you have enough elements, we can transform them into a meaningful payload.

If you'd like to send the CDR as log type for playing, make sure the compute_field actually grabs the right Call-ID column, and then use the following output:

output {
  hep {
    host => 127.0.0.1
    port => 9060
    hep_id => 2022
    hep_type => 100
  }
}

lmangani avatar Nov 19 '20 13:11 lmangani

sounds really good, correlation id is fine. if i run pastash twice on the same file, will I get duplicated entries?

astrakid avatar Nov 19 '20 15:11 astrakid

besides my last question i have some errors, a field seems to be too large in my files:

[Fri, 20 Nov 2020 09:09:48 GMT] INFO Start reading /opt/ciscoCdr/cdr_StandAloneCluster_02_202011191314_46926 at 2 fd 578 [Fri, 20 Nov 2020 09:09:49 GMT] ERROR PREHEP ERROR: RangeError [ERR_OUT_OF_RANGE]: The value of "value" is out of range. It must be >= 0 and <= 65535. Received 1072476

astrakid avatar Nov 20 '20 09:11 astrakid

and last but not least: paStash is running, but i don't see any data in table hep_proto_100_default or hep_proto_100_default__.

how to debug it?

running pastash in debug mode looks quite fine:

[Fri, 20 Nov 2020 11:25:05 GMT] DEBUG CSV Data! { '0': '1', '1': '2', '2': '979553', '3': '41410853', '4': '1605870844', '5': '2', '6': '41410853', '7': '1729258506', '8': '+491794181082', '9': '', '10': '0', '11': '393216', '12': '4', '13': '1729258506', '14': '18130', '15': '4', '16': '20', '17': '0', '18': '0', '19': '0', '20': '0', '21': '0', '22': '0', '23': '0', '24': '0', '25': '41410854', '26': '2', '27': '0', '28': '-354288118', '29': '1306', '30': '1306', '31': 'cuwl', '32': '0', '33': '393216', '34': '4', '35': '-354288118', '36': '18054', '37': '4', '38': '20', '39': '0', '40': '0', '41': '0', '42': '0', '43': '0', '44': '0', '45': '0', '46': '0', '47': '1605870844', '48': '1605870845', '49': '1306', '50': 'c79b1957-08b4-4905-bb43-84fd8998acd1', '51': 'Eingeloggt', '52': '', '53': 'Eingeloggt', '54': 'Eingeloggt', '55': '1', '56': 'big-vacdgw1', '57': 'CSF1306', '58': '18', '59': '18', '60': '0', '61': '0', '62': '0', '63': '0', '64': '0', '65': 'StandAloneCluster', '66': '0', '67': '', '68': '', '69': '0', '70': '', '71': '2', '72': '3', '73': '0', '74': '0', '75': '64', '76': '64', '77': '', '78': '', '79': '', '80': '10.100.18.103', '81': '10.254.225.234', '82': '0', '83': '0', '84': '0', '85': '0', '86': '0', '87': '0', '88': '0', '89': '0', '90': '0', '91': '0', '92': '0', '93': '0', '94': '1', '95': '7E676B8000010000001B46C30212640A', '96': '3', '97': '00000000000EF2610277E12600000000', '98': '0', '99': '0', '100': '0', '101': '', '102': '', '103': '2', '104': '', '105': '', '106': '', '107': '', '108': '', '109': '', '110': '', '111': '', '112': '0', '113': '0', '114': '', '115': '[email protected]', '116': '[email protected]', '117': '[email protected]', '118': '', '119': '', '120': '', '121': '', '122': '0', '123': '0', '124': '0', '125': '1306', '126': '1306', '127': '1306', '128': '', '129': '', '130': '', '131': 'd3e5b1d52522636a5ba969ab83463097', '132': '79df6b5800105000a00098e7434cdd1d' }

config file: output { # stdout {} hep { host => 127.0.0.1 port => 9060 hep_id => 18000 hep_type => 100 } }

netstat on local server: root@sipmon01:/opt/ciscoCdr# netstat -tulpen|grep 9060 tcp6 0 0 :::9060 :::* LISTEN 0 31381 1588/docker-proxy udp 0 0 0.0.0.0:51105 0.0.0.0:* 0 39060831 1022260/node udp6 0 0 :::9060 :::* 0 31438 1616/docker-proxy

astrakid avatar Nov 20 '20 12:11 astrakid

I doubt this would or should work, but I can't tell without seeing the config you're running.

The point of pastash is not sending the data as-is (unless its already well formatted) for rather transforming it into what it should be. In this case, nothing will work without a correlation_id to begin with and sending the raw CSV seems to me the wrong choice. The CSV parser (below) will turn the rows into a JSON object. You might want to try send that AFTER electing a correlation ID (call-id of the sip session, from the CDR)

filter {
  csv {
    headers => ["cdrRecordType","globalCallID_callManagerId","globalCallID_callId","origLegCallIdentifier","dateTimeOrigination","origNodeId","origSpan","origIpAddr","callingPartyNumber","callingPartyUnicodeLoginUserID","origCause_location","origCause_value","origPrecedenceLevel","origMediaTransportAddress_IP","origMediaTransportAddress_Port","origMediaCap_payloadCapability","origMediaCap_maxFramesPerPacket","origMediaCap_g723BitRate","origVideoCap_Codec","origVideoCap_Bandwidth","origVideoCap_Resolution","origVideoTransportAddress_IP","origVideoTransportAddress_Port","origRSVPAudioStat","origRSVPVideoStat","destLegIdentifier","destNodeId","destSpan","destIpAddr","originalCalledPartyNumber","finalCalledPartyNumber","finalCalledPartyUnicodeLoginUserID","destCause_location","destCause_value","destPrecedenceLevel","destMediaTransportAddress_IP","destMediaTransportAddress_Port","destMediaCap_payloadCapability","destMediaCap_maxFramesPerPacket","destMediaCap_g723BitRate","destVideoCap_Codec","destVideoCap_Bandwidth","destVideoCap_Resolution","destVideoTransportAddress_IP","destVideoTransportAddress_Port","destRSVPAudioStat","destRSVPVideoStat","dateTimeConnect","dateTimeDisconnect","lastRedirectDn","pkid","originalCalledPartyNumberPartition","callingPartyNumberPartition","finalCalledPartyNumberPartition","lastRedirectDnPartition","duration","origDeviceName","destDeviceName","origCallTerminationOnBehalfOf","destCallTerminationOnBehalfOf","origCalledPartyRedirectOnBehalfOf","lastRedirectRedirectOnBehalfOf","origCalledPartyRedirectReason","lastRedirectRedirectReason","destConversationId","globalCallId_ClusterID","joinOnBehalfOf","comment","authCodeDescription","authorizationLevel","clientMatterCode","origDTMFMethod","destDTMFMethod","callSecuredStatus","origConversationId","origMediaCap_Bandwidth","destMediaCap_Bandwidth","authorizationCodeValue","outpulsedCallingPartyNumber","outpulsedCalledPartyNumber","origIpv4v6Addr","destIpv4v6Addr","origVideoCap_Codec_Channel2","origVideoCap_Bandwidth_Channel2","origVideoCap_Resolution_Channel2","origVideoTransportAddress_IP_Channel2","origVideoTransportAddress_Port_Channel2","origVideoChannel_Role_Channel2","destVideoCap_Codec_Channel2","destVideoCap_Bandwidth_Channel2","destVideoCap_Resolution_Channel2","destVideoTransportAddress_IP_Channel2","destVideoTransportAddress_Port_Channel2","destVideoChannel_Role_Channel2","IncomingProtocolID","IncomingProtocolCallRef","OutgoingProtocolID","OutgoingProtocolCallRef","currentRoutingReason","origRoutingReason","lastRedirectingRoutingReason","huntPilotPartition","huntPilotDN","calledPartyPatternUsage","IncomingICID","IncomingOrigIOI","IncomingTermIOI","OutgoingICID","OutgoingOrigIOI","OutgoingTermIOI","outpulsedOriginalCalledPartyNumber","outpulsedLastRedirectingNumber","wasCallQueued","totalWaitTimeInQueue","callingPartyNumber_uri","originalCalledPartyNumber_uri","finalCalledPartyNumber_uri","lastRedirectDn_uri","mobileCallingPartyNumber","finalMobileCalledPartyNumber","origMobileDeviceName","destMobileDeviceName","origMobileCallDuration","destMobileCallDuration","mobileCallType","originalCalledPartyPattern","finalCalledPartyPattern","lastRedirectingPartyPattern","huntPilotPattern","origDeviceType","destDeviceType","origDeviceSessionID","destDeviceSessionID"]
  }
  compute_field {
    field => correlation_id
    value => "#{globalCallID_callId}"
  }
}

lmangani avatar Nov 20 '20 12:11 lmangani

ah sorry, just piped the output of my config, the filter was set as suggested from you before.

here completely: `root@sipmon01:/opt/ciscoCdr# cat /opt/paStash/bin/config.yaml input { file { path => '/opt/ciscoCdr/cdr*' start_index => 2 } }

filter { csv { headers => ["cdrRecordType","globalCallID_callManagerId","globalCallID_callId","origLegCallIdentifier","dateTimeOrigination","origNodeId","origSpan","origIpAddr","callingPartyNumber","callingPartyUnicodeLoginUserID","origCause_location","origCause_value","origPrecedenceLevel","origMediaTransportAddress_IP","origMediaTransportAddress_Port","origMediaCap_payloadCapability","origMediaCap_maxFramesPerPacket","origMediaCap_g723BitRate","origVideoCap_Codec","origVideoCap_Bandwidth","origVideoCap_Resolution","origVideoTransportAddress_IP","origVideoTransportAddress_Port","origRSVPAudioStat","origRSVPVideoStat","destLegIdentifier","destNodeId","destSpan","destIpAddr","originalCalledPartyNumber","finalCalledPartyNumber","finalCalledPartyUnicodeLoginUserID","destCause_location","destCause_value","destPrecedenceLevel","destMediaTransportAddress_IP","destMediaTransportAddress_Port","destMediaCap_payloadCapability","destMediaCap_maxFramesPerPacket","destMediaCap_g723BitRate","destVideoCap_Codec","destVideoCap_Bandwidth","destVideoCap_Resolution","destVideoTransportAddress_IP","destVideoTransportAddress_Port","destRSVPAudioStat","destRSVPVideoStat","dateTimeConnect","dateTimeDisconnect","lastRedirectDn","pkid","originalCalledPartyNumberPartition","callingPartyNumberPartition","finalCalledPartyNumberPartition","lastRedirectDnPartition","duration","origDeviceName","destDeviceName","origCallTerminationOnBehalfOf","destCallTerminationOnBehalfOf","origCalledPartyRedirectOnBehalfOf","lastRedirectRedirectOnBehalfOf","origCalledPartyRedirectReason","lastRedirectRedirectReason","destConversationId","globalCallId_ClusterID","joinOnBehalfOf","comment","authCodeDescription","authorizationLevel","clientMatterCode","origDTMFMethod","destDTMFMethod","callSecuredStatus","origConversationId","origMediaCap_Bandwidth","destMediaCap_Bandwidth","authorizationCodeValue","outpulsedCallingPartyNumber","outpulsedCalledPartyNumber","origIpv4v6Addr","destIpv4v6Addr","origVideoCap_Codec_Channel2","origVideoCap_Bandwidth_Channel2","origVideoCap_Resolution_Channel2","origVideoTransportAddress_IP_Channel2","origVideoTransportAddress_Port_Channel2","origVideoChannel_Role_Channel2","destVideoCap_Codec_Channel2","destVideoCap_Bandwidth_Channel2","destVideoCap_Resolution_Channel2","destVideoTransportAddress_IP_Channel2","destVideoTransportAddress_Port_Channel2","destVideoChannel_Role_Channel2","IncomingProtocolID","IncomingProtocolCallRef","OutgoingProtocolID","OutgoingProtocolCallRef","currentRoutingReason","origRoutingReason","lastRedirectingRoutingReason","huntPilotPartition","huntPilotDN","calledPartyPatternUsage","IncomingICID","IncomingOrigIOI","IncomingTermIOI","OutgoingICID","OutgoingOrigIOI","OutgoingTermIOI","outpulsedOriginalCalledPartyNumber","outpulsedLastRedirectingNumber","wasCallQueued","totalWaitTimeInQueue","callingPartyNumber_uri","originalCalledPartyNumber_uri","finalCalledPartyNumber_uri","lastRedirectDn_uri","mobileCallingPartyNumber","finalMobileCalledPartyNumber","origMobileDeviceName","destMobileDeviceName","origMobileCallDuration","destMobileCallDuration","mobileCallType","originalCalledPartyPattern","finalCalledPartyPattern","lastRedirectingPartyPattern","huntPilotPattern","origDeviceType","destDeviceType","origDeviceSessionID","destDeviceSessionID"] } compute_field { field => correlation_id value => "#{globalCallID_callId}" } }

output { # stdout {} hep { host => 127.0.0.1 port => 9060 hep_id => 18000 hep_type => 100 } } `

astrakid avatar Nov 20 '20 12:11 astrakid

Mind the host parameter should be HOMER's IP. Is HOMER running on the same host reading the CDRs? Sniff around port 9060 see if they actually show up.

lmangani avatar Nov 20 '20 12:11 lmangani

yes, homer is running locally. will check communication on port 9060

astrakid avatar Nov 20 '20 12:11 astrakid

lot of traffic from heplify-clients, but no traffic from my local server, where paStash for parsing CDRs and heplify-server (docker) are running. changed the config to host sipmon01.fqdn but no change...

astrakid avatar Nov 20 '20 12:11 astrakid

ok, got a step further. there is traffic from paStash to heplify-server (had to dump the docker interface), but all packets are failing:

14:11:50.762308 IP sipmon01.37059 > 172.18.0.3.9060: UDP, bad length 8151 > 1472 14:11:50.762525 IP sipmon01.37059 > 172.18.0.3.9060: UDP, bad length 8151 > 1472 14:11:50.762659 IP sipmon01.37059 > 172.18.0.3.9060: UDP, bad length 8151 > 1472 14:11:50.763065 IP sipmon01.37059 > 172.18.0.3.9060: UDP, bad length 8151 > 1472 [...]

astrakid avatar Nov 20 '20 13:11 astrakid

Try adding the following filter to generate a custom payload with a few elements of your CDR to test with:

 mustache {
    template => 'CDR: {{direction}} {{caller_id_number}} -> {{destination_number}}: {{lcr_carrier}} {{lcr_rate}}'
    target_field => 'payload'
  }

lmangani avatar Nov 20 '20 13:11 lmangani