JSON logging failing with error: jsonlog: Can't serialize

[devamtech]

Hi All,

need your help

I have deployed cowrie honeypot, I am getting logs in the below format, can you please me to get logs in json format instead of a string.

2021-05-18T14:26:24.763743919Z stdout F 2021-05-18T14:26:24+0000 [stdout#info] jsonlog: Can't serialize: '{'eventid': 'cowrie.login.success', 'username': b'root', 'password': b'kumar', 'message': "login attempt [b'root'/b'kumar'] succeeded", 'sensor': 'cowrie-app-667d44f77-kcdzb', 'timestamp': '2021-05-18T14:26:24.763361Z', 'src_ip': '192.0.2.1', 'session': '24221d408f3c'}'

one latest VC, I am not getting Src_ip and user name and password fields only getting direct user details and password, old cowries has above logs but its in string format. Can you please help me to get it in json format

Many thanks in advance.

[micheloosterhof]

The JSON log is enabled, but it's failing. Which version of Cowrie are you using? Do you use the Docker image, a git clone or a release download?

[devamtech]

thank you so much for your response,

I am using the below yaml file on K8s 1.21 to deploy Cowrie, Can you please tell me where to provide

apiVersion: v1
kind: Service
metadata:
  name: cowrie-lb
  labels:
    app: cowrie-app
spec:
  selector:
    app: cowrie-app
    tier: honeypot
  ports:
    - protocol: "TCP"
      port: 22
      targetPort: 2222
  type: LoadBalancer

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cowrie-app
  labels:
    app: cowrie-app
spec:
  replicas: 4
  selector:
    matchLabels:
      app: cowrie-app
      tier: honeypot
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: cowrie-app
        tier: honeypot
    spec:
      # runtimeClassName: gvisor   # ADD THIS LINE
      containers:
      - image: cowrie/cowrie:latest
        name: cowrie-app
        # env:
        # - name: WORDPRESS_DB_HOST
        #   value: cowrie-app-mysql
        # - name: WORDPRESS_DB_PASSWORD
        #   valueFrom:
        #     secretKeyRef:
        #       name: mysql-pass
        #       key: password
        ports:
        - containerPort: 2222
          name: cowrie-app
      #   volumeMounts:
      #   - name: cowrie-app-persistent-storage
      #     mountPath: /var/www/html
      # volumes:
      # - name: cowrie-app-persistent-storage
      #   persistentVolumeClaim:
      #     claimName: wp-pv-claim

[devamtech]

I am using Cowrie Version 2.2.0. I like this version because of below output but only problem is, getting this as a "String"

eventid': 'cowrie.login.success', 'username': b'root', 'password': b'kumar', 'message': "login attempt [b'root'/b'kumar'] succeeded", 'sensor': 'cowrie-app-667d44f77-kcdzb', 'timestamp': '2021-05-18T14:26:24.763361Z', 'src_ip': '192.0.2.1', 'session': '24221d408f3c'}'

[devamtech]

Hi team,

can you please help me on this issue, If I have json format, its easy to create kibana dashboard

[micheloosterhof]

I just checked the Docker container at hub.docker.com/cowrie/cowrie and that successfully logs to JSON. To get that on persistent storage, assign volumes for VOLUME [ "/cowrie/cowrie-git/var", "/cowrie/cowrie-git/etc" ] The var/log/cowrie/cowrie.json should contain the log in JSON format.

If you get persistent errors with JSON serialisation failing, please ensure all your components are up to date and please share which event is causing the serialisation failure.

[devamtech]

thank you so much Michel for your response.

I am using the cowrie 2.2 version and

Only for login even I am getting error

jsonlog: Can't serialize: '{'eventid': 'cowrie.login.success', 'username': b'root', 'password': b'deva', 'message': "login attempt [b'root'/b'deva'] succeeded", 'sensor': 'cowrie-app-5bcb76dd6b-8s7wn', 'timestamp': '2021-05-22T07:12:08.262918Z', 'src_ip': '10.144.0.2', 'session': '92fcfba6d9b2'}'

I have tested on multiple Kubernetes versions k8s 1.18 and K8s 1.19 and K8s 1.21, on environments, getting same above logs output.

on new cowrie, I am not seeing the error, but the problem is, In the new version, I am not getting src_ip and username and password fields like the above output, that's the reason I am using cowrie 2.2

new cowrie logs

2021-05-20T16:05:48+0000 [cowrie.ssh.factory.CowrieSSHFactory] New connection: 10.163.0.2:9054 (10.163.1.8:2222) [session: 7240a1ba5a5e]
2021-05-20T16:05:48+0000 [HoneyPotSSHTransport,0,10.163.0.2] Remote SSH version: b'SSH-2.0-OpenSSH_7.8'
2021-05-20T16:05:48+0000 [HoneyPotSSHTransport,0,10.163.0.2] SSH client hassh fingerprint: 93726f9c35049b4369578bad80d3a24a
2021-05-20T16:05:48+0000 [cowrie.ssh.transport.HoneyPotSSHTransport#debug] kex alg=b'ecdh-sha2-nistp256' key alg=b'ssh-rsa'
2021-05-20T16:05:48+0000 [cowrie.ssh.transport.HoneyPotSSHTransport#debug] outgoing: b'aes128-ctr' b'hmac-sha1' b'none'
2021-05-20T16:05:48+0000 [cowrie.ssh.transport.HoneyPotSSHTransport#debug] incoming: b'aes128-ctr' b'hmac-sha1' b'none'
2021-05-20T16:05:50+0000 [cowrie.ssh.transport.HoneyPotSSHTransport#debug] NEW KEYS
2021-05-20T16:05:50+0000 [cowrie.ssh.transport.HoneyPotSSHTransport#debug] starting service b'ssh-userauth'
2021-05-20T16:05:50+0000 [cowrie.ssh.userauth.HoneyPotSSHUserAuthServer#debug] b'root' trying auth b'none'
2021-05-20T16:05:52+0000 [cowrie.ssh.userauth.HoneyPotSSHUserAuthServer#debug] b'root' trying auth b'password'
2021-05-20T16:05:52+0000 [HoneyPotSSHTransport,0,10.163.0.2] Could not read etc/userdb.txt, default database activated
2021-05-20T16:05:52+0000 [HoneyPotSSHTransport,0,10.163.0.2] login attempt [b'root'/b'deva'] succeeded

I am looking for Old cowrie logs format. Can you please help Many thanks in advance.

[micheloosterhof]

I'm very confused by what version you are using. If you are running on k8s, you are using cowrie/cowrie:latest container from Docker Hub? Or did you create your own container from an old version?

Changing k8s versions is not going to change a single thing, it's only responsible for starting your container. Your root cause will be somewhere inside the container.

Cowrie has 2 log files. cowrie.log and cowrie.json. If you are looking at cowrie.log you will see the text based logs. Inside the .json file are the JSON logs. Exceptions/errors will go into the .log file, not in the .json files.

[devamtech]

10 days before cowrie/cowrie:latest was having a different image, the same image I pushed to my local repository and using the same

[devamtech]

can you please help me to get the below kind of output in cowrie/cowrie:latest

eventid': 'cowrie.login.success', 'username': b'root', 'password': b'deva', 'message': "login attempt [b'root'/b'deva'] succeeded", 'sensor': 'cowrie-app-5bcb76dd6b-8s7wn', 'timestamp': '2021-05-22T07:12:08.262918Z', 'src_ip': '10.144.0.2', 'session': '92fcfba6d9b2'}'

[micheloosterhof]

Asking the same question over and over will not give different results.

[micheloosterhof]

So I think you are not looking at the file output, you are looking at standard output from the container? That's not in JSON. that's text. The JSON log goes into a file on the file system. If you want to put it elsewhere, use an output plugin. There are elastic search output plugins that can send directly to Elasticsearch.

[devamtech]

The same format its coming in elastic search also, because the below logs is considering a string, if I use parsers and filter in fluent bit, its not working.

log": [ "2021-05-22T12:21:57.553505608Z stdout F 2021-05-22T12:21:57+0000 [stdout#info] jsonlog: Can't serialize: '{'eventid': 'cowrie.login.success', 'username': b'root', 'password': b'maatv', 'message': "login attempt [b'root'/b'maatv'] succeeded", 'sensor': 'cowrie-app-bcd99c59f-pdcz6', 'timestamp': '2021-05-22T12:21:57.553068Z', '@kibana-highlighted-field@src_ip@/kibana-highlighted-field@': '192.0.6.1', 'session': '44530278b1e6'}'" ]

[micheloosterhof]

Please share your configuration. What you are looking at is an error message. This is not the message you want to parse. It's the error message it can't write that data to JSON. Please share your exact configuration, how you are sending to Elastic, etc.

[devamtech]

Thank you Michel for your response.

This is my fluentbit Configuration file.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            10.98.41.40
        Port            9200
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        # http://rubular.com/r/tjUt3Awgg4
        Name cri
        Format regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

[micheloosterhof]

I just checked the latest docker container and JSON logging seems okay. I suspect your JSON log can't write its data because you haven't defined the volume. Disable the JSON logging, and enable the Elasticsearch output plugin instead.

You can use environment variables.

COWRIE_OUTPUT_JSONLOG_ENABLED=false
COWRIE_OUTPUT_ELASTICSEARCH_ENABLED=true
COWRIE_OUTPUT_ELASTICSEARCH_HOST=hostname

[alexey-kleymenov]

Hi @micheloosterhof, nice to meet you! I experience the same issue with broken authentication records using the cowrie/cowrie:latest image. The issue persists for at least 1 month (when I deployed it the first time) and up until now. Here is the command line I'm using:

docker run -p 22:2222 -p 23:2223 -e COWRIE_TELNET_ENABLED=yes -v /home/ubuntu/honeypot/logs:/cowrie/cowrie-git/var/log/cowrie/ -v /home/ubuntu/honeypot/samples:/cowrie/cowrie-git/var/lib/cowrie/downloads cowrie/cowrie

Here is an example stdout log snippet with 2 errors (the first and the last lines):

2021-06-11T10:51:33+0000 [stdout#info] jsonlog: Can't serialize: '{'eventid': 'cowrie.client.kex', 'hassh': '2f300334eb474e4d5ef932343447dd80', 'hasshAlgorithms': '[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1;aes128-ctr,aes192-ctr,aes256-ctr,[email protected],arcfour256,arcfour128,aes128-cbc,arcfour,3des-cbc;[email protected],hmac-sha2-256,hmac-sha1,hmac-sha1-96;none', 'kexAlgs': [b'[email protected]', b'ecdh-sha2-nistp256', b'ecdh-sha2-nistp384', b'ecdh-sha2-nistp521', b'diffie-hellman-group14-sha1', b'diffie-hellman-group1-sha1'], 'keyAlgs': [b'[email protected]', b'[email protected]', b'[email protected]', b'[email protected]', b'[email protected]', b'[email protected]', b'ecdsa-sha2-nistp256', b'ecdsa-sha2-nistp384', b'ecdsa-sha2-nistp521', b'ssh-rsa', b'ssh-dss', b'ssh-ed25519'], 'encCS': [b'aes128-ctr', b'aes192-ctr', b'aes256-ctr', b'[email protected]', b'arcfour256', b'arcfour128', b'aes128-cbc', b'arcfour', b'3des-cbc'], 'macCS': [b'[email protected]', b'hmac-sha2-256', b'hmac-sha1', b'hmac-sha1-96'], 'compCS': [b'none'], 'langCS': [b''], 'message': 'SSH client hassh fingerprint: 2f300334eb474e4d5ef932343447dd80', 'sensor': 'dda70ef8e80a', 'timestamp': '2021-06-11T10:51:33.923804Z', 'src_ip': '78.128.113.150', 'session': '79d7d0a46cee'}'
2021-06-11T10:51:33+0000 [cowrie.ssh.transport.HoneyPotSSHTransport#debug] kex alg=b'[email protected]' key alg=b'ssh-rsa'
2021-06-11T10:51:33+0000 [cowrie.ssh.transport.HoneyPotSSHTransport#debug] outgoing: b'aes128-ctr' b'hmac-sha1' b'none'
2021-06-11T10:51:33+0000 [cowrie.ssh.transport.HoneyPotSSHTransport#debug] incoming: b'aes128-ctr' b'hmac-sha1' b'none'
2021-06-11T10:51:34+0000 [cowrie.ssh.transport.HoneyPotSSHTransport#debug] NEW KEYS
2021-06-11T10:51:34+0000 [cowrie.ssh.transport.HoneyPotSSHTransport#debug] starting service b'ssh-userauth'
2021-06-11T10:51:34+0000 [cowrie.ssh.userauth.HoneyPotSSHUserAuthServer#debug] b'root' trying auth b'none'
2021-06-11T10:51:34+0000 [cowrie.ssh.userauth.HoneyPotSSHUserAuthServer#debug] b'root' trying auth b'password'
2021-06-11T10:51:34+0000 [HoneyPotSSHTransport,37,78.128.113.150] Could not read etc/userdb.txt, default database activated
2021-06-11T10:51:34+0000 [stdout#info] jsonlog: Can't serialize: '{'eventid': 'cowrie.login.success', 'username': b'root', 'password': b'admin', 'message': "login attempt [b'root'/b'admin'] succeeded", 'sensor': 'dda70ef8e80a', 'timestamp': '2021-06-11T10:51:34.241754Z', 'src_ip': '78.128.113.150', 'session': '79d7d0a46cee'}'

And here are a few of the many broken JSON entries in the cowrie.json. As you can see, they don't follow the JSON format and the username and password values are not available:

{"eventid":"cowrie.client.kex","hassh":"2f300334eb474e4d5ef932343447dd80","hasshAlgorithms":"[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1;aes128-ctr,aes192-ctr,aes256-ctr,[email protected],arcfour256,arcfour128,aes128-cbc,arcfour,3des-cbc;[email protected],hmac-sha2-256,hmac-sha1,hmac-sha1-96;none","kexAlgs":[{"eventid":"cowrie.login.success","username":{"eventid":"cowrie.direct-tcpip.request","dst_ip":"ya.ru","dst_port":80,"src_ip":"92.118.36.4","src_port":0,"message":"direct-tcp connection request to ya.ru:80 from 0.0.0.0:0","sensor":"dda70ef8e80a","timestamp":"2021-06-11T10:43:02.863361Z","session":"fc8657d24108"}

{"eventid":"cowrie.login.success","username":{"eventid":"cowrie.session.params","arch":"linux-x64-lsb","message":[],"sensor":"05944c27d909","timestamp":"2021-06-11T00:00:17.447110Z","src_ip":"60.171.154.30","session":"bf66dfc101ed"}

[micheloosterhof]

This is really strange, like it's mixing 2 events together. In both cases it's the username field containing another event. @alexey-kleymenov your issue is not the same as the previous one. the ticket opener does not have malformed JSON entries but issues writing to a file. Could you open a new ticket for this, please?

[alexey-kleymenov]

@micheloosterhof, sure! Here we go: https://github.com/cowrie/docker-cowrie/issues/28. Originally, I put it here because I saw that the topic starter had the same jsonlog serialization error message in logs when parsing cowrie.login.success records.

[micheloosterhof]

I just checked the latest docker container and JSON logging seems okay. I suspect your JSON log can't write its data because you haven't defined the volume. Disable the JSON logging, and enable the Elasticsearch output plugin instead.

You can use environment variables.

COWRIE_OUTPUT_JSONLOG_ENABLED=false
COWRIE_OUTPUT_ELASTICSEARCH_ENABLED=true
COWRIE_OUTPUT_ELASTICSEARCH_HOST=hostname

@devamtech are you still experiencing issues or did this work for you?