tlsdump can decrypt SSL/TLS traffic?

[federicofantini]

Hi, today I was looking at this really interesting feature in the file: analyzer/windows/modules/auxiliary/tlsdump.py. At the end of the analysis I can download the pcap and log files containing respectively the whole network traffic and the dumped keys. CAPEv2 is able to decrypt the traffic? In this way the suricata signatures will be more effective!

I also tried to import these two files to wireshark in order to decrypt the traffic. This feature of wireshark is supported only for TLS <=1.2 versions but also in this legacy communication version I can’t see clear traffic. Maybe not all keys are collected? I did a test by analyzing my exe file that contacts (HTTP GET) to this URL: https://tls-v1-2.badssl.com:1012/

P.S. I know I'm not using templates but I don't know if these issues I'm opening are "bug" or "feature request" or something else... I hope it's not a problem.

Thanks in advance!

[github-actions[bot]]

@federicofantini: hello! :wave:

This issue is being automatically closed because it does not follow the issue template.

This is open source project! So please apreciate our time that we sacrify from other thing that we could enjoy, instead of asking boring things over and over.

[kevoreilly]

Yes indeed the tlsdump feature should mean that the encrypted traffic in the pcap gets decrypted and thus seen by suricata - you should therefore see the results of this in the network tab when you run your test exe - is this the case?

[federicofantini]

Yes, that would be my use case! So I intend to both see the decrypted traffic in the pcap in the network tab and apply suricata to the decrypted traffic to get a better vision. I thought it was already like this but actually reading the sources I can't find the point where the decryption is done, I only see the tlsdump.log file is saved and proposed in the gui.

Also, trying to decrypt the traffic manually from wireshark (need to convert the tlsdump.log file to the CLIENT_RANDOM supported format by wireshark), I can't see the decrypted traffic for TLSv1.2 connections.

[kevoreilly]

No I didn't mean "is this your use case?" - I meant "is it the case that your exe produces the expected output in the network tab?"!

[federicofantini]

Nope, the pcap still encrypted and I'm unable to decrypt with wireshark. The tabs in the network tab are:

• Hosts (7)
• DNS (8)
• TCP (18)
• UDP (33)
• HTTP (0)
• SMTP (0)
• IRC (0)
• ICMP (1)
• Suricata Alerts (0)
• Suricata TLS (0)
• Suricata HTTP (0)
• Suricata Files (0)

And my test exe is just a py2exe conversion of this script:

import requests
import time

urls = [
    "https://www.google.com",
    "https://www.apple.com",
    "https://www.aruba.it",
    "https://www.repubblica.it",
    "https://tls-v1-2.badssl.com:1012/",
    "https://tls-v1-1.badssl.com:1011/",
    "https://tls-v1-0.badssl.com:1010/"
]

for u in urls:
    r = requests.get(u)
    print(r.status_code)
    time.sleep(5)

I was looking at the public instance and I saw that this feature is already present and that it works on win7: https://capesandbox.com/analysis/374610/ (sorry for the misunderstanding). However I have a question: why some traffic is not decrypted on that analysis? I see some TCP communications on port 443.

[federicofantini]

Hi, I went a little bit deeper and I want to update you about this issue.

Thanks to this script https://github.com/lbirchler/tls-decryption I was able to see the decrypted version of the dump.pcap file by passing in input client_random/server_random bytes and the master_secret key. The decrypted traffic contains stuff about windows noisy telemetry and updates but nothing about my forged requests. I don't understand why this decrypted but noisy traffic, which contains some HTTP requests, is not visible on the Network HTTP(s) tab.

Futhermore in the journalctl of cape-processor I can see this skipped errors:

[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d[g?\xee(\xb3\x92\xbc\xd9\x9a\xc7*\xb1\x81KM\xb6*\xf1\xb1\xe1\xd0F\xb0\x93\x9f4+\x>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\\\x83=Vw\xb4\x91_\x89@L\xe8\xb2e\x9dCU\x96\x83\xe3\xcb\x8f\xb4\xc5\x04+;\xdaR', >
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d]\xb1\xef0\t\xe7K\xf2,3\xa1\xf15g\xa1\xd5\xa5AQ\xb6]0\xccm;W\x82|\xed', skipping >
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d`H\x9aT\x8609\\\x12\x0c0%?\x9e\xa3\xf6\xe5\xeb\xaeB{\xd8ua]\x0b\x81Q\xa1', skippi>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b"d\x14\x9d`\x8a\xcb\x0b\xbfzL\x82}\xe4\x91N\n\x15\xda\xd9V\x98\x98zaCr\x85\xe2\xb1'\x16\xc8>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b"d\x14\x9d`\xaf~]`\xd2\xd9b\x13z\x98\xdfc\\\x9ay\xd0\x90\x17m\x8b6\xb5\r'\xa8\xbb\xfd\x93",>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9dhXE\xd7\x85\x83k1\xb2a~\x0b\xf5I\x94\xcc\x89Y\xce\x84}\xde3e\x9b\xc5\xf9\x8fn', s>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9di\x85\xf6,f?\xd7\xfe\xb5\\Z\xc3\x96\x1e:\xabW\xe1\xe7\xd1\xa5?\xc5/\x85\x049\xb7\>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9diO\x87\xe8I \xc5\xe1\xb4\xe2N8\xf3^`Vu\x9a_A\xfb\xde\x1bU9<\xf9l\xe6', skipping it
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9dii\xb8G\x99\xa2S%\x84\xacA\x88\xbe\xaa\x99\x8f\\\x84\xc9\xa2\x8a\x8f\xec\xae\x7f;>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\x917T(1\xa1P\xab\xe3W\x88_ZoCOb\xee\xe5\x97\x0c8\xc4\xe8\\=\xee\xb0\xdd', skippi>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\x92=\xb3\xe9*6\xa3\x15\xa9\x13\x8ea\xf01a\xd2\x84\xf4\xff\x82x`G\xe5\x00Wr\xac\x>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\x92[\xba\xb5\x87\x1e\x17\xe5\xe0\xb7#=\x18\x05\x96L\xd0)z\xbf\x85m\xe0\x94\x97\x>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\x95\xd6\xa0g\xd3U\x8aM\xe8\xa1\xd3\xf6\x94\xear\x8fJ\x1e\x01\x9f\xe0[bF&>t\xd3\x>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\x9e\xd6^\x91\x1e\xf8\x92\x8a\x91\x19\xc2x\x92 \xc8M3(B\x9a\x1b\xf2\x7f\xb9\xbc>w>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\x9fwq\xa8U\xa3\x05v\x13\x9b6H\xdaW\xc7\xf5L\xda\xdf\xafx\x8fY\xf0\xbc\xcb\x82l\x>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\xa2&\xd2\xb9i\xbe\x97\x8b\xe0\xa0\xb7\x92\xce\xc1"\xecV\xb0J\xb9192<\xbc\x16\x80>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b"d\x14\x9d\xa2\x9dt\x1c\xb3\xad\xf8\xfc\xdc\xd6\xcb,\xb8'\x8d]\xae|\xb8\xec\xb5 \xc4\xef*=v>

I think this analysis https://capesandbox.com/analysis/374610/ may have something like mine for the TCP 443 requests.

[daothinh]

Hi @federicofantini, Can you show me the configuration of INETSIM for CAPE?, i can't find any instructions :(

[federicofantini]

@daothinh you need to install inetsim on a dedicated VM inside the libvirt network: https://www.techanarchy.net/installing-and-configuring-inetsim/ When the installation is completed insert the IP address of the machine here: https://capev2.readthedocs.io/en/latest/installation/host/routing.html?highlight=inetsim#inetsim-routing

[mvasilescu]

Yes indeed the tlsdump feature should mean that the encrypted traffic in the pcap gets decrypted and thus seen by suricata

@kevoreilly i'm not sure how this feature works - are there any specific CAPE configuration requirements for this to work? Does suricata have to work in socket mode ? Does it work in cli mode as well? I've looked through the documentation and haven't found anything related to this.

In my scenario, i have some HTTPs traffic (that i can see in the CAPE networking tab under HTTP(s) - but does not appear under Suricata HTTP tab or Suricata alerts - regardless of the suricata rules that i wrote)

[kevoreilly]

The HTTP(s) field is straight from the pcap whereas the suricata fields depend on suricata processing the pcap. So it sounds like the capture to pcap and decryption is fine, suricata side is not working.

[mvasilescu]

Hi @kevoreilly - appreciate you taking the time to respond.

I've also added some "test" rules for suricata for that pcap - and i see them in the report. I was not able however to get a suricata signature to trigger on HTTP(s) decrypted traffic. I saw your reply - and was wondering how is this actually done?

Yes indeed the tlsdump feature should mean that the encrypted traffic in the pcap gets decrypted and thus seen by suricata My understanding is that for a HTTPs request that is normally encrypted in a PCAP - it actually gets decrypted and passed to suricata. i don't understand how to get this working.

e.g. alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"test rule"; flow:established,from_client; http.method; content:"GET"; http.uri; content:"prof"; sid:99999997;) this will trigger on a HTTP request but not on a HTTPs decrypted request.

Is there anything i'm missing? Thanks!

[doomedraven]

we don't decrypt pcap. we decrypt them for us to show in cape in network data processing, if you need that to be working with suricata, you can write extension for cape to decrypt pcap and save updated pcap, as far as i know you just need to supply certificate, maybe even you can do that with suricata if they support that

[mvasilescu]

Hi @doomedraven - thanks for the update. That sounds more like what i thought it would be. Suricata does not do any decryption - if you want to run signatures on that traffic - you need to decrypt it beforehand. So yeah, probably the decrypt pcap extension sounds the best path for now.

[doomedraven]

i know that people has code for that, but they don't share code, i don't need this feature so i don't do it by myself. but if you want to contribute you are more than welcome to implement it

[kevoreilly]

I will be making a PR for this soon - watch this space.

[CarsonHrusovsky]

Nope, the pcap still encrypted and I'm unable to decrypt with wireshark. The tabs in the network tab are:

• Hosts (7)
• DNS (8)
• TCP (18)
• UDP (33)
• HTTP (0)
• SMTP (0)
• IRC (0)
• ICMP (1)
• Suricata Alerts (0)
• Suricata TLS (0)
• Suricata HTTP (0)
• Suricata Files (0)

And my test exe is just a py2exe conversion of this script:

import requests
import time

urls = [
    "https://www.google.com",
    "https://www.apple.com",
    "https://www.aruba.it",
    "https://www.repubblica.it",
    "https://tls-v1-2.badssl.com:1012/",
    "https://tls-v1-1.badssl.com:1011/",
    "https://tls-v1-0.badssl.com:1010/"
]

for u in urls:
    r = requests.get(u)
    print(r.status_code)
    time.sleep(5)

I was looking at the public instance and I saw that this feature is already present and that it works on win7: https://capesandbox.com/analysis/374610/ (sorry for the misunderstanding). However I have a question: why some traffic is not decrypted on that analysis? I see some TCP communications on port 443.

Hello did you ever get this to work with your setup? I see now that official CAPE doesn't contain any http or https traffic anymore - I hope this is still possible. Is it as simple as taking the tlsdump.log and using that to decrypt the pcap - then just feed that pcap to the mechanism responsible for populating the Network Analysis tab on the UI?

[doomedraven]

It works just fine on my private setup, I would think of public cape sandbox as demo version, if you have skills you can do black magic with code ;)

El 15 sept 2023, a las 21:47, CarsonHrusovsky @.***> escribió:

Nope, the pcap still encrypted and I'm unable to decrypt with wireshark. The tabs in the network tab are:

Hosts (7) DNS (8) TCP (18) UDP (33) HTTP (0) SMTP (0) IRC (0) ICMP (1) Suricata Alerts (0) Suricata TLS (0) Suricata HTTP (0) Suricata Files (0) And my test exe is just a py2exe conversion of this script:

import requests import time

urls = [ "https://www.google.com", "https://www.apple.com", "https://www.aruba.it", "https://www.repubblica.it", "https://tls-v1-2.badssl.com:1012/", "https://tls-v1-1.badssl.com:1011/", "https://tls-v1-0.badssl.com:1010/" ]

for u in urls: r = requests.get(u) print(r.status_code) time.sleep(5) I was looking at the public instance and I saw that this feature is already present and that it works on win7: https://capesandbox.com/analysis/374610/ (sorry for the misunderstanding). However I have a question: why some traffic is not decrypted on that analysis? I see some TCP communications on port 443.

Hello did you ever get this to work with your setup? I see now that official CAPE doesn't contain any http or https traffic anymore - I hope this is still possible. Is it as simple as taking the tlsdump.log and using that to decrypt the pcap - then just feed that pcap to the mechanism responsible for populating the Network Analysis tab on the UI?

— Reply to this email directly, view it on GitHub https://github.com/kevoreilly/CAPEv2/issues/1437#issuecomment-1721761411, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOFH3Y5XOMGUIANQFF2XB3X2SWFVANCNFSM6AAAAAAV36IAZ4. You are receiving this because you were mentioned.

[kevoreilly]

This is still very much a feature, nothing has been removed from 'official' cape!

[doomedraven]

Kev is due that VPN is down