Parse TLS `ClientHello` messages to obtain more meaningful domain names

[GyulyVGC]

So far Sniffnet has always retrieved domain names simply by performing reverse DNS lookups. This is problematic because most of the times servers involved in the network connections are behind CDNs that are owned by third party providers, and performing a rDNS will reveal the name of that provider instead the name of the server we're receiving data from.

To partially fix this, we can parse the SNI (Server Name Indication) that is a TLS extension reporting the name of the original server. Such extension can be found in ClientHello messages.

When this particular extension isn't available or a connection isn't using TLS, we can fallback to use reverse DNS.

Implementing this feature doesn't only require parsing the SNI, but also needs to come up with a new strategy to map IP addresses to network hosts: it will no longer be possible to associate an IP to a unique name, since behind the same IP there could be multiple servers with different names.

[ignoramous]

also needs to come up with a new strategy to map IP addresses to network hosts

We do this for Rethink DNS + Firewall (an Android app) by running a stub DNS resolver that responds with unique but pseudo IPs (from RFC6598 & RFC8215) for A/AAAA/HTTPS/SVCB queries. There's also a knob for users to firewall / block all other IPs (ie, IPs not generated by our stub resolver). It turns out, this was a source of never ending edge-cases and bugs... super, super painful (:

[it-lgtm]

I guess this could be taken as an inspiration to how this parsing could be implemented. As far as I can say, looking at the code, the tls.sni parsing take place as follows: the matcher matches for tls part, then, when the tls handshake proceeds (when the tls state reaches to client hello state), the related server name is retrieved from the buffer. Quite straightforward.

To help you, I considered contacting the original maintainer of the code, but decided that it isn't necessary(the lines were written possibly ~18 years ago).

Below are some excerpts I gathered by chatting with Gemini:

Q1: From a high level, what is the goal of the tls.sni keyword's registration?

The primary goal is performance. Suricata processes enormous amounts of traffic and must be incredibly efficient. The tls.sni keyword is registered in a way that supports Suricata's two-phase detection strategy: a very fast initial check (pre-filtering) followed by a more detailed inspection only when necessary. This ensures that the engine doesn't waste time on expensive analysis for rules that clearly won't match.

Q2: How does this two-phase detection work, and why is the tls.sni registration split between two functions?

The two phases are handled by two different parts of the engine, which is why the tls.sni keyword must be registered with both:

1. Phase 1: Prefiltering (The Fast Check): This is handled by the Multi-Pattern Matcher (MPM) engine. The DetectAppLayerMpmRegister function gives the MPM engine the ability to get the tls.sni buffer. The MPM quickly scans this buffer for simple strings from content keywords. If no match is found, the rule is dropped for that packet.
2. Phase 2: Full Inspection (The Detailed Check): If the pre-filter finds a potential match, the rule is passed to the main inspection engine. The DetectAppLayerInspectEngineRegister function tells this engine how to get the tls.sni buffer so it can perform a full, detailed evaluation of every keyword in the rule (e.g., pcre, distance, etc.).

This split ensures the expensive full inspection only happens on a small, pre-qualified subset of traffic, making the whole system much faster.

Q3: What does the MPM engine actually look for in the tls.sni buffer?

The MPM engine looks for the value of the content keyword that is used alongside tls.sni in a rule.

Consider the rule: alert tls any any -> ... (tls.sni; content:"evil.com"; ...)

• The tls.sni keyword tells the engine which buffer to inspect.
• The content:"evil.com" keyword provides the pattern that the MPM engine will search for within that buffer.

The MPM is not aware of the tls.sni keyword itself; it only knows about the pattern "evil.com" and that it needs to look for it in the buffer that tls.sni provides.

Q4: What is the role of the DetectTlsSniSetup function? How is it different from the Register functions?

The Register functions are global and run once at startup, while the Setup function is local to each rule.

• Register (Global): Tells the entire Suricata engine that the tls.sni keyword exists and what function (GetData) to call to retrieve its data.
• Setup (Per-Rule): Runs for every single rule that uses the tls.sni keyword. The DetectTlsSniSetup function configures that specific rule by:
  1. Activating the Buffer: Officially linking the rule to the tls.sni buffer, telling the engine that this rule requires that specific piece of data for evaluation.
  2. Sets the Protocol: Enforcing that the rule only runs on TLS traffic (ALPROTO_TLS), which is a critical optimization.

Q5: How does the GetData function work? Does it have to re-extract the SNI every time it's called?

No, it's more efficient than that. The GetData function uses a per-transaction cache.

• The first time it's called for a specific TLS handshake (e.g., by the MPM engine), it does the work of extracting the SNI from the flow's state and stores it in a special InspectionBuffer.
• On any subsequent call for that same handshake (e.g., by the full inspection engine), it finds the buffer is already populated and returns the cached data instantly, avoiding redundant work.

This ensures that the SNI is extracted at most once per transaction, even though it's used by multiple parts of the detection engine.

Now, moving from a specific keyword (tls.sni) to the broader mechanism of protocol detection. The "exact matching" for the TLS handshake itself isn't done by a single keyword, but by the TLS application-layer parser. This parser is a state machine that analyzes the raw TCP stream to identify and decode each part of the TLS handshake.

The keywords we've discussed (like tls.sni) are consumers of the data that this parser produces. Let's trace how this works.

The Code Path for TLS Handshake Detection

1. Registering the TLS Parser

   The process starts with registering the TLS parser itself. This tells Suricata that it knows how to understand TLS. After reviewing src/app-layer-ssl.c, the key registration happens in the RegisterSSLParsers function.

   // https://github.com/OISF/suricata/blob/master/src/app-layer-ssl.c
   3219 |     if (SCAppLayerParserConfParserEnabled("tcp", proto_name)) {
   3220 |         AppLayerParserRegisterParser(IPPROTO_TCP, ALPROTO_TLS, STREAM_TOSERVER,
   3221 |                                      SSLParseClientRecord);
   3222 | 
   3223 |         AppLayerParserRegisterParser(IPPROTO_TCP, ALPROTO_TLS, STREAM_TOCLIENT,
   3224 |                                      SSLParseServerRecord);
   

   These lines tell the stream engine: "For traffic identified as TLS (ALPROTO_TLS), direct the client-to-server stream to SSLParseClientRecord and the server-to-client stream to SSLParseServerRecord."

2. The TLS Parser State Machine (SSLDecode)

   Both SSLParseClientRecord and SSLParseServerRecord are wrappers around the main parsing function, SSLDecode. This function is the heart of the TLS handshake detection. It's a state machine that consumes bytes from the TCP stream and tries to match them against the expected structure of a TLS handshake.

   It contains a large while (input_len > 0) loop (line 2697) that processes records. Inside, it determines whether the record is SSLv2 or SSLv3/TLS and calls the appropriate record parser (SSLv2Decode or SSLv3Decode).

3. Parsing Handshake Messages (SSLv3ParseHandshakeProtocol)

   Within SSLv3Decode, once a handshake record (SSLV3_HANDSHAKE_PROTOCOL) is identified, it calls SSLv3ParseHandshakeProtocol. This function contains a switch statement (line 1722) that handles the different handshake message types, such as:

   • SSLV3_HS_CLIENT_HELLO
   • SSLV3_HS_SERVER_HELLO
   • SSLV3_HS_CERTIFICATE

   As it successfully parses each component, it populates an SSLState struct with the extracted data (version, cipher suites, certificates, and of course, the SNI).

4. Generating Application-Layer Events

   Crucially, as the parser moves from one state to another, it updates the state of the connection. These state changes act as application-layer events, announcing that a specific milestone in the protocol handshake has been reached.

   For example, after successfully parsing the entire Client Hello message in TLSDecodeHandshakeHello, it calls:

   // src/app-layer-ssl.c
   1546 |     if (ssl_state->curr_connp == &ssl_state->client_connp) {
   1547 |         UpdateClientState(ssl_state, TLS_STATE_CLIENT_HELLO_DONE);
   1548 |     }
   

5. Triggering the Detection Keywords

   This is where it all connects back to our keywords. The registration for tls.sni included this parameter: TLS_STATE_CLIENT_HELLO_DONE.

   // src/detect-tls-sni.c
   77 |     DetectAppLayerInspectEngineRegister("tls.sni", ALPROTO_TLS, SIG_FLAG_TOSERVER,
   78 |             TLS_STATE_CLIENT_HELLO_DONE, DetectEngineInspectBufferGeneric, GetData);
   

   This tells the detection engine: "Do not even attempt to run the tls.sni logic until the TLS parser has updated the state to TLS_STATE_CLIENT_HELLO_DONE."

Summary of the "Exact Match" Flow

The "exact match" for the TLS handshake is a multi-step process that enables keyword-based matching:

1. TCP Stream Assembly: The stream engine pieces together TCP packets into an ordered data stream.
2. Parser Invocation: The stream is fed to the registered TLS parser (SSLDecode).
3. Stateful Parsing: The parser walks through the stream, byte by byte, validating that it conforms to the TLS handshake protocol specification. This is the "exact match" of the protocol itself.
4. Data Extraction: As it parses, it extracts values like the SNI and stores them in the flow's SSLState.
5. State Update (Event): Upon successfully parsing a major part of the handshake, the parser updates its state (e.g., to TLS_STATE_CLIENT_HELLO_DONE).
6. Keyword Triggering: The detection engine sees this state change and says, "Now I will run all registered keywords that are waiting for this state," which includes tls.sni.
7. Keyword Execution: The tls.sni keyword's GetData function is finally called, which can now confidently read the SNI value from the SSLState because the parser has already done the hard work of finding, extracting, and validating it.

[GyulyVGC]

Thanks for your feedback @it-lgtm I'm not too worried about parsing the SNI itself... elements that are more concerning are:

1. trying to detect ClientHello messages will inevitably require to do some checks on every single packet, decreasing processing speed
2. right now Sniffnet sets a snaplen of 256 bytes since it only processes packet headers, but to read the SNI we'll need to remove the snaplen, capturing the whole packets (which could further decrease performance)
3. I read somewhere that the ClientHello could be split in multiple different packets in cases where it's too long... this would require reassembling flows... but I'd like to avoid this, to keep packet analysis an "on-the-fly" mechanism... so I believe I'll only support ClientHello that aren't fragmented
4. ClientHellos aren't only in TCP packets over TLS (typically HTTPS) but also in UDP packets using QUIC, so there's the need to also parse those
5. other protocols like POP over TLS (POPS), IMAP over TLS (IMAPS), etc, have different structures that require different parsing algorithm to detect the SNI... but probably I'll starts just by supporting HTTPS & QUIC
6. this discussion is only limited to TLS-based protocols, but other protocols like unencrypted HTTP carry the server name in plaintext and require once again a different parsing algorithm... but this will probably be discussed in separate issue

[it-lgtm]

• number 1 That's why suricata uses 2 step checking: a quick filtering step (mpm) and then a precise matching step.
• The points 2 and 3 point to considerations about buffer and stream processing. When we have a stream parser that changes its states, i.e. a state machine, the parsing will be abstracted from however messages it comes in. The ClientHello coming in fragments shouldn't matter, since the parsing will be abstracted from the lower level packets into a stream.
• number 4 I witnessed that suricata matching sometimes doesn't work, this could be because they don't yet support Quic, but still their codebase seems a nice inspiration point, handling many packets, efficiently, with a high level of testing/acceptance standard.
• As you say in number 5, we could start by supporting the https-based flows. Hence, we could try to reduce the initial task further, to facilitate next steps. Hence, here's my two cents to conceptually bring together an initial starter task:
  • Let's start by gathering several examples/test cases: Creating a simple test cases/pcaps for several different usages, including reducing the buffer size and forcing the clienthello be split into several packets.
  • Then, creating three simple parsers;
    • one is for tls state matching,
    • another is for quick filtering(optional) and
    • the other as a precise tns.sni matcher.

[GyulyVGC]

The way I would go with this is to simply check individual packets to see if they contain any form of ClientHello (be it TCP or QUIC); if a packet does contain it, I'd just parse it to find the SNI and extract the server name; finally, I'd associate that flow with the specific named server. This is why I say that fragmented messages would be problematic.

I'm ok with starting by just supporting TCP + UDP HTTPS. And I really like the idea of using PCAPs to test the behaviour — Sniffnet now natively supports importing them. Concerning the quick filter, I'd skip it for now.