groups and formats are lost during tls session ticket

Open fkasler opened this issue 1 year ago • 24 comments

Per the original JA3 project README:

"We also needed to introduce some code to account for Google’s GREASE (Generate Random Extensions And Sustain Extensibility) as described here. Google uses this as a mechanism to prevent extensibility failures in the TLS ecosystem. JA3 ignores these values completely to ensure that programs utilizing GREASE can still be identified with a single JA3 hash."

Original implementation: GREASE_TABLE = {0x0a0a: True, 0x1a1a: True, 0x2a2a: True, 0x3a3a: True, 0x4a4a: True, 0x5a5a: True, 0x6a6a: True, 0x7a7a: True, 0x8a8a: True, 0x9a9a: True, 0xaaaa: True, 0xbaba: True, 0xcaca: True, 0xdada: True, 0xeaea: True, 0xfafa: True}

GREASE_TABLE Ref: https://tools.ietf.org/html/draft-davidben-tls-grease-00

Here are some sample different JA3 for Chrome using the example server in the README:

user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36 ja3: 772,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513,29-23-24,0 ja3_hash: 59686f806cae30344b525e99af5b655d ssl_greased: 1

user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36 ja3: 772,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513 ja3_hash: 217b989439002f3b88747b7b038532d9 ssl_greased: 1

Jul 23 '22 02:07 fkasler

@fkasler Can you help make a request for reproducing? Possible collect a packet via wireshark or tcpdump?

Jul 24 '22 13:07 deancn

In logs on server i have this:

771,49172-53-49171-47-49196-49195-49200-157-49199-156,10-11-13-23-0-65281 771,49172-53-49171-47-49196-49195-49200-157-49199-156,10-11-13-23-0-65281,23-24-25-9-10-11-12-13-14-22,0

when http_ssl_ja3 == 0

Jul 27 '22 09:07 paragor

should be related with tls session ticket issue, we're working on it.

Jul 27 '22 09:07 phuslu

I tried caching ja3 at the connection level in nginx module, but it does not work: ja3 changed only after reconnect tcp session.

and when I tried to cache it, I came across another bug:
nginx-ssl-fingerprint return

772,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513,29-48640-48830,0

but https://github.com/open-ch/ja3 return

771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513,29-23-24,0

so ssl version detection is not correct - nginx module say = 772 (equal TLSv1.3), but in logs i see SSL: TLSv1.2 and in wireshark TLSv1.2 on client hello. full log:

2022/07/28 00:28:21 [debug] 3455249#0: accept on 0.0.0.0:443, ready: 0
2022/07/28 00:28:21 [debug] 3455249#0: posix_memalign: 0000616000003380:512 @16
2022/07/28 00:28:21 [debug] 3455249#0: *3 accept: 185.57.31.13:59601 fd:3
2022/07/28 00:28:21 [debug] 3455249#0: *3 event timer add: 3: 60000:19407440384
2022/07/28 00:28:21 [debug] 3455249#0: *3 reusable connection: 1
2022/07/28 00:28:21 [debug] 3455249#0: *3 epoll add event: fd:3 op:1 ev:80002001
2022/07/28 00:28:21 [debug] 3455249#0: timer delta: 116
2022/07/28 00:28:21 [debug] 3455249#0: worker cycle
2022/07/28 00:28:21 [debug] 3455249#0: epoll timer: 60000
2022/07/28 00:28:21 [debug] 3455249#0: epoll: fd:3 ev:0001 d:00007F309B3C3AD1
2022/07/28 00:28:21 [debug] 3455249#0: *3 http check ssl handshake
2022/07/28 00:28:21 [debug] 3455249#0: *3 http recv(): 1
2022/07/28 00:28:21 [debug] 3455249#0: *3 https ssl handshake: 0x16
2022/07/28 00:28:21 [debug] 3455249#0: *3 tcp_nodelay
2022/07/28 00:28:21 [debug] 3455249#0: *3 reusable connection: 0
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL server name: "p1.paragor.ru"
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL ALPN supported by client: h2
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL ALPN supported by client: http/1.1
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL ALPN selected: http/1.1
2022/07/28 00:28:21 [debug] 3455249#0: *3 posix_memalign: 0000616000003C80:512 @16
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL_do_handshake: -1
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL_get_error: 2
2022/07/28 00:28:21 [debug] 3455249#0: timer delta: 4
2022/07/28 00:28:21 [debug] 3455249#0: worker cycle
2022/07/28 00:28:21 [debug] 3455249#0: epoll timer: 59996
2022/07/28 00:28:21 [debug] 3455249#0: epoll: fd:3 ev:0001 d:00007F309B3C3AD1
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL handshake handler: 0
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL_do_handshake: 1
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL: TLSv1.2, cipher: "ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESGCM(128) Mac=AEAD"
2022/07/28 00:28:21 [debug] 3455249#0: *3 reusable connection: 1
2022/07/28 00:28:21 [debug] 3455249#0: *3 http wait request handler
2022/07/28 00:28:21 [debug] 3455249#0: *3 malloc: 0000619000009680:1024
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL_read: 717
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL_read: -1
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL_get_error: 2
2022/07/28 00:28:21 [debug] 3455249#0: *3 reusable connection: 0
2022/07/28 00:28:21 [debug] 3455249#0: *3 posix_memalign: 000062100001B900:4096 @16
2022/07/28 00:28:21 [debug] 3455249#0: *3 http process request line
2022/07/28 00:28:21 [debug] 3455249#0: *3 http request line: "GET / HTTP/1.1"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http uri: "/"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http args: ""
2022/07/28 00:28:21 [debug] 3455249#0: *3 http exten: ""
2022/07/28 00:28:21 [debug] 3455249#0: *3 posix_memalign: 000062100001CD00:4096 @16
2022/07/28 00:28:21 [debug] 3455249#0: *3 http process request header line
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "Host: p1.paragor.ru"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "Connection: keep-alive"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "Cache-Control: max-age=0"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103""
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "sec-ch-ua-mobile: ?0"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "sec-ch-ua-platform: "macOS""
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "DNT: 1"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "Upgrade-Insecure-Requests: 1"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "Sec-Fetch-Site: none"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "Sec-Fetch-Mode: navigate"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "Sec-Fetch-User: ?1"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "Sec-Fetch-Dest: document"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "Accept-Encoding: gzip, deflate, br"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header: "Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http header done
2022/07/28 00:28:21 [debug] 3455249#0: *3 event timer del: 3: 19407440384
2022/07/28 00:28:21 [debug] 3455249#0: *3 rewrite phase: 0
2022/07/28 00:28:21 [debug] 3455249#0: *3 test location: "/"
2022/07/28 00:28:21 [debug] 3455249#0: *3 using configuration "=/"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http cl:-1 max:1048576
2022/07/28 00:28:21 [debug] 3455249#0: *3 rewrite phase: 2
2022/07/28 00:28:21 [debug] 3455249#0: *3 http set discard body
2022/07/28 00:28:21 [debug] 3455249#0: *3 ssl fingerprint alloc bytes: [342]

2022/07/28 00:28:21 [debug] 3455249#0: *3 ssl fingerprint fixup for tls1.3
2022/07/28 00:28:21 [debug] 3455249#0: *3 ssl fingerprint: [772,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513,29-48640-48830,0]

2022/07/28 00:28:21 [debug] 3455249#0: *3 ssl fingerprint put to cache[146]

2022/07/28 00:28:21 [debug] 3455249#0: *3 ssl fingerprint alloc bytes: [342]

2022/07/28 00:28:21 [debug] 3455249#0: *3 ssl fingerprint take from cache[146]

2022/07/28 00:28:21 [debug] 3455249#0: *3 http script copy: "user_agent:	"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http script var: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http script copy: "
ja3:	"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http script var: "772,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513,29-48640-48830,0"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http script copy: "
ja3_hash:	"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http script var: "47babddf303795311778fbdcd6b6e8fa"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http script copy: "
ssl_greased: "
2022/07/28 00:28:21 [debug] 3455249#0: *3 http script var: "1"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http script copy: "
"
2022/07/28 00:28:21 [debug] 3455249#0: *3 HTTP/1.1 200 OK
Server: nginx/1.18.0
Date: Wed, 27 Jul 2022 14:28:21 GMT
Content-Type: text/plain
Content-Length: 340
Connection: keep-alive

2022/07/28 00:28:21 [debug] 3455249#0: *3 write new buf t:1 f:0 000062100001D430, pos 000062100001D430, size: 149 file: 0, size: 0
2022/07/28 00:28:21 [debug] 3455249#0: *3 http write filter: l:0 f:0 s:149
2022/07/28 00:28:21 [debug] 3455249#0: *3 http output filter "/?"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http copy filter: "/?"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http postpone filter "/?" 00007FFECE6071C0
2022/07/28 00:28:21 [debug] 3455249#0: *3 write old buf t:1 f:0 000062100001D430, pos 000062100001D430, size: 149 file: 0, size: 0
2022/07/28 00:28:21 [debug] 3455249#0: *3 write new buf t:0 f:0 0000000000000000, pos 000062100001D236, size: 340 file: 0, size: 0
2022/07/28 00:28:21 [debug] 3455249#0: *3 http write filter: l:1 f:0 s:489
2022/07/28 00:28:21 [debug] 3455249#0: *3 http write filter limit 0
2022/07/28 00:28:21 [debug] 3455249#0: *3 malloc: 00006290000AA200:16384
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL buf copy: 149
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL buf copy: 340
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL to write: 489
2022/07/28 00:28:21 [debug] 3455249#0: *3 SSL_write: 489
2022/07/28 00:28:21 [debug] 3455249#0: *3 http write filter 0000000000000000
2022/07/28 00:28:21 [debug] 3455249#0: *3 http copy filter: 0 "/?"
2022/07/28 00:28:21 [debug] 3455249#0: *3 http finalize request: 0, "/?" a:1, c:1
2022/07/28 00:28:21 [debug] 3455249#0: *3 set http keepalive handler
2022/07/28 00:28:21 [debug] 3455249#0: *3 http close request

mb fixup for tls1.3 work incorrect?

P.S. 2022/07/28 00:28:21 [debug] 3455249#0: *3 ssl fingerprint put to cache[146] and 2022/07/28 00:28:21 [debug] 3455249#0: *3 ssl fingerprint take from cache[146] - just my debug strs

Jul 27 '22 14:07 paragor

hard to fix in nginx layer, I'm re-writing patches and migrate ja3 logic into openssl layer.

see https://github.com/phuslu/nginx-ssl-fingerprint/tree/re/patches

Jul 27 '22 16:07 phuslu

should fixed in master branch. please take a look @fkasler @paragor

Jul 28 '22 15:07 phuslu

@phuslu i test localy - groups and formats present after reconnect - so origin bug is fixed.

But - there are some difference from https://github.com/open-ch/ja3:

772,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,65281-0-11-10-35-5-16-18-23-13-43-45-51-27-17513,29-23-24,0 #phuslu/nginx-ssl-fingerprint
771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513,29-23-24,0 #open-ch/ja3
771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513-,29-23-24,0 #wireshark

first difference is ssl version (i think it is due dirty hack for tls1.3) second difference is order of groups 65281-0-11-10-35-5-16-18-23-13-43-45-51-27-17513 vs 0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513

and now i wanna test this version on prod.

Jul 29 '22 00:07 paragor

attach pcap file pcap.pcap.zip

Jul 29 '22 00:07 paragor

1st is intended currently, but still in discussion and may change in future. 2nd seems a bug, will look today.

Jul 29 '22 01:07 phuslu

@phuslu on production catch segfault :(

nginx[184475]: segfault at 78 ip 000055b0f3d54ea4 sp 00007ffc732f71c8 error 4 in nginx[55b0f3c85000+2aa000]

Jul 29 '22 02:07 paragor

2022/07/29 12:54:19 [debug] 237797#237797: *1558 SSL ALPN supported by client: h2
2022/07/29 12:54:19 [debug] 237797#237797: *1558 SSL ALPN supported by client: http/1.1
2022/07/29 12:54:19 [debug] 237797#237797: *1558 SSL ALPN selected: h2
2022/07/29 12:54:19 [debug] 237797#237797: *1558 SSL_do_handshake: -1
2022/07/29 12:54:19 [debug] 237797#237797: *1558 SSL_client_hello_get_ja3_str: [771,49200-49196-49192-49188-49172-49162-159-107-57-52393-52392-52394-65413-196-136-129-157-61-53-192-132-49199-49195-49191-49187-49171-49161-158-103-51-190-69-156-60-47-186-65-49169-49159-5-4-49170-49160-22-10-255,0-11-10-16-13,29-23-24,0]
2022/07/29 12:54:19 [debug] 237797#237797: *1558 SSL_get_error: 2
2022/07/29 12:54:19 [debug] 242191#242191: epoll add event: fd:7 op:1 ev:10000001

and then segfault

Jul 29 '22 02:07 paragor

Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000558e679ddf54 in ngx_http_ssl_fingerprint_hash (r=0x558e6bba1050, v=0x558e6f947220, data=0) at /usr/src/ja3-phuslu-r2/nginx-ssl-fingerprint/src/ngx_http_ssl_fingerprint_module.c:86

warning: Source file is more recent than executable. 86 v->data = r->connection->ssl->fp_ja3_md5.data;

Jul 29 '22 03:07 paragor

refactor patches in master (should be more performant), now the extensions difference is fixed. for the crash/segfault issue, let me add some guard code to fix it.

@paragor thanks for your testing and report, would you please give a local test again?

Jul 29 '22 18:07 phuslu

local test: pcap.pcap.zip

first request in pcap is curl

771,49200-49196-49192-49188-49172-49162-159-107-57-52393-52392-52394-65413-196-136-129-157-61-53-192-132-49199-49195-49191-49187-49171-49161-158-103-51-190-69-156-60-47-186-65-49169-49159-5-4-49170-49160-22-10-255,0-11-10-13-16,29-23-24,0 # phuslu/nginx-ssl-fingerprint
771,49200-49196-49192-49188-49172-49162-159-107-57-52393-52392-52394-65413-196-136-129-157-61-53-192-132-49199-49195-49191-49187-49171-49161-158-103-51-190-69-156-60-47-186-65-49169-49159-5-4-49170-49160-22-10-255,0-11-10-13-16,29-23-24,0 # open-ch/ja3
771,49200-49196-49192-49188-49172-49162-159-107-57-52393-52392-52394-65413-196-136-129-157-61-53-192-132-49199-49195-49191-49187-49171-49161-158-103-51-190-69-156-60-47-186-65-49169-49159-5-4-49170-49160-22-10-255,0-11-10-13-16,29-23-24,0 # wireshark

result: all ok

next requests are chrome macos

772,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513-21,29-23-24,0 # phuslu/nginx-ssl-fingerprint
771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513-21,29-23-24,0 # open-ch/ja3
771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513-21,29-23-24,0 # wireshark

result: order is fixed, but ssl version is wrong

groups and formats always presents

tests on production load can perform only monday

Jul 30 '22 02:07 paragor

Aligened internally and disabled the "tls1.3 fixup" in nginx patch. Now the version should be "ok"

Personally i don't think it is "wrong" version but we want re-use existing ja3 databases so seems that we have to drop this fixup

Jul 30 '22 06:07 phuslu

now localy - ja3 provided by nginx-ssl-fingerprint mathes ja3 provided by open-ch/ja3 and wireshark

so, i suggest waiting for production tests on Monday

Jul 30 '22 07:07 paragor

and i wanna play with https://github.com/tlsfuzzer/tlsfuzzer to to make sure there is no SIGSEGV

Jul 30 '22 07:07 paragor

Thanks and no need rush, i still need to add some guard code, like https://github.com/phuslu/nginx-ssl-fingerprint/commit/06fb0a65ee813c401532573b04d108052cde37cf

Jul 30 '22 08:07 phuslu

I think it's ready for production test, our side will also do it. thanks again.

currently we calculate ja3 string&hash in nginx handshake workflow not in nginx module, it may decrease qps or increase latency, but seems better than "repeated calculation" issue in nginx module. Not very sure, let us see the production results.

Jul 31 '22 07:07 phuslu

SEGV when curl http port

    server {
        listen                 0.0.0.0:8888;
        default_type           "application/json";
        return                 200 '{\n  "ua": "$http_user_agent",\n  "ja3": "$http_ssl_ja3",\n  "ja3_hash": "$http_ssl_ja3_hash",\n  "greased": $http_ssl_greased\n}';
    }

AddressSanitizer:DEADLYSIGNAL
=================================================================
==24==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000078 (pc 0x55b3156c552e bp 0x7fff06d52a60 sp 0x7fff06d52a60 T0)
==24==The signal is caused by a READ memory access.
==24==Hint: address points to the zero page.
    #0 0x55b3156c552e in ngx_http_ssl_fingerprint /build/nginx/../nginx-ssl-fingerprint/src/ngx_http_ssl_fingerprint_module.c:68
    #1 0x55b3155a1ce6 in ngx_http_get_indexed_variable src/http/ngx_http_variables.c:631
    #2 0x55b3155a5711 in ngx_http_script_copy_var_len_code src/http/ngx_http_script.c:944
    #3 0x55b3155a632a in ngx_http_complex_value src/http/ngx_http_script.c:83
    #4 0x55b31556caa0 in ngx_http_send_response src/http/ngx_http_core_module.c:1769
    #5 0x55b3155abfa8 in ngx_http_script_return_code src/http/ngx_http_script.c:1499
    #6 0x55b31562c46a in ngx_http_rewrite_handler src/http/modules/ngx_http_rewrite_module.c:180
    #7 0x55b315569ea1 in ngx_http_core_rewrite_phase src/http/ngx_http_core_module.c:929
    #8 0x55b31555bfb1 in ngx_http_core_run_phases src/http/ngx_http_core_module.c:875
    #9 0x55b31555c196 in ngx_http_handler src/http/ngx_http_core_module.c:858
    #10 0x55b315581457 in ngx_http_process_request src/http/ngx_http_request.c:2094
    #11 0x55b31558283d in ngx_http_process_request_headers src/http/ngx_http_request.c:1496
    #12 0x55b3155834b5 in ngx_http_process_request_line src/http/ngx_http_request.c:1163
    #13 0x55b315583a52 in ngx_http_wait_request_handler src/http/ngx_http_request.c:501
    #14 0x55b31552ed1f in ngx_epoll_process_events src/event/modules/ngx_epoll_module.c:901
    #15 0x55b31550aaf9 in ngx_process_events_and_timers src/event/ngx_event.c:248
    #16 0x55b3155286a8 in ngx_worker_process_cycle src/os/unix/ngx_process_cycle.c:721
    #17 0x55b315523c51 in ngx_spawn_process src/os/unix/ngx_process.c:199
    #18 0x55b315526e46 in ngx_start_worker_processes src/os/unix/ngx_process_cycle.c:344
    #19 0x55b31552a0e5 in ngx_master_process_cycle src/os/unix/ngx_process_cycle.c:130
    #20 0x55b3154a7e69 in main src/core/nginx.c:383
    #21 0x7f88581bed8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
    #22 0x7f88581bee3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
    #23 0x55b3154a4194 in _start (/build/nginx/objs/nginx+0x10e194)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /build/nginx/../nginx-ssl-fingerprint/src/ngx_http_ssl_fingerprint_module.c:68 in ngx_http_ssl_fingerprint
==24==ABORTING

Jul 31 '22 08:07 paragor

Oh, stupid bug, I fixed in master https://github.com/phuslu/nginx-ssl-fingerprint/commit/ab266f737f8b05bb6fa3bfad86aaa0d0e6361119

Jul 31 '22 08:07 phuslu

test-client-hello-max-size.py trigger SEGV on OpenSSL_1_1_1-q (and on OpenSSL_1_1_1-stable)

2022/07/31 13:39:16 [debug] 606#0: epoll: fd:5 ev:0001 d:00007FCA33DB4800
2022/07/31 13:39:16 [debug] 606#0: accept on 0.0.0.0:8444, ready: 0
2022/07/31 13:39:16 [debug] 606#0: posix_memalign: 0000616000002480:512 @16
2022/07/31 13:39:16 [debug] 606#0: *4 accept: 172.17.96.0:55024 fd:6
2022/07/31 13:39:16 [debug] 606#0: *4 event timer add: 6: 60000:5062269
2022/07/31 13:39:16 [debug] 606#0: *4 reusable connection: 1
2022/07/31 13:39:16 [debug] 606#0: *4 epoll add event: fd:6 op:1 ev:80002001
2022/07/31 13:39:16 [debug] 606#0: timer delta: 22539
2022/07/31 13:39:16 [debug] 606#0: worker cycle
2022/07/31 13:39:16 [debug] 606#0: epoll timer: 60000
2022/07/31 13:39:16 [debug] 606#0: epoll: fd:6 ev:0001 d:00007FCA33DB49D1
2022/07/31 13:39:16 [debug] 606#0: *4 http check ssl handshake
2022/07/31 13:39:16 [debug] 606#0: *4 http recv(): 1
2022/07/31 13:39:16 [debug] 606#0: *4 https ssl handshake: 0x16
2022/07/31 13:39:16 [debug] 606#0: *4 tcp_nodelay
2022/07/31 13:39:16 [debug] 606#0: *4 reusable connection: 0
2022/07/31 13:39:16 [debug] 606#0: *4 ngx_ssl_client_hello_ja3_cb: alloc 12 bytes
2022/07/31 13:39:16 [debug] 606#0: *4 ngx_ssl_client_hello_ja3_cb: used 13 bytes
2022/07/31 13:39:16 [debug] 606#0: *4 SSL server name: null
2022/07/31 13:39:16 [debug] 606#0: *4 posix_memalign: 0000616000002D80:512 @16
2022/07/31 13:39:16 [debug] 606#0: *4 SSL_do_handshake: -1
2022/07/31 13:39:16 [debug] 606#0: *4 ngx_ssl_client_hello_ja3_fp: str=[771,47-255], length=[10], md5=[50dcb475711e270135af62647b490103]
2022/07/31 13:39:16 [debug] 606#0: *4 SSL_get_error: 1
2022/07/31 13:39:16 [info] 606#0: *4 SSL_do_handshake() failed (SSL: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher) while SSL handshaking, client: 172.17.96.0, server: 0.0.0.0:8444
2022/07/31 13:39:16 [debug] 606#0: *4 close http connection: 6
2022/07/31 13:39:16 [debug] 606#0: *4 event timer del: 6: 5062269
2022/07/31 13:39:16 [debug] 606#0: *4 reusable connection: 0
2022/07/31 13:39:16 [debug] 606#0: *4 free: 0000616000002480, unused: 29
2022/07/31 13:39:16 [debug] 606#0: *4 free: 0000616000002D80, unused: 448
2022/07/31 13:39:16 [debug] 606#0: timer delta: 1
2022/07/31 13:39:16 [debug] 606#0: worker cycle
2022/07/31 13:39:16 [debug] 606#0: epoll timer: -1
2022/07/31 13:39:16 [debug] 606#0: epoll: fd:5 ev:0001 d:00007FCA33DB4800
2022/07/31 13:39:16 [debug] 606#0: accept on 0.0.0.0:8444, ready: 0
2022/07/31 13:39:16 [debug] 606#0: posix_memalign: 0000616000003080:512 @16
2022/07/31 13:39:16 [debug] 606#0: *5 accept: 172.17.96.0:55028 fd:6
2022/07/31 13:39:16 [debug] 606#0: *5 event timer add: 6: 60000:5062274
2022/07/31 13:39:16 [debug] 606#0: *5 reusable connection: 1
2022/07/31 13:39:16 [debug] 606#0: *5 epoll add event: fd:6 op:1 ev:80002001
2022/07/31 13:39:16 [debug] 606#0: timer delta: 4
2022/07/31 13:39:16 [debug] 606#0: worker cycle
2022/07/31 13:39:16 [debug] 606#0: epoll timer: 60000
2022/07/31 13:39:16 [debug] 606#0: epoll: fd:6 ev:0001 d:00007FCA33DB49D0
2022/07/31 13:39:16 [debug] 606#0: *5 http check ssl handshake
2022/07/31 13:39:16 [debug] 606#0: *5 http recv(): 1
2022/07/31 13:39:16 [debug] 606#0: *5 https ssl handshake: 0x16
2022/07/31 13:39:16 [debug] 606#0: *5 tcp_nodelay
2022/07/31 13:39:16 [debug] 606#0: *5 reusable connection: 0
AddressSanitizer:DEADLYSIGNAL
=================================================================
==606==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x56273b9812f8 bp 0x7fff75370640 sp 0x7fff75370510 T0)
==606==The signal is caused by a READ memory access.
==606==Hint: address points to the zero page.
    #0 0x56273b9812f8 in ngx_ssl_client_hello_ja3_fp src/event/ngx_event_openssl.c:1805
    #1 0x56273b9866b2 in ngx_ssl_handshake src/event/ngx_event_openssl.c:1898
    #2 0x56273b9d2128 in ngx_http_ssl_handshake src/http/ngx_http_request.c:757
    #3 0x56273b97bd1f in ngx_epoll_process_events src/event/modules/ngx_epoll_module.c:901
    #4 0x56273b957af9 in ngx_process_events_and_timers src/event/ngx_event.c:248
    #5 0x56273b9756a8 in ngx_worker_process_cycle src/os/unix/ngx_process_cycle.c:721
    #6 0x56273b970c51 in ngx_spawn_process src/os/unix/ngx_process.c:199
    #7 0x56273b97775a in ngx_reap_children src/os/unix/ngx_process_cycle.c:598
    #8 0x56273b97775a in ngx_master_process_cycle src/os/unix/ngx_process_cycle.c:174
    #9 0x56273b8f4e69 in main src/core/nginx.c:383
    #10 0x7fca36bbdd8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
    #11 0x7fca36bbde3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
    #12 0x56273b8f1194 in _start (/build/nginx/objs/nginx+0x10e194)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV src/event/ngx_event_openssl.c:1805 in ngx_ssl_client_hello_ja3_fp
==606==ABORTING
2022/07/31 13:39:16 [notice] 604#0: signal 17 (SIGCHLD) received from 606
2022/07/31 13:39:16 [notice] 604#0: worker process 606 exited with code 1

pcap.pcap.zip

it is not valid request, but i think nginx should not have SEGV. So we need more guards :)

Jul 31 '22 13:07 paragor

thanks again, i will look into today.

Aug 01 '22 02:08 phuslu

I tested in production latest commit (https://github.com/phuslu/nginx-ssl-fingerprint/commit/40c3798ba713f20e3298fa6661a9cc65deaac186) so its pretty good:

popular ja3 results equals https://github.com/open-ch/ja3
no segv
no empty http_ssl_ja3_hash when ssl connection is established

in prod we use:

nginx version: nginx/1.22.0
openssl 1.1.1q

so i think https://github.com/phuslu/nginx-ssl-fingerprint/commit/40c3798ba713f20e3298fa6661a9cc65deaac186 is good candidate for 0.4.1 release

and we can close this issue

Aug 08 '22 00:08 paragor

many thanks @paragor, v0.4.1 tagged.

Aug 09 '22 03:08 phuslu

nginx-ssl-fingerprint nginx-ssl-fingerprint copied to clipboard

groups and formats are lost during tls session ticket

GREASE_TABLE Ref: https://tools.ietf.org/html/draft-davidben-tls-grease-00

first request in pcap is curl

next requests are chrome macos

nginx-ssl-fingerprint
nginx-ssl-fingerprint copied to clipboard