SSLproxy icon indicating copy to clipboard operation
SSLproxy copied to clipboard
verified publisher icon sonertari

Crash seen after adding FilterRule

Open pranavbhalerao opened this issue 1 year ago • 6 comments

sslproxy is getting crashed after adding Filter rule Following Proxyspec configuration:

ProxySpec { Proto ssl Addr 127.0.0.1 Port 8443 DivertPort 8080 Divert no Passthrough yes CACert /home/pranav/ca.crt CAKey /home/pranav/ca.key ForceSSLProto tls12 VerifyPeer no

FilterRule {
   Action Pass
   SrcIp 1.1.1.1
   Passthrough yes

   DstIp 2.2.2.2
   CACert /home/pranav/ca.crt
   CAKey /home/pranav/ca.key
   ForceSSLProto tls12
   VerifyPeer no
}

}

If I initiate a command wget https://2.2.2.2 --no-check-certificate from host 1.1.1.3. Crash is observed.

GDB crash is pointing following location

00007f1ec5346b14 in X509_get_subject_name () from /lib/x86_64-linux-gnu/libcrypto.so.3 (gdb) bt #0 0x00007f1ec5346b14 in X509_get_subject_name () from /lib/x86_64-linux-gnu/libcrypto.so.3 #1 0x000055f378ce2060 in ssl_x509_forge (cacrt=0x0, cakey=0x0, origcrt=0x7f1ebc0156a0, key=0x55f3844cd890, extraname=extraname@entry=0x0, crlurl=0x0) at ssl.c:978 #2 0x000055f378cd8b02 in protossl_srccert_create (ctx=0x55f3844d9b40) at protossl.c:591 #3 protossl_srcssl_create (ctx=ctx@entry=0x55f3844d9b40, origssl=) at protossl.c:915 #4 0x000055f378cd8c2f in protossl_setup_src_ssl (ctx=0x55f3844d9b40) at protossl.c:1574 #5 0x000055f378cd90b2 in protossl_bev_eventcb_connected_srvdst (bev=, ctx=0x55f3844d9b40) at protossl.c:1701 #6 protossl_bev_eventcb_srvdst (bev=, events=, ctx=0x55f3844d9b40) at protossl.c:1757 #7 0x000055f378cdf0b3 in pxy_bev_eventcb (bev=0x7f1ebc007530, events=, arg=0x55f3844d9b40) at pxyconn.c:1482 #8 0x00007f1ec50ce1b4 in ?? () from /lib/x86_64-linux-gnu/libevent-2.1.so.7 #9 0x00007f1ec50d8b9b in ?? () from /lib/x86_64-linux-gnu/libevent-2.1.so.7 #10 0x00007f1ec50da8a7 in event_base_loop () from /lib/x86_64-linux-gnu/libevent-2.1.so.7 #11 0x000055f378ce0a96 in pxy_thr (arg=0x55f3844cc510) at pxythr.c:354 #12 0x00007f1ec4d8aac3 in start_thread (arg=) at ./nptl/pthread_create.c:442 #13 0x00007f1ec4e1c850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 (gdb)

root@ngfw:/home/pranav/SSLproxy/src# ./sslproxy -D -f spec ./sslproxy: overriding -r ssl version option SSLproxy v0.9.7-dirty (built 2024-12-15) Copyright (c) 2017-2024, Soner Tari [email protected] https://github.com/sonertari/SSLproxy Copyright (c) 2009-2019, Daniel Roethlisberger [email protected] https://www.roe.ch/SSLsplit Build info: V:GIT Features: -DHAVE_NETFILTER NAT engines: netfilter* tproxy netfilter: IP_TRANSPARENT IP6T_SO_ORIGINAL_DST Local process info support: no compiled against OpenSSL 3.0.2 15 Mar 2022 (30000020) rtlinked against OpenSSL 3.0.2 15 Mar 2022 (30000020) OpenSSL has support for TLS extensions TLS Server Name Indication (SNI) supported OpenSSL is thread-safe with THREADID OpenSSL has engine support Using SSL_MODE_RELEASE_BUFFERS SSL/TLS protocol availability: tls10 tls11 tls12 tls13 SSL/TLS algorithm availability: !SHA0 RSA DSA ECDSA DH ECDH EC OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG compiled against libevent 2.1.12-stable rtlinked against libevent 2.1.12-stable compiled against libnet 1.1.6 rtlinked against libnet 1.1.6 compiled against libpcap n/a rtlinked against libpcap 1.10.1 (with TPACKET_V3) compiled against sqlite 3.37.2 rtlinked against sqlite 3.37.2 4 CPU cores detected Generated 2048 bit RSA key for leaf certs. Global conn opts: negotiate>=tls10<=tls13|ALL:-aNULL|TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256|no ecdhcurve|no leafcrlurl|remove_http_referer|verify_peer|no user_auth_url|300|8192 proxyspecs:

  • listen=[127.0.0.1]:8443 ssl netfilter divert addr= [127.0.0.1]:8080 return addr= [127.0.0.1]:0 opts= conn opts: tls12>=tls10<=tls13|passthrough|ALL:-aNULL|TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256|no ecdhcurve|no leafcrlurl|remove_http_referer|no user_auth_url|300|8192 split|| filter rule 0: dstip=2.2.2.2, dstport=, srcip=1.1.1.3, user=, desc=, exact=site||ip||, all=|||, action=||pass||, log=|||||, precedence=2 conn opts: tls12>=tls10<=tls13|passthrough|ALL:-aNULL|TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256|no ecdhcurve|no leafcrlurl|remove_http_referer|no user_auth_url|300|8192 filter=> userdesc_filter_exact-> userdesc_filter_substring-> user_filter_exact-> user_filter_substring-> desc_filter_exact-> desc_filter_substring-> user_filter_all-> ip_filter_exact-> ip 0 1.1.1.3 (exact)= ip exact: 0: 2.2.2.2 (exact, action=||pass||, log=|||||, precedence=2 conn opts: tls12>=tls10<=tls13|passthrough|ALL:-aNULL|TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256|no ecdhcurve|no leafcrlurl|remove_http_referer|no user_auth_url|300|8192) ip_filter_substring-> filter_all->

WARNING: Divert address specified in split mode No Global CA loaded. Loaded ProxySpec CA: '/C=IN/ST=KAR/L=BLR/O=ipfence.org/OU=SBG/CN=ipfence.org' Loaded FilterRule CA: '/C=IN/ST=KAR/L=BLR/O=ipfence.org/OU=SBG/CN=ipfence.org' SSL/TLS leaf certificates taken from:

  • Global connection drop Privsep fastpath disabled Created self-pipe [r=3,w=4] Created chld-pipe [r=5,w=6] Created socketpair 0 [p=7,c=8] Created socketpair 1 [p=9,c=10] Created socketpair 2 [p=11,c=12] Created socketpair 3 [p=13,c=14] Created socketpair 4 [p=15,c=16] Created socketpair 5 [p=17,c=18] Privsep parent pid 2272 Privsep child pid 2273 Using libevent backend 'epoll' Event base supports: edge yes, O(1) yes, anyfd no Received privsep req type 03 sz 9 on srvsock 7 Dropped privs to user nobody group - chroot - Inserted events: 0x55f3844d3148 [fd 4] Read Persist Internal 0x55f3844d3320 [fd 6] Read Persist Internal 0x55f3844d33f8 [fd 7] Read Persist 0x55f3844ce630 [sig 1] Signal Persist 0x55f3844c8a40 [sig 2] Signal Persist Received privsep req type 00 sz 1 on srvsock 9 Received privsep req type 00 sz 1 on srvsock 11 0x55f3844ce5a0 [sig 3] Signal Persist Received privsep req type 00 sz 1 on srvsock 13 0x55f3844d2380 [sig 10] Signal Persist Received privsep req type 00 sz 1 on srvsock 15 0x55f3844cf110 [sig 13] Signal Persist Received privsep req type 00 sz 1 on srvsock 17 0x55f3844cf310 [sig 15] Signal Persist 0x55f3844d4460 [fd -1] Persist Timeout=1734247837.481022 Active events: Initialized 8 connection handling threads Started 8 connection handling threads Starting main event loop. SNI peek: [n/a] [complete], fd=41 Connecting to [2.2.2.2]:443 ===> Original server certificate: Subject DN: /C=IN/ST=KAR/L=BLR/O=nginx/OU=web/CN=nginx.com Common Names: nginx.com Fingerprint: 6C:E9:B1:E5:E7:20:68:95:2B:3EBE:3D:85:59:79:E1:2F:85:4D:12 Certificate cache: MISS

pranavbhalerao avatar Dec 15 '24 07:12 pranavbhalerao

I think this is related to "No Global CA loaded." My current theory is that if no global ca crt/key specified, for some reason we don't have a ca crt/key pair if the specified filter rule does NOT match.

Can you add your CACert/Key globally as well and try again? For example, copy-paste the following lines from your ProxySpec to global config above the ProxySpec:

CACert /home/pranav/ca.crt
CAKey /home/pranav/ca.key

Also, can you rebuild sslproxy with DEBUG_PROXY and DEBUG_OPTS switches enabled in Mk/main.mk, and then start sslproxy with the -D4 option please (make sure you remove the global ca crt/key to reproduce the issue)? This should give us verbose debug logs to see how ca crt/key are NULL while forging the server crt (cacrt=0x0, cakey=0x0 in the gdb bt you provided).

sonertari avatar Dec 15 '24 09:12 sonertari

Hi Sonertari, Thanks for your quick reply. You were absolutely right, after adding global CA and Key crash is not observed.

However, if I interpreted statement correctly, this issue should be seen only if filter rule does not MATCH but this scenario should work if we match filter rule ? I have modified my filter as below to match with incoming traffic. I am initiating traffic from 1.1.1.3 and destination is 2.2.2.2 but still I see in logs "No filter match with port: 1.1.1.3:43924, 2.2.2.2:443" what is making traffic not to match filter ?

FilterRule { Action Pass SrcIp 1.1.1.3 Passthrough yes

   DstIp 2.2.2.2
   CACert /home/pranav/ca.crt
   CAKey /home/pranav/ca.key
   ForceSSLProto tls12
   VerifyPeer no

}

I have one more question about FilterRule, I see ip and dstip in struct filter_rule defined as char* which looks like user is only suppose to configure it as host address. Is subnet address for SrcIp/DstIp supported like 1.1.1.0/24 ?

Following are the logs after enabling DEBUG_OPTS and DEBUG_PROXY and running sslproxy with -D4

Started 8 connection handling threads
Starting main event loop.

[FINEST] proxy_listener_acceptcb: ENTER, fd=41 [FINEST] proxy_conn_ctx_new: ENTER, fd=41 [FINEST] [0.0 fd=41 cfd=0] proxy_conn_ctx_new: Created new conn [FINEST] [0.0 fd=41 cfd=0] pxy_thrmgr_assign_thr: ENTER [FINEST] [0.0 fd=41 cfd=0] protossl_init_conn: ENTER [FINEST] [0.0 fd=41 cfd=0] pxy_conn_init: ENTER [FINEST] [0.0 fd=41 cfd=0] pxy_thr_attach: Adding conn [FINER] [0.0 fd=41 cfd=0] check_fd_usage: descriptor_table_size=1024, dtablecount=43, reserve=10 [FINEST] [0.0 fd=41 cfd=0] pxy_conn_init: srcaddr= [1.1.1.3]:43924 [FINEST] [0.0 fd=41 cfd=0] protossl_fd_readcb: ENTER SNI peek: [n/a] [complete], fd=41 [FINEST] [0.0 fd=41 cfd=0] pxy_conn_connect: ENTER [FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter: Searching ip exact: 1.1.1.3 [FINE] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Found site (line=13): 2.2.2.2 for 1.1.1.3:43924, 2.2.2.2:443 [FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Match exact with dst (line=13): 2.2.2.2, 2.2.2.2 [FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter_port: No filter match with port: 1.1.1.3:43924, 2.2.2.2:443 [FINE] [0.0 fd=41 cfd=0] pxy_conn_set_filter_action: Filter pass action for 2.2.2.2, precedence 2 (line=13) [FINE] [0.0 fd=41 cfd=0] pxy_conn_apply_filter: Deferring pass action Connecting to [2.2.2.2]:443 [FINEST] [0.0 fd=41 cfd=0] protossl_conn_connect: ENTER [FINEST] [0.0 fd=41 cfd=0] protossl_bufferevent_setup: ENTER, fd=-1 [FINEST] [0.0 fd=41 cfd=0] protossl_bufferevent_setup: bufferevent_openssl_set_allow_dirty_shutdown, fd=-1 [FINEST] [0.0 fd=41 cfd=0] protossl_bev_eventcb_connected_srvdst: ENTER [FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter: Searching ip exact: 1.1.1.3 [FINE] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Found site (line=13): 2.2.2.2 for 1.1.1.3:43924, 2.2.2.2:443 [FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Match exact with dst (line=13): 2.2.2.2, 2.2.2.2 [FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter_port: No filter match with port: 1.1.1.3:43924, 2.2.2.2:443 [FINE] [0.0 fd=41 cfd=0] pxy_conn_set_filter_action: Filter pass action for 2.2.2.2, precedence 2 (line=13) [FINE] [0.0 fd=41 cfd=0] pxy_conn_apply_filter: Deferring pass action ===> Original server certificate: Subject DN: /C=IN/ST=KAR/L=BLR/O=nginx/OU=web/CN=nginx.com Common Names: nginx.com Fingerprint: 6C:E9:B1:E5:E7:20:68:95:2B:3EBE:3D:85:59:79:E1:2F:85:4D:12 Certificate cache: MISS Child pid 4372 killed by signal 11
root@ngfw:/home/pranav/SSLproxy/src#

pranavbhalerao avatar Dec 15 '24 17:12 pranavbhalerao

This is a bug I can reproduce here myself too. Thanks for reporting. I'm working on a fix, please use a global crt/key pair for now.

Btw, the port related log simply says that there is no port-only filter rule matching that connection, that's all. Otherwise, your src/dst ip filter rule successfully matches twice (and is deferred twice) as expected. No problem there.

sonertari avatar Dec 16 '24 21:12 sonertari

I have just pushed a fix to the develop branch. It seems to work. Can you test it please?

sonertari avatar Dec 17 '24 09:12 sonertari

I have tested new change and it is working absolutely fine. I did not observe any crash. Thanks for fixing it quickly.

I am observing another issue with filter pls clarify if any configuration mistake is present. As you mentioned earlier, filter match is not happening because Port configuration is missing and that is correct. After adding DstPort 443 its matching filter and taking action. Looking at code, DstPort also accept * (for all ports) so if Port is configured as * filter is not matching. Filter is configured as follow.

FilterRule {
   Action Pass
   SrcIp 1.1.1.3
   Passthrough yes

   DstIp 2.2.2.2
   DstPort *
   CACert /home/pranav/ca.crt
   CAKey /home/pranav/ca.key
   ForceSSLProto tls12
   VerifyPeer no
}

Proxy Debug logs:

[FINE] [0.0 fd=41 cfd=0] protopassthrough_engage: ENTER [FINEST] [0.0 fd=41 cfd=0] pxy_conn_free_children: ENTER [FINER] [0.0 fd=41 cfd=0] protossl_bufferevent_free_and_close_fd: in=0, out=0, fd=42 SSL_free() in state 00000001 = 0001 = SSLOK (SSL negotiation finished successfully) [connect socket] [FINER] [0.0 fd=41 cfd=0] protossl_bufferevent_free_and_close_fd: fd=42, SSL_free() in state 00000001 = 0001 = SSLOK (SSL negotiation finished successfully) [connect socket] [FINEST] [0.0 fd=41 cfd=0] pxy_conn_connect: ENTER [FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter: Searching ip exact: 1.1.1.3 [FINE] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Found site (line=0): 2.2.2.2 for 1.1.1.3:42684, 2.2.2.2:443 [FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Rule precedence lower than conn filter precedence 0 < 3 (line=0): 2.2.2.2, 2.2.2.2 [FINEST] [0.0 fd=41 cfd=0] pxy_conn_dsthost_filter: No filter match with ip: 1.1.1.3:42684, 2.2.2.2:443 [FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter: Searching ip substring: 1.1.1.3 [FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter: Searching all [FINEST] [0.0 fd=41 cfd=0] pxy_conn_dsthost_filter: No filter match with ip: 1.1.1.3:42684, 2.2.2.2:443 Connecting to [2.2.2.2]:443

pranavbhalerao avatar Dec 19 '24 16:12 pranavbhalerao

I am not sure if I understand your description correctly, but I've tried the following:

First:

SrcIp 1.1.1.1
DstIp 2.2.2.2
DstPort 443

And second:

SrcIp 1.1.1.1
DstIp 2.2.2.2
DstPort *

They both pass connections to port 443 through, as expected.

Note that those debug logs are verbose logs for different functions, i.e. for different types of rules. The one you highlighted is for exact match. Since you don't have exact ip rule in the second test case, pxy_conn_dsthost_filter() simply says that it cannot find any match. That's all.

Otherwise looking at your debug logs, I don't see any issues:

[FINE] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Found site (line=0): 2.2.2.2 for 1.1.1.3:42684, 2.2.2.2:443

And even though you did not provide the rest of the logs, my guess is that your filter rule successfully passes the connection through, right?

Btw, I hope you use -D4 as the first command line option, e.g.: sslproxy -D4 -f sslproxy.conf, so that you can also see verbose logs for filter rules too.

On a related matter, I have just pushed a change to the develop branch to force SSL/TLS configuration in SSL proxyspecs. Thanks for testing.

sonertari avatar Dec 19 '24 20:12 sonertari