Crash seen after adding FilterRule

[pranavbhalerao]

sslproxy is getting crashed after adding Filter rule
Following Proxyspec configuration:

ProxySpec {
Proto ssl
Addr 127.0.0.1
Port 8443
DivertPort 8080
Divert no
Passthrough yes
CACert /home/pranav/ca.crt
CAKey /home/pranav/ca.key
ForceSSLProto tls12
VerifyPeer no

FilterRule {
   Action Pass
   SrcIp 1.1.1.1
   Passthrough yes

   DstIp 2.2.2.2
   CACert /home/pranav/ca.crt
   CAKey /home/pranav/ca.key
   ForceSSLProto tls12
   VerifyPeer no
}

}

If I initiate a command wget https://2.2.2.2 --no-check-certificate from host 1.1.1.3. Crash is observed.

GDB crash is pointing following location

00007f1ec5346b14 in X509_get_subject_name () from /lib/x86_64-linux-gnu/libcrypto.so.3
(gdb) bt
#0 0x00007f1ec5346b14 in X509_get_subject_name () from /lib/x86_64-linux-gnu/libcrypto.so.3
#1 0x000055f378ce2060 in ssl_x509_forge (cacrt=0x0, cakey=0x0, origcrt=0x7f1ebc0156a0, key=0x55f3844cd890, extraname=extraname@entry=0x0, crlurl=0x0) at ssl.c:978
#2 0x000055f378cd8b02 in protossl_srccert_create (ctx=0x55f3844d9b40) at protossl.c:591
#3 protossl_srcssl_create (ctx=ctx@entry=0x55f3844d9b40, origssl=) at protossl.c:915
#4 0x000055f378cd8c2f in protossl_setup_src_ssl (ctx=0x55f3844d9b40) at protossl.c:1574
#5 0x000055f378cd90b2 in protossl_bev_eventcb_connected_srvdst (bev=, ctx=0x55f3844d9b40) at protossl.c:1701
#6 protossl_bev_eventcb_srvdst (bev=, events=, ctx=0x55f3844d9b40) at protossl.c:1757
#7 0x000055f378cdf0b3 in pxy_bev_eventcb (bev=0x7f1ebc007530, events=, arg=0x55f3844d9b40) at pxyconn.c:1482
#8 0x00007f1ec50ce1b4 in ?? () from /lib/x86_64-linux-gnu/libevent-2.1.so.7
#9 0x00007f1ec50d8b9b in ?? () from /lib/x86_64-linux-gnu/libevent-2.1.so.7
#10 0x00007f1ec50da8a7 in event_base_loop () from /lib/x86_64-linux-gnu/libevent-2.1.so.7
#11 0x000055f378ce0a96 in pxy_thr (arg=0x55f3844cc510) at pxythr.c:354
#12 0x00007f1ec4d8aac3 in start_thread (arg=) at ./nptl/pthread_create.c:442
#13 0x00007f1ec4e1c850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)

root@ngfw:/home/pranav/SSLproxy/src# ./sslproxy -D -f spec
./sslproxy: overriding -r ssl version option
SSLproxy v0.9.7-dirty (built 2024-12-15)
Copyright (c) 2017-2024, Soner Tari sonertari@gmail.com
https://github.com/sonertari/SSLproxy
Copyright (c) 2009-2019, Daniel Roethlisberger daniel@roe.ch
https://www.roe.ch/SSLsplit
Build info: V:GIT
Features: -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT IP6T_SO_ORIGINAL_DST
Local process info support: no
compiled against OpenSSL 3.0.2 15 Mar 2022 (30000020)
rtlinked against OpenSSL 3.0.2 15 Mar 2022 (30000020)
OpenSSL has support for TLS extensions
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
OpenSSL has engine support
Using SSL_MODE_RELEASE_BUFFERS
SSL/TLS protocol availability: tls10 tls11 tls12 tls13
SSL/TLS algorithm availability: !SHA0 RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION
SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.1.12-stable
rtlinked against libevent 2.1.12-stable
compiled against libnet 1.1.6
rtlinked against libnet 1.1.6
compiled against libpcap n/a
rtlinked against libpcap 1.10.1 (with TPACKET_V3)
compiled against sqlite 3.37.2
rtlinked against sqlite 3.37.2
4 CPU cores detected
Generated 2048 bit RSA key for leaf certs.
Global conn opts: negotiate>=tls10<=tls13|ALL:-aNULL|TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256|no ecdhcurve|no leafcrlurl|remove_http_referer|verify_peer|no user_auth_url|300|8192
proxyspecs:

• listen=[127.0.0.1]:8443 ssl netfilter
  divert addr= [127.0.0.1]:8080
  return addr= [127.0.0.1]:0
  opts= conn opts: tls12>=tls10<=tls13|passthrough|ALL:-aNULL|TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256|no ecdhcurve|no leafcrlurl|remove_http_referer|no user_auth_url|300|8192
  split||
  filter rule 0: dstip=2.2.2.2, dstport=, srcip=1.1.1.3, user=, desc=, exact=site||ip||, all=|||, action=||pass||, log=|||||, precedence=2
  conn opts: tls12>=tls10<=tls13|passthrough|ALL:-aNULL|TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256|no ecdhcurve|no leafcrlurl|remove_http_referer|no user_auth_url|300|8192
  filter=>
  userdesc_filter_exact->
  userdesc_filter_substring->
  user_filter_exact->
  user_filter_substring->
  desc_filter_exact->
  desc_filter_substring->
  user_filter_all->
  ip_filter_exact->
  ip 0 1.1.1.3 (exact)=
  ip exact:
  0: 2.2.2.2 (exact, action=||pass||, log=|||||, precedence=2
  conn opts: tls12>=tls10<=tls13|passthrough|ALL:-aNULL|TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256|no ecdhcurve|no leafcrlurl|remove_http_referer|no user_auth_url|300|8192)
  ip_filter_substring->
  filter_all->

WARNING: Divert address specified in split mode
No Global CA loaded.
Loaded ProxySpec CA: '/C=IN/ST=KAR/L=BLR/O=ipfence.org/OU=SBG/CN=ipfence.org'
Loaded FilterRule CA: '/C=IN/ST=KAR/L=BLR/O=ipfence.org/OU=SBG/CN=ipfence.org'
SSL/TLS leaf certificates taken from:

• Global connection drop
  Privsep fastpath disabled
  Created self-pipe [r=3,w=4]
  Created chld-pipe [r=5,w=6]
  Created socketpair 0 [p=7,c=8]
  Created socketpair 1 [p=9,c=10]
  Created socketpair 2 [p=11,c=12]
  Created socketpair 3 [p=13,c=14]
  Created socketpair 4 [p=15,c=16]
  Created socketpair 5 [p=17,c=18]
  Privsep parent pid 2272
  Privsep child pid 2273
  Using libevent backend 'epoll'
  Event base supports: edge yes, O(1) yes, anyfd no
  Received privsep req type 03 sz 9 on srvsock 7
  Dropped privs to user nobody group - chroot -
  Inserted events:
  0x55f3844d3148 [fd 4] Read Persist Internal
  0x55f3844d3320 [fd 6] Read Persist Internal
  0x55f3844d33f8 [fd 7] Read Persist
  0x55f3844ce630 [sig 1] Signal Persist
  0x55f3844c8a40 [sig 2] Signal Persist
  Received privsep req type 00 sz 1 on srvsock 9
  Received privsep req type 00 sz 1 on srvsock 11
  0x55f3844ce5a0 [sig 3] Signal Persist
  Received privsep req type 00 sz 1 on srvsock 13
  0x55f3844d2380 [sig 10] Signal Persist
  Received privsep req type 00 sz 1 on srvsock 15
  0x55f3844cf110 [sig 13] Signal Persist
  Received privsep req type 00 sz 1 on srvsock 17
  0x55f3844cf310 [sig 15] Signal Persist
  0x55f3844d4460 [fd -1] Persist Timeout=1734247837.481022
  Active events:
  Initialized 8 connection handling threads
  Started 8 connection handling threads
  Starting main event loop.
  SNI peek: [n/a] [complete], fd=41
  Connecting to [2.2.2.2]:443
  ===> Original server certificate:
  Subject DN: /C=IN/ST=KAR/L=BLR/O=nginx/OU=web/CN=nginx.com
  Common Names: nginx.com
  Fingerprint: 6C:E9:B1:E5:E7:20:68:95:2B:3EBE:3D:85:59:79:E1:2F:85:4D:12
  Certificate cache: MISS

[sonertari]

I think this is related to "No Global CA loaded." My current theory is that if no global ca crt/key specified, for some reason we don't have a ca crt/key pair if the specified filter rule does NOT match.

Can you add your CACert/Key globally as well and try again?
For example, copy-paste the following lines from your ProxySpec to global config above the ProxySpec:

CACert /home/pranav/ca.crt
CAKey /home/pranav/ca.key

Also, can you rebuild sslproxy with DEBUG_PROXY and DEBUG_OPTS switches enabled in Mk/main.mk, and then start sslproxy with the -D4 option please (make sure you remove the global ca crt/key to reproduce the issue)? This should give us verbose debug logs to see how ca crt/key are NULL while forging the server crt (cacrt=0x0, cakey=0x0 in the gdb bt you provided).

[pranavbhalerao]

Hi Sonertari,
Thanks for your quick reply. You were absolutely right, after adding global CA and Key crash is not observed.

However, if I interpreted statement correctly, this issue should be seen only if filter rule does not MATCH but this scenario should work if we match filter rule ?
I have modified my filter as below to match with incoming traffic. I am initiating traffic from 1.1.1.3 and destination is 2.2.2.2 but still I see in logs "No filter match with port: 1.1.1.3:43924, 2.2.2.2:443" what is making traffic not to match filter ?

FilterRule {
Action Pass
SrcIp 1.1.1.3
Passthrough yes

   DstIp 2.2.2.2
   CACert /home/pranav/ca.crt
   CAKey /home/pranav/ca.key
   ForceSSLProto tls12
   VerifyPeer no

}

I have one more question about FilterRule, I see ip and dstip in struct filter_rule defined as char* which looks like user is only suppose to configure it as host address.
Is subnet address for SrcIp/DstIp supported like 1.1.1.0/24 ?

Following are the logs after enabling DEBUG_OPTS and DEBUG_PROXY and running sslproxy with -D4

Started 8 connection handling threads
Starting main event loop.

[FINEST] proxy_listener_acceptcb: ENTER, fd=41
[FINEST] proxy_conn_ctx_new: ENTER, fd=41
[FINEST] [0.0 fd=41 cfd=0] proxy_conn_ctx_new: Created new conn
[FINEST] [0.0 fd=41 cfd=0] pxy_thrmgr_assign_thr: ENTER
[FINEST] [0.0 fd=41 cfd=0] protossl_init_conn: ENTER
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_init: ENTER
[FINEST] [0.0 fd=41 cfd=0] pxy_thr_attach: Adding conn
[FINER] [0.0 fd=41 cfd=0] check_fd_usage: descriptor_table_size=1024, dtablecount=43, reserve=10
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_init: srcaddr= [1.1.1.3]:43924
[FINEST] [0.0 fd=41 cfd=0] protossl_fd_readcb: ENTER
SNI peek: [n/a] [complete], fd=41
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_connect: ENTER
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter: Searching ip exact: 1.1.1.3
[FINE] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Found site (line=13): 2.2.2.2 for 1.1.1.3:43924, 2.2.2.2:443
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Match exact with dst (line=13): 2.2.2.2, 2.2.2.2
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter_port: No filter match with port: 1.1.1.3:43924, 2.2.2.2:443
[FINE] [0.0 fd=41 cfd=0] pxy_conn_set_filter_action: Filter pass action for 2.2.2.2, precedence 2 (line=13)
[FINE] [0.0 fd=41 cfd=0] pxy_conn_apply_filter: Deferring pass action
Connecting to [2.2.2.2]:443
[FINEST] [0.0 fd=41 cfd=0] protossl_conn_connect: ENTER
[FINEST] [0.0 fd=41 cfd=0] protossl_bufferevent_setup: ENTER, fd=-1
[FINEST] [0.0 fd=41 cfd=0] protossl_bufferevent_setup: bufferevent_openssl_set_allow_dirty_shutdown, fd=-1
[FINEST] [0.0 fd=41 cfd=0] protossl_bev_eventcb_connected_srvdst: ENTER
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter: Searching ip exact: 1.1.1.3
[FINE] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Found site (line=13): 2.2.2.2 for 1.1.1.3:43924, 2.2.2.2:443
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Match exact with dst (line=13): 2.2.2.2, 2.2.2.2
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter_port: No filter match with port: 1.1.1.3:43924, 2.2.2.2:443
[FINE] [0.0 fd=41 cfd=0] pxy_conn_set_filter_action: Filter pass action for 2.2.2.2, precedence 2 (line=13)
[FINE] [0.0 fd=41 cfd=0] pxy_conn_apply_filter: Deferring pass action
===> Original server certificate:
Subject DN: /C=IN/ST=KAR/L=BLR/O=nginx/OU=web/CN=nginx.com
Common Names: nginx.com
Fingerprint: 6C:E9:B1:E5:E7:20:68:95:2B:3EBE:3D:85:59:79:E1:2F:85:4D:12
Certificate cache: MISS
Child pid 4372 killed by signal 11
root@ngfw:/home/pranav/SSLproxy/src#

[sonertari]

This is a bug I can reproduce here myself too. Thanks for reporting.
I'm working on a fix, please use a global crt/key pair for now.

Btw, the port related log simply says that there is no port-only filter rule matching that connection, that's all. Otherwise, your src/dst ip filter rule successfully matches twice (and is deferred twice) as expected. No problem there.

[sonertari]

I have just pushed a fix to the develop branch. It seems to work. Can you test it please?

[pranavbhalerao]

I have tested new change and it is working absolutely fine. I did not observe any crash. Thanks for fixing it quickly.

I am observing another issue with filter pls clarify if any configuration mistake is present.
As you mentioned earlier, filter match is not happening because Port configuration is missing and that is correct. After adding DstPort 443 its matching filter and taking action.
Looking at code, DstPort also accept * (for all ports) so if Port is configured as * filter is not matching.
Filter is configured as follow.

FilterRule {
   Action Pass
   SrcIp 1.1.1.3
   Passthrough yes

   DstIp 2.2.2.2
   DstPort *
   CACert /home/pranav/ca.crt
   CAKey /home/pranav/ca.key
   ForceSSLProto tls12
   VerifyPeer no
}

Proxy Debug logs:

[FINE] [0.0 fd=41 cfd=0] protopassthrough_engage: ENTER
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_free_children: ENTER
[FINER] [0.0 fd=41 cfd=0] protossl_bufferevent_free_and_close_fd: in=0, out=0, fd=42
SSL_free() in state 00000001 = 0001 = SSLOK (SSL negotiation finished successfully) [connect socket]
[FINER] [0.0 fd=41 cfd=0] protossl_bufferevent_free_and_close_fd: fd=42, SSL_free() in state 00000001 = 0001 = SSLOK (SSL negotiation finished successfully) [connect socket]
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_connect: ENTER
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter: Searching ip exact: 1.1.1.3
[FINE] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Found site (line=0): 2.2.2.2 for 1.1.1.3:42684, 2.2.2.2:443
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Rule precedence lower than conn filter precedence 0 < 3 (line=0): 2.2.2.2, 2.2.2.2
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_dsthost_filter: No filter match with ip: 1.1.1.3:42684, 2.2.2.2:443
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter: Searching ip substring: 1.1.1.3
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_filter: Searching all
[FINEST] [0.0 fd=41 cfd=0] pxy_conn_dsthost_filter: No filter match with ip: 1.1.1.3:42684, 2.2.2.2:443
Connecting to [2.2.2.2]:443

[sonertari]

I am not sure if I understand your description correctly, but I've tried the following:

First:

SrcIp 1.1.1.1
DstIp 2.2.2.2
DstPort 443

And second:

SrcIp 1.1.1.1
DstIp 2.2.2.2
DstPort *

They both pass connections to port 443 through, as expected.

Note that those debug logs are verbose logs for different functions, i.e. for different types of rules. The one you highlighted is for exact match. Since you don't have exact ip rule in the second test case, pxy_conn_dsthost_filter() simply says that it cannot find any match. That's all.

Otherwise looking at your debug logs, I don't see any issues:

[FINE] [0.0 fd=41 cfd=0] pxy_conn_filter_match_ip: Found site (line=0): 2.2.2.2 for 1.1.1.3:42684, 2.2.2.2:443

And even though you did not provide the rest of the logs, my guess is that your filter rule successfully passes the connection through, right?

Btw, I hope you use -D4 as the first command line option, e.g.: sslproxy -D4 -f sslproxy.conf, so that you can also see verbose logs for filter rules too.

On a related matter, I have just pushed a change to the develop branch to force SSL/TLS configuration in SSL proxyspecs.
Thanks for testing.