sonertari
ssslproxy + iproute (mark)
Hello,
I have a question and would like to know if you could clarify. I'm trying to set a mark(setsockopt) to use with iproute2, so where the socket is being opened with the real destination server and internal proxy (squid or e2guardian).
The listening program (squid or e2guardian) gives the packets back to sslproxy, which re-encrypts and sends them to the server (the original destination). So sslproxy opens (must open) the sockets on the server side. See the diagram on the README.
Hello, thanks for the explanation, I understood the diagram, about how the communication between squid/e2guardian and also how SSLProxy opens the connection with the real destination server. I'm trying to set up a mark to route network traffic through other internet links. I did a test by setting the mark in pxy_listener_acceptcb_child and pxy_setup_child_listener. The test was performed using Centos 7
diff -Nru SSLproxy-orig/pxyconn.c SSLproxy-mod/pxyconn.c
--- SSLproxy-orig/pxyconn.c 2020-03-02 20:51:48.925419129 +0000
+++ SSLproxy-mod/pxyconn.c 2020-03-02 20:51:13.695265218 +0000
@@ -1160,6 +1160,13 @@
conn->child_dst_fd = ctx->dst_fd;
conn->thr->max_fd = MAX(conn->thr->max_fd, ctx->dst_fd);
// Do not return here, but continue and check term/enomem flags below
+
+ int mark = 10001;
+ if (setsockopt(ctx->dst_fd, SOL_SOCKET, SO_MARK, &mark, sizeof(mark)) < 0) {
+ log_err_level_printf(LOG_CRIT, "Error setting fwmark: %s (%i)\n", strerror(errno), errno);
+ goto out;
+ }
+
out:
// @attention Do not use ctx->conn here, ctx may be uninitialized
// @attention Call pxy_conn_free() directly, not pxy_conn_term() here
@@ -1244,6 +1251,13 @@
pxy_conn_term(ctx, 1);
return -1;
}
+
+ int marklo = 10001;
+ if (setsockopt(ctx->dst_fd, SOL_SOCKET, SO_MARK, &marklo, sizeof(marklo)) < 0) {
+ log_err_level_printf(LOG_CRIT, "Error setting fwmark: %s (%i)\n", strerror(errno), errno);
+ return -1;
+ }
+
// printf(3): "snprintf() will write at most size-1 of the characters (the size'th character then gets the terminating NULL)"
// So, +1 for NULL
But the five first packets are not marked, I believe that I am not defining the mark in the correct place.
Connection log
[Mon Mar 2 20:45:17 2020] TEST:IN= OUT=eth0 SRC=192.168.137.111 DST=93.184.216.34 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=53376 DF PROTO=TCP SPT=48466 DPT=443 WINDOW=42340 RES=0x00 SYN URGP=0 [Mon Mar 2 20:45:17 2020] TEST:IN= OUT=eth0 SRC=192.168.137.111 DST=93.184.216.34 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=53377 DF PROTO=TCP SPT=48466 DPT=443 WINDOW=21 RES=0x00 ACK URGP=0 [Mon Mar 2 20:45:17 2020] TEST:IN= OUT=eth0 SRC=192.168.137.111 DST=93.184.216.34 LEN=569 TOS=0x00 PREC=0x00 TTL=64 ID=53378 DF PROTO=TCP SPT=48466 DPT=443 WINDOW=21 RES=0x00 ACK PSH URGP=0 [Mon Mar 2 20:45:17 2020] TEST:IN= OUT=eth0 SRC=192.168.137.111 DST=93.184.216.34 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=53379 DF PROTO=TCP SPT=48466 DPT=443 WINDOW=22 RES=0x00 ACK URGP=0 [Mon Mar 2 20:45:17 2020] TEST:IN= OUT=eth0 SRC=192.168.137.111 DST=93.184.216.34 LEN=103 TOS=0x00 PREC=0x00 TTL=64 ID=53380 DF PROTO=TCP SPT=48466 DPT=443 WINDOW=22 RES=0x00 ACK PSH URGP=0 [Mon Mar 2 20:45:17 2020] TEST:IN= OUT=eth0 SRC=192.168.137.111 DST=93.184.216.34 LEN=97 TOS=0x00 PREC=0x00 TTL=64 ID=53381 DF PROTO=TCP SPT=48466 DPT=443 WINDOW=22 RES=0x00 ACK PSH URGP=0 MARK=0x2711 [Mon Mar 2 20:45:17 2020] TEST:IN= OUT=eth0 SRC=192.168.137.111 DST=93.184.216.34 LEN=788 TOS=0x00 PREC=0x00 TTL=64 ID=53382 DF PROTO=TCP SPT=48466 DPT=443 WINDOW=22 RES=0x00 ACK PSH URGP=0 MARK=0x2711 [Mon Mar 2 20:45:18 2020] TEST:IN= OUT=eth0 SRC=192.168.137.111 DST=93.184.216.34 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=53383 DF PROTO=TCP SPT=48466 DPT=443 WINDOW=22 RES=0x00 ACK URGP=0 MARK=0x2711 [Mon Mar 2 20:45:18 2020] TEST:IN= OUT=eth0 SRC=192.168.137.111 DST=93.184.216.34 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=53384 DF PROTO=TCP SPT=48466 DPT=443 WINDOW=23 RES=0x00 ACK URGP=0 MARK=0x2711
===================================================
iproute2 rules:
[root@centos ~]# ip rule list 0: from all lookup local 100: from all fwmark 0x2711 lookup link_eth1 32766: from all lookup main 32767: from all lookup default
routing table:
[root@centos ~]# ip route show table link_eth1 default via 200.200.200.70 dev eth1
The srvdst connection end sees all the packets of a connection. So I think you can use the connection handling functions for srvdst. I see two possibilities:
-
You can insert your setsockopt() code at the end of proto*_conn_connect() functions, right before the
return 0lines. I think this would make sure that all packets are marked. But, there is an important issue with this option: I can assure you that sslproxy is going to crash eventually due to multithreading issues. Please read the comments in prototcp_conn_connect() for explanation. -
You can insert your setsockopt() code in the beginning of proto*_bev_eventcb_connected_srvdst() and proto*_bev_writecb_srvdst() functions. Please read the comments in pxy_connect_srvdst() for the explanation of why you need to use the writecb function too. This options is completely thread-safe. But since those cb functions are called after the connection is established with the server, you may not be able to mark the packets until that point in the lifetime of a connection.
So, the decision is yours. I suggest that you try the first option first, to see if it is going to work, but don't rely on this option as your actual solution, because, I repeat, sslproxy will certainly crash with the option 1, eventually. Afterwards, you can try the second option, if you don't need the first couple of packets exhanged during connection establishment between server and sslproxy, in which case you may miss not only tcp but also ssl handshake packets.
If you don't need handshake packets (i.e. if the second options is fine with you), you can perhaps use the cb functions for parent's src connection end too, so that the packets diverted to the listening programs would be marked as well. This would be thread-safe too. For example, you can insert your code into prototcp_enable_src().
HTH,
The multithreading issues I mention in my last comment has been solved with SSLproxy 0.8.0. So sslproxy shouldn't crash with the option 1 there.