ss-tproxy icon indicating copy to clipboard operation
ss-tproxy copied to clipboard

执行 nft list ruleset 报警告

Open ir1ka opened this issue 1 year ago • 27 comments

执行 nft 看到如下报错。但执行 iptablesip6tables 没有报错,是正常的吗? 命令如下:

sudo nft list ruleset 1>/dev/null
sudo iptables -nvL -t nat
sudo iptables -nvL -t mangle
sudo ip6tables -nvL -t nat
sudo ip6tables -nvL -t mangle
# Warning: table ip nat is managed by iptables-nft, do not touch!
# Warning: table ip mangle is managed by iptables-nft, do not touch!
# Warning: table ip6 mangle is managed by iptables-nft, do not touch!
# Warning: XT target TPROXY not found
# Warning: XT target TPROXY not found
Warning: Extension owner is not supported, missing kernel module?
Warning: Extension owner is not supported, missing kernel module?
Warning: Extension owner is not supported, missing kernel module?
# Warning: table ip6 nat is managed by iptables-nft, do not touch!
# Warning: XT target REDIRECT not found
# Warning: XT target REDIRECT not found
Warning: Extension owner is not supported, missing kernel module?
Warning: Extension owner is not supported, missing kernel module?
# Warning: XT target REDIRECT not found
Warning: Extension owner is not supported, missing kernel module?
Warning: Extension owner is not supported, missing kernel module?
# Warning: XT target REDIRECT not found
# Warning: XT target SNAT not found

环境: OS: VyOS 1.5-rolling-202409080006 容器:Alpine Linux 3.20.3 ss-tproxy 版本:ss-tproxy v4.8.3 (2024-07-24)

ir1ka avatar Sep 22 '24 13:09 ir1ka

提示很明显了,安装缺失的 iptables 内核模块。Google 下有解决方案的。

zfl9 avatar Sep 22 '24 14:09 zfl9

另外,在使用 iptables-nft 后端时,不要执行任何 nft 指令(正如给出的警告所说)。

像往常一样,使用 iptables、ip6tables 即可,内核中将自动使用 nftables 作为后端。

zfl9 avatar Sep 22 '24 14:09 zfl9

另外,执行 ss-tproxy restart 时是否存在任何报错?

zfl9 avatar Sep 22 '24 14:09 zfl9

检查了一下已加载内核模块,所有报错相关的模块都以加载:

vyos@Gateway:~$ lsmod | grep xt_
xt_MASQUERADE          16384  0
xt_TPROXY              12288  4
nf_tproxy_ipv6         16384  1 xt_TPROXY
nf_tproxy_ipv4         16384  1 xt_TPROXY
xt_mark                12288  2
xt_connmark            12288  8
xt_set                 20480  4
xt_addrtype            12288  12
xt_nat                 12288  2
xt_conntrack           12288  12
xt_REDIRECT            16384  8
xt_owner               12288  14
xt_tcpudp              16384  0
ip_set                 65536  2 xt_set,ip_set_hash_net
nf_nat                 57344  6 nft_nat,xt_nat,nft_masq,nft_chain_nat,xt_MASQUERADE,xt_REDIRECT
nf_conntrack          192512  10 xt_conntrack,nf_nat,nfnetlink_cthelper,nft_ct,nft_nat,xt_nat,xt_connmark,nft_masq,xt_MASQUERADE,xt_REDIRECT
nf_defrag_ipv6         24576  2 nf_conntrack,xt_TPROXY
nf_defrag_ipv4         12288  2 nf_conntrack,xt_TPROXY
x_tables               61440  13 xt_conntrack,nft_compat,xt_tcpudp,xt_addrtype,xt_nat,xt_owner,xt_set,xt_TPROXY,xt_connmark,ip_tables,xt_MASQUERADE,xt_REDIRECT,xt_mark

我尝试重启下系统

ir1ka avatar Sep 22 '24 15:09 ir1ka

部分运行起来了,目前仅本机的 ipv4 能走透明代理,本机 ipv6、局域网内其他主机 ipv6 暂时没有走代码。并且局域网内的 ipv4 无法工作了。 继续调试。

ir1ka avatar Sep 22 '24 17:09 ir1ka

先排除 nft 规则的影响,在 ss-tproxy start 之前,nft list ruleset 最好是空的。

zfl9 avatar Sep 23 '24 07:09 zfl9

但是部署到路由器上的的话,没法保证 ss-tproxy start 前,nft list ruleset 为空呀,只能看看啥原因了

ir1ka avatar Sep 23 '24 10:09 ir1ka

pre_start 里面执行 nft flush ruleset ?

zfl9 avatar Sep 23 '24 10:09 zfl9

额,那网关的规则都搞没了呀,虽然能保证 nft 为空,但可能路由器的功能会异常吧

ir1ka avatar Sep 23 '24 10:09 ir1ka

看起来 sudo nft list ruleset 中报的错都来自于 ip6 的 表。

看起来这个报错是因为这个:https://forum.kaspersky.com/topic/nft-utility-errors-xt-target-tproxy-not-found-caused-by-wtpntp-task-kes-for-linux-37528/ 因为如果我执行 sudo nft list table ip6 natsudo nft list table ip6 mangle 就没有这个报错。

ir1ka avatar Sep 24 '24 11:09 ir1ka

ipv4 内网访问搞定了,需要放行 fwmark 等于 0x2333 的包。从 log 看到,0x2333 的包被默认丢掉了。 具体原因就是 ss-tproxy 会把需要代理的包通过 fwmark 和路由规则的方式路由到 lo,并进入 input 规则中,并且这些包不带 out if 信息,导致我设置的允许从 oif == lo 的规则没有生效。 通过加入规则通过所有 fwmark == 0x233 的包,ipv4 可以工作了。

ir1ka avatar Sep 24 '24 17:09 ir1ka

加了这几行 log,然后测试 ipv6(内网),命令 curl -6 ip.sb,只打印了如下 log:

Sep 25 03:53:19 kernel: [SSTP_RULE CONNMARK set]IN=eth0 OUT= MAC=24:4b:fe:d4:3d:a0:52:54:00:ed:65:b5:86:dd SRC=xxxx::yyyy DST=2606:4700:0020:0000:0000:0000:681a:0d1f LEN=72 TC=0 HOPLIMIT=64 FLOWLBL=462883 PROTO=TCP SPT=55566 DPT=80 WINDOW=64800 RES=0x00 SYN URGP=0
Sep 25 03:53:19 kernel: [SSTP_PREROUTING tcp]IN=eth0 OUT= MAC=24:4b:fe:d4:3d:a0:52:54:00:ed:65:b5:86:dd SRC=xxxx::yyyy DST=2606:4700:0020:0000:0000:0000:681a:0d1f LEN=72 TC=0 HOPLIMIT=64 FLOWLBL=462883 PROTO=TCP SPT=55566 DPT=80 WINDOW=64800 RES=0x00 SYN URGP=0
Sep 25 03:53:20 kernel: [SSTP_PREROUTING tcp]IN=pppoe0 OUT= MAC= SRC=2606:4700:0020:0000:0000:0000:681a:0d1f DST=xxxx::yyyy LEN=72 TC=0 HOPLIMIT=52 FLOWLBL=333531 PROTO=TCP SPT=80 DPT=55566 WINDOW=65535 RES=0x00 ACK SYN URGP=0
Sep 25 03:53:20 kernel: [SSTP_PREROUTING tcp]IN=eth0 OUT= MAC=24:4b:fe:d4:3d:a0:52:54:00:ed:65:b5:86:dd SRC=xxxx::yyyy DST=2606:4700:0020:0000:0000:0000:681a:0d1f LEN=60 TC=0 HOPLIMIT=64 FLOWLBL=462883 PROTO=TCP SPT=55566 DPT=80 WINDOW=1025 RES=0x00 ACK URGP=0
Sep 25 03:53:20 kernel: [SSTP_PREROUTING tcp]IN=eth0 OUT= MAC=24:4b:fe:d4:3d:a0:52:54:00:ed:65:b5:86:dd SRC=xxxx::yyyy DST=2606:4700:0020:0000:0000:0000:681a:0d1f LEN=129 TC=0 HOPLIMIT=64 FLOWLBL=462883 PROTO=TCP SPT=55566 DPT=80 WINDOW=1025 RES=0x00 ACK PSH URGP=0
Sep 25 03:53:20 kernel: [SSTP_PREROUTING tcp]IN=pppoe0 OUT= MAC= SRC=2606:4700:0020:0000:0000:0000:681a:0d1f DST=xxxx::yyyy LEN=60 TC=0 HOPLIMIT=52 FLOWLBL=333531 PROTO=TCP SPT=80 DPT=55566 WINDOW=8 RES=0x00 ACK URGP=0
Sep 25 03:53:20 kernel: [SSTP_PREROUTING tcp]IN=pppoe0 OUT= MAC= SRC=2606:4700:0020:0000:0000:0000:681a:0d1f DST=xxxx::yyyy LEN=687 TC=0 HOPLIMIT=52 FLOWLBL=333531 PROTO=TCP SPT=80 DPT=55566 WINDOW=8 RES=0x00 ACK PSH URGP=0
Sep 25 03:53:20 kernel: [SSTP_PREROUTING tcp]IN=pppoe0 OUT= MAC= SRC=2606:4700:0020:0000:0000:0000:681a:0d1f DST=xxxx::yyyy LEN=65 TC=0 HOPLIMIT=52 FLOWLBL=333531 PROTO=TCP SPT=80 DPT=55566 WINDOW=8 RES=0x00 ACK PSH URGP=0
Sep 25 03:53:20 kernel: [SSTP_PREROUTING tcp]IN=eth0 OUT= MAC=24:4b:fe:d4:3d:a0:52:54:00:ed:65:b5:86:dd SRC=xxxx::yyyy DST=2606:4700:0020:0000:0000:0000:681a:0d1f LEN=60 TC=0 HOPLIMIT=64 FLOWLBL=462883 PROTO=TCP SPT=55566 DPT=80 WINDOW=1022 RES=0x00 ACK URGP=0
Sep 25 03:53:20 kernel: [SSTP_PREROUTING tcp]IN=eth0 OUT= MAC=24:4b:fe:d4:3d:a0:52:54:00:ed:65:b5:86:dd SRC=xxxx::yyyy DST=2606:4700:0020:0000:0000:0000:681a:0d1f LEN=60 TC=0 HOPLIMIT=64 FLOWLBL=462883 PROTO=TCP SPT=55566 DPT=80 WINDOW=1022 RES=0x00 ACK FIN URGP=0
Sep 25 03:53:20 kernel: [SSTP_PREROUTING tcp]IN=pppoe0 OUT= MAC= SRC=2606:4700:0020:0000:0000:0000:681a:0d1f DST=xxxx::yyyy LEN=60 TC=0 HOPLIMIT=52 FLOWLBL=333531 PROTO=TCP SPT=80 DPT=55566 WINDOW=8 RES=0x00 ACK FIN URGP=0
Sep 25 03:53:20 kernel: [SSTP_PREROUTING tcp]IN=eth0 OUT= MAC=24:4b:fe:d4:3d:a0:52:54:00:ed:65:b5:86:dd SRC=xxxx::yyyy DST=2606:4700:0020:0000:0000:0000:681a:0d1f LEN=60 TC=0 HOPLIMIT=64 FLOWLBL=462883 PROTO=TCP SPT=55566 DPT=80 WINDOW=1022 RES=0x00 ACK URGP=0

log 中,没有任何 matched 的 log,很奇怪。请问可能是什么原因呢?

规则:

# ip6tables -vL --line-numbers -t mangle
Chain PREROUTING (policy ACCEPT 645K packets, 358M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     645K  358M SSTP_PREROUTING  all  --  any    any     anywhere             anywhere

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 437K packets, 91M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     437K   91M SSTP_OUTPUT  all  --  any    any     anywhere             anywhere

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain SSTP_OUTPUT (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1    42235 6651K RETURN     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL
2        0     0 RETURN     all  --  any    any     anywhere             anywhere             ctdir REPLY
3     284K   60M RETURN     all  --  any    any     anywhere             anywhere             owner GID match proxy
4        0     0 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:domain ! owner GID match proxy_dns
5        0     0 RETURN     udp  --  any    any     anywhere             anywhere             udp dpt:domain ! owner GID match proxy_dns
6       24  1920 SSTP_RULE  tcp  --  any    any     anywhere             anywhere             tcp flags:FIN,SYN,RST,ACK/SYN
7        0     0 SSTP_RULE  udp  --  any    any     anywhere             anywhere             ctstate NEW,RELATED
8       20  1594 LOG        tcp  --  any    any     anywhere             anywhere             LOG level warn prefix "[SSTP_OUTPUT tcp]"
9        0     0 LOG        all  --  any    any     anywhere             anywhere             connmark match  0x2333 LOG level warn prefix "[SSTP_OUTPUT matched]"
10       0     0 MARK       all  --  any    any     anywhere             anywhere             connmark match  0x2333 MARK set 0x2333

Chain SSTP_PREROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     379K  190M RETURN     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL
2        0     0 RETURN     all  --  any    any     anywhere             anywhere             ctdir REPLY
3    40198 2894K SSTP_RULE  tcp  --  any    any     anywhere             anywhere             tcp dpt:!domain flags:FIN,SYN,RST,ACK/SYN ADDRTYPE match src-type !LOCAL
4        0     0 SSTP_RULE  udp  --  any    any     anywhere             anywhere             udp dpt:!domain ctstate NEW,RELATED ADDRTYPE match src-type !LOCAL
5      197 37445 LOG        tcp  --  any    any     anywhere             anywhere             LOG level warn prefix "[SSTP_PREROUTING tcp]"
6        0     0 LOG        all  --  any    any     anywhere             anywhere             connmark match  0x2333 LOG level warn prefix "[SSTP_PREROUTING matched]"
7        0     0 TPROXY     tcp  --  any    any     anywhere             anywhere             connmark match  0x2333 TPROXY redirect ::1:60080 mark 0x2333/0xffffffff
8        0     0 TPROXY     udp  --  any    any     anywhere             anywhere             connmark match  0x2333 TPROXY redirect ::1:60080 mark 0x2333/0xffffffff

Chain SSTP_RULE (4 references)
num   pkts bytes target     prot opt in     out     source               destination
1      137  9864 RETURN     all  --  any    any     anywhere             anywhere             match-set sstp_white6 dst ! match-set sstp_black6 dst
2    40085 2886K CONNMARK   all  --  any    any     anywhere             anywhere             CONNMARK set 0x2333
3        5   360 LOG        all  --  any    any     anywhere             anywhere             LOG level warn prefix "[SSTP_RULE CONNMARK set]"

ir1ka avatar Sep 24 '24 19:09 ir1ka

nft list ruleset 看看全部规则?


建议使用 raw 表的 -j TRACE 看日志,比手动加 LOG 更方便一些。


log 中,没有任何 matched 的 log,很奇怪。请问可能是什么原因呢?

确实很奇怪,中间也没有其他规则干扰,难道是 ipv6 的 connmark 模块有问题?

zfl9 avatar Sep 25 '24 02:09 zfl9

有点像 ipv6 的 connmark set 0x2333 没有生效

ir1ka avatar Sep 25 '24 09:09 ir1ka

TRACE 需要如何加规则?我用命令 ip6tables -t raw -A PREROUTING -p tcp -j TRACE,然后测试,没有看到人的 trace 的打印。此命令是否有误;

ir1ka avatar Sep 25 '24 10:09 ir1ka

你测试的 本机代理 还是 局域网代理 ?

我看 ipv4 已经搞定了(本机、局域网) ipv6 要不先测下本机代理?

先以 tcp 2001:4860:4860::8888 为测试用例,避免 log 过多。

ip6tables -t raw -I OUTPUT -p tcp -d 2001:4860:4860::8888 -j TRACE
ip6tables -t raw -I OUTPUT -p tcp -s 2001:4860:4860::8888 -j TRACE
ip6tables -t raw -I PREROUTING -p tcp -s 2001:4860:4860::8888 -j TRACE
ip6tables -t raw -I PREROUTING -p tcp -d 2001:4860:4860::8888 -j TRACE

然后本机执行 dig +tcp @2001:4860:4860::8888 google.com,观察 dmesg 日志。

zfl9 avatar Sep 25 '24 10:09 zfl9

测试的 ipv6 局域网代理,本地 ipv6 代理也不工作。均走的直连。 用 tcp dns 应该会转发到 chinadns-ng 吧?

ir1ka avatar Sep 25 '24 10:09 ir1ka

好像是哈,切换下 group 身份,sg proxy_dns 'dig +tcp @2001:4860:4860::8888 google.com',跳过 chinadns-ng。

zfl9 avatar Sep 25 '24 10:09 zfl9

打印的 log 如下,没有观察到 TRACE 的 log,难道是 VyOS 有问题?

Sep 25 19:55:19 kernel: [SSTP_RULE CONNMARK set]IN= OUT=pppoe0 SRC=xxxx::yyyy DST=2001:4860:4860:0000:0000:0000:0000:8888 LEN=80 TC=0 HOPLIMIT=64 FLOWLBL=502873 PROTO=TCP SPT=46469 DPT=53 WINDOW=64660 RES=0x00 SYN URGP=0
Sep 25 19:55:20 kernel: [SSTP_RULE CONNMARK set]IN= OUT=pppoe0 SRC=xxxx::yyyy DST=2001:4860:4860:0000:0000:0000:0000:8888 LEN=80 TC=0 HOPLIMIT=64 FLOWLBL=815264 PROTO=TCP SPT=45921 DPT=53 WINDOW=64660 RES=0x00 SYN URGP=0
Sep 25 19:55:20 kernel: [SSTP_RULE CONNMARK set]IN= OUT=pppoe0 SRC=xxxx::yyyy DST=2001:4860:4860:0000:0000:0000:0000:8888 LEN=80 TC=0 HOPLIMIT=64 FLOWLBL=599366 PROTO=TCP SPT=40217 DPT=53 WINDOW=64660 RES=0x00 SYN URGP=0

规则列表如下:

/ # ip6tables -vL --line-numbers -t raw
# Table `raw' contains incompatible base-chains, use 'nft' tool to list them.
Chain PREROUTING (policy ACCEPT 21365 packets, 17M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        3   245 TRACE      tcp  --  any    any     anywhere             dns.google
2       97 12843 TRACE      tcp  --  any    any     dns.google           anywhere

Chain OUTPUT (policy ACCEPT 8339 packets, 1241K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 TRACE      tcp  --  any    any     dns.google           anywhere
2       71  4842 TRACE      tcp  --  any    any     anywhere             dns.google

Chain vyos_global_rpfilter (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 RETURN     all  --  any    any     anywhere             anywhere
/ # ip6tables -vL --line-numbers -t mangle
Chain PREROUTING (policy ACCEPT 720K packets, 388M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     720K  388M SSTP_PREROUTING  all  --  any    any     anywhere             anywhere

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 495K packets, 100M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     495K  100M SSTP_OUTPUT  all  --  any    any     anywhere             anywhere

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain SSTP_OUTPUT (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1    47823 7587K RETURN     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL
2        0     0 RETURN     all  --  any    any     anywhere             anywhere             ctdir REPLY
3     333K   67M RETURN     all  --  any    any     anywhere             anywhere             owner GID match proxy
4       53  3180 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:domain ! owner GID match proxy_dns
5        0     0 RETURN     udp  --  any    any     anywhere             anywhere             udp dpt:domain ! owner GID match proxy_dns
6       41  3280 SSTP_RULE  tcp  --  any    any     anywhere             anywhere             tcp flags:FIN,SYN,RST,ACK/SYN
7        0     0 SSTP_RULE  udp  --  any    any     anywhere             anywhere             ctstate NEW,RELATED
8        0     0 MARK       all  --  any    any     anywhere             anywhere             connmark match  0x2333 MARK set 0x2333

Chain SSTP_PREROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     437K  213M RETURN     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL
2        0     0 RETURN     all  --  any    any     anywhere             anywhere             ctdir REPLY
3    40835 2940K SSTP_RULE  tcp  --  any    any     anywhere             anywhere             tcp dpt:!domain flags:FIN,SYN,RST,ACK/SYN ADDRTYPE match src-type !LOCAL
4        0     0 SSTP_RULE  udp  --  any    any     anywhere             anywhere             udp dpt:!domain ctstate NEW,RELATED ADDRTYPE match src-type !LOCAL
5        0     0 TPROXY     tcp  --  any    any     anywhere             anywhere             connmark match  0x2333 TPROXY redirect ::1:60080 mark 0x2333/0xffffffff
6        0     0 TPROXY     udp  --  any    any     anywhere             anywhere             connmark match  0x2333 TPROXY redirect ::1:60080 mark 0x2333/0xffffffff

Chain SSTP_RULE (4 references)
num   pkts bytes target     prot opt in     out     source               destination
1      144 10368 RETURN     all  --  any    any     anywhere             anywhere             match-set sstp_white6 dst ! match-set sstp_black6 dst
2    40732 2933K CONNMARK   all  --  any    any     anywhere             anywhere             CONNMARK set 0x2333
3      652 47080 LOG        all  --  any    any     anywhere             anywhere             LOG level warn prefix "[SSTP_RULE CONNMARK set]"

ir1ka avatar Sep 25 '24 11:09 ir1ka

你这个内核怕是有点问题,第一次见这种情况。

会不会是 ipv6 的 conntrack 内核模块未加载?

zfl9 avatar Sep 25 '24 12:09 zfl9

nft 规则列表如下,移除了隐私相关以及 ipv4 规则。

:~$ sudo nft list ruleset
# Warning: table ip nat is managed by iptables-nft, do not touch!
table inet mangle {
        chain FORWARD {
                type filter hook forward priority mangle; policy accept;
        }
}
table ip6 raw {
        chain VYOS_TCP_MSS {
                type filter hook forward priority raw; policy accept;
                oifname "pppoe0" tcp flags syn / syn,rst tcp option maxseg size set rt mtu
        }

        chain vyos_global_rpfilter {
                return
        }

        chain vyos_rpfilter {
                type filter hook prerouting priority raw; policy accept;
                counter packets 722306 bytes 389764469 jump vyos_global_rpfilter
        }

        chain VYOS_PREROUTING_HOOK {
                type filter hook prerouting priority raw; policy accept;
        }

        chain PREROUTING {
                type filter hook prerouting priority raw; policy accept;
                ip6 daddr 2001:4860:4860::8888 meta l4proto tcp counter packets 3 bytes 245 meta nftrace set 1
                ip6 saddr 2001:4860:4860::8888 meta l4proto tcp counter packets 97 bytes 12843 meta nftrace set 1
        }

        chain OUTPUT {
                type filter hook output priority raw; policy accept;
                ip6 saddr 2001:4860:4860::8888 meta l4proto tcp counter packets 0 bytes 0 meta nftrace set 1
                ip6 daddr 2001:4860:4860::8888 meta l4proto tcp counter packets 71 bytes 4842 meta nftrace set 1
        }
}
table inet vrf_zones {
        map ct_iface_map {
                typeof iifname : ct zone
        }

        chain vrf_zones_ct_in {
                type filter hook prerouting priority raw; policy accept;
        }

        chain vrf_zones_ct_out {
                type filter hook output priority raw; policy accept;
        }
}
# Warning: table ip6 mangle is managed by iptables-nft, do not touch!
table ip6 mangle {
        chain SSTP_PREROUTING {
                fib daddr type local counter packets 437076 bytes 212778279 return
                ct direction reply counter packets 0 bytes 0 return
                tcp flags syn / fin,syn,rst,ack tcp dport != 53 fib saddr type != local counter packets 40839 bytes 2940408 jump SSTP_RULE
                udp dport != 53 ct state new,related fib saddr type != local counter packets 0 bytes 0 jump SSTP_RULE
                meta l4proto tcp ct mark 0x2333 counter packets 0 bytes 0 xt target "TPROXY"
                meta l4proto udp ct mark 0x2333 counter packets 0 bytes 0 xt target "TPROXY"
        }

        chain SSTP_OUTPUT {
                fib daddr type local counter packets 47853 bytes 7590570 return
                ct direction reply counter packets 0 bytes 0 return
                skgid 38888 counter packets 332805 bytes 67225096 return
                tcp dport 53 skgid != 39999 counter packets 53 bytes 3180 return
                udp dport 53 skgid != 39999 counter packets 0 bytes 0 return
                tcp flags syn / fin,syn,rst,ack counter packets 41 bytes 3280 jump SSTP_RULE
                meta l4proto udp ct state new,related counter packets 0 bytes 0 jump SSTP_RULE
                ct mark 0x2333 counter packets 0 bytes 0 meta mark set 0x2333
        }

        chain SSTP_RULE {
                xt match "set" xt match "set" counter packets 144 bytes 10368 return
                counter packets 40736 bytes 2933320 ct mark set 0x2333
                counter packets 656 bytes 47368 log prefix "[SSTP_RULE CONNMARK set]"
        }

        chain PREROUTING {
                type filter hook prerouting priority mangle; policy accept;
                counter packets 723774 bytes 390853563 jump SSTP_PREROUTING
        }

        chain OUTPUT {
                type route hook output priority mangle; policy accept;
                counter packets 495247 bytes 99745222 jump SSTP_OUTPUT
        }
}
# Warning: table ip6 nat is managed by iptables-nft, do not touch!
table ip6 nat {
        chain SSTP_PREROUTING {
                tcp flags syn / fin,syn,rst,ack tcp dport 53 fib saddr type != local counter packets 0 bytes 0 redirect to :5353
                udp dport 53 ct state new fib saddr type != local counter packets 0 bytes 0 redirect to :5353
        }

        chain SSTP_OUTPUT {
                tcp flags syn / fin,syn,rst,ack tcp dport 53 skgid != 38888 skgid != 39999 counter packets 0 bytes 0 redirect to :5353
                udp dport 53 ct state new skgid != 38888 skgid != 39999 counter packets 0 bytes 0 redirect to :5353
        }

        chain SSTP_POSTROUTING {
                ip6 saddr != ::1 ip6 daddr ::1 counter packets 0 bytes 0 snat to ::1
        }

        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                counter packets 0 bytes 0 jump SSTP_PREROUTING
        }

        chain OUTPUT {
                type nat hook output priority dstnat; policy accept;
                counter packets 0 bytes 0 jump SSTP_OUTPUT
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                counter packets 0 bytes 0 jump SSTP_POSTROUTING
        }
}
table ip6 vyos_filter {
        chain VYOS_IPV6_FORWARD_filter {
                type filter hook forward priority filter; policy accept;
                counter packets 8301 bytes 6776849 accept comment "FWD-filter default-action accept"
        }

        chain VYOS_IPV6_INPUT_filter {
                type filter hook input priority filter; policy accept;
                counter packets 3301 bytes 2520406 accept comment "INP-filter default-action accept"
        }

        chain VYOS_IPV6_OUTPUT_raw {
                type filter hook output priority raw; policy accept;
                counter packets 2955 bytes 444963 accept comment "OUT-raw default-action accept"
        }

        chain VYOS_IPV6_OUTPUT_filter {
                type filter hook output priority filter; policy accept;
                counter packets 2955 bytes 444963 accept comment "OUT-filter default-action accept"
        }

        chain VYOS_IPV6_PREROUTING_raw {
                type filter hook prerouting priority raw; policy accept;
                counter packets 11604 bytes 9311787 accept comment "PRE-raw default-action accept"
        }

        chain VYOS_FRAG6_MARK {
                type filter hook prerouting priority -450; policy accept;
                exthdr frag exists meta mark set 0x000ffff1 return
        }
}
table bridge vyos_filter {
        chain VYOS_FORWARD_filter {
                type filter hook forward priority filter; policy accept;
                counter packets 0 bytes 0 accept comment "FWD-filter default-action accept"
        }

        chain VYOS_INPUT_filter {
                type filter hook input priority filter; policy accept;
                counter packets 0 bytes 0 accept comment "INP-filter default-action accept"
        }

        chain VYOS_OUTPUT_filter {
                type filter hook output priority filter; policy accept;
                counter packets 0 bytes 0 accept comment "OUT-filter default-action accept"
        }

        chain VYOS_PREROUTING_filter {
                type filter hook prerouting priority filter; policy accept;
                counter packets 0 bytes 0 accept comment "PRE-filter default-action accept"
        }
}
table ip6 vyos_mangle {
        chain VYOS_PBR6_PREROUTING {
                type filter hook prerouting priority mangle; policy accept;
        }

        chain VYOS_PBR6_POSTROUTING {
                type filter hook postrouting priority mangle; policy accept;
        }
}

ir1ka avatar Sep 25 '24 12:09 ir1ka

你这个内核怕是有点问题,第一次见这种情况。

会不会是 ipv6 的 conntrack 内核模块未加载?

有可能,还需要排除 VyOS 本身的问题。

ir1ka avatar Sep 25 '24 12:09 ir1ka

lsmod | grep conntrack 看看有哪些

zfl9 avatar Sep 25 '24 12:09 zfl9

执行结果

:~$ lsmod | grep conntrack
xt_conntrack           12288  12
nf_conntrack          192512  9 xt_conntrack,nf_nat,nfnetlink_cthelper,nft_ct,nft_nat,xt_nat,xt_connmark,nft_masq,xt_REDIRECT
nf_defrag_ipv6         24576  2 nf_conntrack,xt_TPROXY
nf_defrag_ipv4         12288  2 nf_conntrack,xt_TPROXY
x_tables               61440  14 xt_conntrack,nft_compat,xt_TRACE,xt_LOG,xt_tcpudp,xt_addrtype,xt_nat,xt_owner,xt_set,xt_TPROXY,xt_connmark,ip_tables,xt_REDIRECT,xt_mark

ir1ka avatar Sep 25 '24 12:09 ir1ka

看规则是没有问题的,ss-tproxy 中的 iptables 和 ip6tables 代码是复用的(同一个函数)。

根据上面一系列反馈,核心问题还是 ip6 的 connmark set 未生效(不知道是这个模块问题还是内核配置少了什么)。

zfl9 avatar Sep 25 '24 12:09 zfl9

IPv6 代理失效(connmark set 没有工作)有没有可能是流量被硬件 offload 了?

ir1ka avatar Sep 28 '24 11:09 ir1ka

重装了一下 VyOS,重新做了所有配置,新增了 IPv6 的防火墙规则,目前本地及内网设备的 IPv6 和 IPv4 代理均已正常工作。

还需要确认之前 IPv6 代理不能工作的原因,以避免以后 IPv6 代理失效。

ir1ka avatar Sep 29 '24 12:09 ir1ka