sshttpd[2818]: sshttp::loop::NS_Socket::dstaddr::getsockopt:Protocol not available

[finjulhich]

1. On trying to connect over https towards port 1081 that my sshttpd is listening to, /var/log/messages shows me this failure. I've built sshttpd from master branch. linux kernel 4.10.10

2. I was surprised to see some random ip address in netstat tcp 0 0 192.168.1.5:1081 123.152.106.180:57183 SYN_RECV Would iptables the way to block this 123.152.106.180 entirely?

[stealth]

Error message looks like your kernel has no transparent proxy support enabled?

[warewolf]

I just ran into the same problem :/

I can confirm that I have transparent proxying enabled in the kernel, and the modules are loaded.

[root@test netfilter]# lsmod | grep -i prox
xt_TPROXY              20480  0
nf_defrag_ipv6         36864  2 xt_socket,xt_TPROXY
nf_defrag_ipv4         16384  3 xt_socket,nf_conntrack_ipv4,xt_TPROXY
[root@test netfilter]# lsmod | grep -i sock
xt_socket              16384  0
nf_socket_ipv4         16384  1 xt_socket
nf_socket_ipv6         16384  1 xt_socket
nf_defrag_ipv6         36864  2 xt_socket,xt_TPROXY
nf_defrag_ipv4         16384  3 xt_socket,nf_conntrack_ipv4,xt_TPROXY
[root@test netfilter]#

Here's the output of strace -p `pidof sshttpd` when it's running with one thread (-n 1)

strace: Process 16599 attached
restart_syscall(<... resuming interrupted poll ...>) = 0
poll([{fd=-1}, {fd=-1}, {fd=-1}, {fd=-1}, {fd=4, events=POLLIN|POLLOUT}], 5, 1000) = 0 (Timeout)
poll([{fd=-1}, {fd=-1}, {fd=-1}, {fd=-1}, {fd=4, events=POLLIN|POLLOUT}], 5, 1000) = 0 (Timeout)
poll([{fd=-1}, {fd=-1}, {fd=-1}, {fd=-1}, {fd=4, events=POLLIN|POLLOUT}], 5, 1000) = 1 ([{fd=4, revents=POLLIN}])
accept4(4, {sa_family=AF_INET, sa_data="\304\\\n\0070\217"}, [8->16], SOCK_NONBLOCK) = 7
setsockopt(7, SOL_TCP, TCP_NODELAY, [1], 4) = 0
accept4(4, 0x7ffe56345e10, [16], SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=-1}, {fd=-1}, {fd=-1}, {fd=-1}, {fd=4, events=POLLIN|POLLOUT}, {fd=-1}, {fd=-1}, {fd=7, events=POLLIN}], 8, 1000) = 1 ([{fd=7, revents=POLLIN}])
getsockopt(7, SOL_IP, 0x50 /* IP_??? */, 0x7ffe56345e20, [16]) = -1 ENOENT (No such file or directory)
close(7)                                = 0
getpid()                                = 16599
sendto(3, "<27>Mar 13 19:36:16 sshttpd[1659"..., 106, MSG_NOSIGNAL, NULL, 0) = 106
poll([{fd=-1}, {fd=-1}, {fd=-1}, {fd=-1}, {fd=4, events=POLLIN|POLLOUT}, {fd=-1}, {fd=-1}], 7, 1000) = 0 (Timeout)
poll([{fd=-1}, {fd=-1}, {fd=-1}, {fd=-1}, {fd=4, events=POLLIN|POLLOUT}], 5, 1000) = 0 (Timeout)
poll([{fd=-1}, {fd=-1}, {fd=-1}, {fd=-1}, {fd=4, events=POLLIN|POLLOUT}], 5, 1000) = 0 (Timeout)

Weird thing, is I had this working under the same OS; but I'm completely lost on what I broke when I reinstalled a new copy of the same OS.

[warewolf]

It is something kernel related, sshttpd doesn't work on Fedora 27's 4.15.7-300.fc27.x86_64 but does work on 4.13.9-300.fc27.x86_64.

[stealth]

Whats the exact sshttp commandline, iptables rules and commands you try to get the connect?

[stealth]

strace looks like the getsockopt() is called on a socket that was not slipped through one of the netfilter rules, but received via a direct connect to one of the "hidden" ports (-S or -H).

[warewolf]

@stealth literally all I did to make it work again was use an older kernel, I'm suspecting a kernel bug.

I did telnet 192.168.100.158 22, and would get an sshttpd that would get stuck in disk io wait, unkillable. I would have to reboot to clear it. I'm using this to multiplex Elasticsearch's http interface on port 22 with SSH. SSHD is bound to 222. Command line:

/usr/sbin/sshttpd -n 1 -S 222 -H 9200 -L 22 -l 192.168.100.158 -U root -R /var/sshttp

nf-setup

#!/bin/sh

# sshttp netfilter rules
#

# If you mux SSH/SMTP (rather than HTTP), then HTTP_PORT is your
# alternate SMTP port. e.g. 2525 and sshttp needs to be started with
# '-L 25 -H 2525'

DEV=en0

# The ports you want to mux:
# -S <port> -H <port> and any other -N SNI:<ports> (in case of HTTPS)
# do NOT add the -L port here
# standard SSH / HTTP mux looks like this (sshttpd -S 22 -H 8080 -L 80)
PORTS="222 9200"

# a SSH / HTTPS mux with https server on port 4433 and a drops
# on port 7350 looks like this (sshttpd -S 22 -H 4433 -L 443 -N drops.v2:7350)
#PORTS="22 4433 7350"

# SNI-only mux without SSH (sshttpd -S 0 -H 4433 -L 443 -N drops.v2:7350)
#PORTS="4433 7350"

#if it clashes with complex NATing rules, try this
iptables -t mangle -F
iptables -t nat -F
iptables -t raw -F

modprobe nf_conntrack_ipv4 || true
iptables -t mangle -N DIVERT || true

echo "Using network device $DEV"

for p in $PORTS; do
        echo "Setting up port $p ..."

        # block direct access from outside
        iptables -A INPUT -i $DEV -p tcp --dport $p -j DROP

        # and divert anything back to sshttpd that comes from the muxed services
        # so sshttpd can see it
        iptables -t mangle -A OUTPUT -p tcp -o $DEV --sport $p -j DIVERT
done

iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT

iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT

ip rule add fwmark 1 lookup 123 || true
ip route add local 0.0.0.0/0 dev lo table 123

[stealth]

Its a bit unusual to use port 22 for -L, since its that port that would also serve the web pages at the end.

Also, you are showing me nf-setup, but if you want to use tproxy, you should use nf-tproxy and the -T switch for sshttpd (its missing in the help).

So, do you want to use the tproxy mode? If not, nf-setup is fine and it really looks like kernel issue which I cant help with

[stealth]

fixed