tunctl with gen_udp

[SergejJurecko]

I'm trying to create two interfaces, have an erlang process for each and use gen_udp to communicate between them. But on OSX it always crashes. One process is able to send over gen_udp, the other one not. Do you have any insight why the attached code does not work?

Exception: ** Generic server <0.37.0> terminating ** Last message in was {'EXIT',<0.35.0>, {{badmatch,{error,ehostunreach}}, [{tt,route2,1, [{file,"src/tt.erl"},{line,50}]}]}}

Code:

-module(tt).
-compile(export_all).

s() ->
    IP1 = {192,168,100,100},
    IP2 = {192,168,100,101},
    Pid1 = spawn(fun() -> proc(IP1,IP2,0,{0,0}) end), %{1000000,1}
    Pid2 = spawn(fun() -> proc(IP2,IP1,1,{0,0}) end),
    Pid1 ! {peerlist,[Pid2]},
    Pid2 ! {peerlist,[Pid1]},
    ok.


-record(dev,{name, dev, ip, index, peers = [], delay = {0,0},
    online = true, udp,ip2,
    in_buf = [], in_reversed = [], in_unreversed = [],
    out_buf = [], out_reversed = [], out_unreversed = []}).

proc(IP1,IP2,N,Delay) ->
    Nm = "tap"++integer_to_list(N),
    {ok, Dev} = tuncer:create(Nm, [tap, no_pi, {active, true}]),
    ok = tuncer:up(Dev, IP1),
    io:format("Network up ~p~n",[IP1]),
    {ok,Udp} = gen_udp:open(5000,[binary,{ip,IP1},{active,true}]),
    R = #dev{dev = Dev, ip = IP1, index = N, udp = Udp, ip2 = IP2,delay = Delay},
    erlang:send_after(2000,self(),send),
    route2(R).


route2(R) ->
    receive
            {tuntap, _Dev, Data} ->
                    io:format("rec tuntap ~p~n",[self()]),
                    [P ! Data || P <- R#dev.peers],
                    route2(R);
            <<_/binary>> = Bin ->
                    io:format("rec bin ~p~n",[self()]),
                    tuncer:send(R#dev.dev,Bin),
                    route2(R);
            {udp, _Socket, _IP, _InPortNo, Packet} ->
                    io:format("Received udp from time=~p, now=~p~n",[binary_to_term(Packet),os:timestamp()]),
                    self() ! send,
                    route2(R);
            {peerlist,L} ->
                    route2(R#dev{peers = L});
            send ->
                    io:format("Sending ~p to=~p~n",[self(),R#dev.ip2]),
                    ok = gen_udp:send(R#dev.udp,R#dev.ip2,5000,term_to_binary(os:timestamp())),
                    io:format("Sent to=~p~n",[R#dev.ip2]),
                    route2(R)
    end.

[msantos]

I think the {error,ehostunreach} is happening because of the way routing is handled when both interfaces are local. The kernel is generating an ICMP host unreachable because it doesn't know of a path to get from 192.168.100.100 to 192.168.100.101 using tap0. I don't have an OS X system to test on, so this is just a guess.

I tried your code on Ubuntu and FreeBSD and didn't get the host unreachable error. Creating the interfaces in the repl:

1> {ok,D0} = tuncer:create(<<"tap0">>, [tap,no_pi,{active,true}]).
{ok,<0.42.0>}
2> {ok,D1} = tuncer:create(<<"tap1">>, [tap,no_pi,{active,true}]).
{ok,<0.47.0>}
3> tuncer:up(D0, {192,168,100,100}).
ok
4> tuncer:up(D1, {192,168,100,101}).
ok
5> flush().
% a bunch of ARP and ICMPv6 neighbour discovery packets

Then in another shell:

$ nc -s 192.168.100.100 192.168.100.101 22
SSH-2.0-OpenSSH_6.6.1_hpn13v11 FreeBSD-20140420

Back in the erlang repl:

6> flush().
ok
% no packets

So since both tap interfaces are local, the routing is short circuited and goes directly rather than via the tap devices despite the socket being bound to the ip address.

[SergejJurecko]

Oh that's a shame. It kind of ruins my plans of using tunctl for running erlang clusters locally and simulating network conditions (delays,packetloss,splits). Thanks for the help.

[joagre]

I realised the same when fiddling around with github/anond and did go down the VM path instead.

/Joakim

On 2015-01-25 18:11, Sergej Jurečko wrote:

Oh that's a shame. It kind of ruins my plans of using tunctl for running erlang clusters locally and simulating network conditions (delays,packetloss,splits)

— Reply to this email directly or view it on GitHub https://github.com/msantos/tunctl/issues/4#issuecomment-71381533.

[msantos]

@SergejJurecko Basically the 192.168.100.100 interface is the gateway to the 192.168.100.0/24 network. If a packet is sent to, e.g., 192.168.100.33, the packet will be sent via the tunnel. Since this is a tap interface, the erlang code will have to handle arp, etc. lansim seems to work this way:

https://chromium.googlesource.com/chromiumos/third_party/autotest/+/master/client/deps/lansim/control

The other way is to use a tun device which will create a point to point device but you'll still need to handle the IP headers.

Both of these are probably going too far down the rabbit hole if you want to simulate network failures on the loopback. I think it would be better if there were a higher level library, something like Joakim's anond, that let you programmatically set up the network.

@joagre now that is cool! Too bad it didn't work out but that is what experiments are for. So did each node end up running inside a VM or does each node's interface live in a network namespace like mininet?