openmptcprouter icon indicating copy to clipboard operation
openmptcprouter copied to clipboard

Published 20 hours ago • verified publisher icon Ysurac

Multipath not restored after interface is restarted, OMR becomes single path TCP router.

Open ioogithub opened this issue 2 years ago • 60 comments

Expected Behavior

Multipath should be restored to the same state after omr-tracker stops and starts an interface with all wan connections used.

Current Behavior

Multipath is broken after omr-tracker restarts an interface, traffic now only uses one wan, OpenMPTCProuter becomes OpenSPTCProuter (single path TCP router).

Steps to Reproduce the Problem

  1. Port forwarding is using v2ray and is working as expected, traffic uses both wan1 and wan2 interfaces.
  2. Start a file upload.
  3. Observe o the bandwidth page that OMR is correctly using all wan connections (wan1 and wan2): https://ibb.co/2YZDzBB
  4. OMR-tracker switches wan2 interface off (due to ping or error):
Thu Aug 24 17:54:56 2023 user.notice post-tracking-post-tracking: wan2 (eth2) switched off because check error and ping from 10.0.0.202 error (9.9.9.9,1.0.0.1,114.114.115.115)
Thu Aug 24 17:54:56 2023 user.notice post-tracking-post-tracking: Delete default route to x.x.x.x via y.y.y.y dev eth2
  1. OMR-tracker switches wan2 interface back on:
Thu Aug 24 18:01:45 2023 user.notice post-tracking-post-tracking: wan2 (eth2) switched up
Thu Aug 24 18:01:47 2023 user.notice post-tracking-post-tracking: Interface route not yet set, set route ip r add default via y.y.y.y dev eth2 metric 4
Thu Aug 24 18:01:47 2023 user.notice post-tracking-post-tracking: New public ip detected for wan2 (eth2): x.x.x.x
Thu Aug 24 18:01:47 2023 user.notice post-tracking-post-tracking: Reload MPTCP for eth2

  1. Start a new upload, traffic now only uses wan1, router is not Multi-path any longer: https://ibb.co/94QZJLq

  2. Router never recovers from this state. Start another upload 1 hour later and it still only uses wan1.

  3. ip r shows there are routes and default routes for both wan1 and wan2 but MPTCP refuses to use wan2 after OM-tracker restarts it.

Possible Solution 1

I tried two steps to fix the problem:

  1. First I tried resetting using this command: /etc/init.d/openmptcprouter-vps restart
  2. When I observelogread -fon OMR and journalctl -f on VPS I do not see any log events after this comamnd! This command executed and exits but it didn't do anything observable from the logs

Possible Solution 2:

  1. Click Save and Apply from the wizard page.
  2. This ultimately fixes the problem and traffic starts using wan1 and wan2 again however this is not a good solution as it majorly disrupts the network and is manual intervention.
  3. I can see from the log above: Thu Aug 24 18:01:47 2023 user.notice post-tracking-post-tracking: Reload MPTCP for eth2 I guess there is where the bug is, Reload MPTCP is not properly restoring the MPTCP bond. Is there a way to get more information on what MPTCP is doing here, is there any debug mode?
  • The best solution would be to have OMR fix itself properly after the omr-tracker event.
  • Is there an temporary solution where I can detect if OMR-tracker runs and then run a comamnd to restore the MPTCP bond properly?

I have been tracking this problem for a long time where I see the performance of the system degrade over time. I didn't have the knowledge until recently to isolate the bug and report it. I am available to test any solutions.

Context (Environment)

The issue is bad because it effectively breaks OMR. If only a single path is used after OMR-tracker then there is no purpose to run OMR at all. Also there is no way currently to recover from this problem.

Specifications

  • OpenMPTCProuter version: v0.59.1-5.4 r0+16594-ce92de8c8c
  • OpenMPTCProuter VPS version: VPS 0.1028
  • OpenMPTCProuter VPS provider:
  • OpenMPTCProuter platform: x86_64)

ioogithub avatar Aug 24 '23 23:08 ioogithub

I have more information. I can reproduce the bug, not with omr-tracker but by stopping the interface (wan2) and starting again. After I start it, OMR never uses it for MPTCP so it becomes single path router. Here is the log wan2 started again:

Thu Aug 24 19:54:50 2023 kern.info kernel: [23103.714924] igc 0000:04:00.0 eth2: igc: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Thu Aug 24 19:54:56 2023 daemon.notice netifd: wan2 (30320): udhcpc: sending select for x.x.x.x
Thu Aug 24 19:54:56 2023 daemon.notice netifd: wan2 (30320): udhcpc: lease of x.x.x.x obtained, lease time 172800
Thu Aug 24 19:54:56 2023 daemon.notice netifd: Interface 'wan2' is now up
Thu Aug 24 19:54:56 2023 daemon.notice netifd: Network device 'eth2' link is down
Thu Aug 24 19:54:56 2023 daemon.notice netifd: Interface 'wan2' has link connectivity loss
Thu Aug 24 19:54:57 2023 user.notice MPTCP: Flush route cache
Thu Aug 24 19:54:57 2023 user.notice firewall: Reloading firewall due to ifup of wan2 (eth2)
Thu Aug 24 19:54:57 2023 user.notice firewall.omr-server: Firewall reload, set server part firewall reloading
Thu Aug 24 19:54:57 2023 user.notice mptcp: Reloading mptcp config due to ifup of wan2 (eth2)
Thu Aug 24 19:54:59 2023 user.notice post-tracking-post-tracking: Set firewall on server vps
Thu Aug 24 19:55:00 2023 daemon.notice netifd: Network device 'eth2' link is up
Thu Aug 24 19:55:00 2023 daemon.notice netifd: Interface 'wan2' has link connectivity
Thu Aug 24 19:55:00 2023 kern.info kernel: [23113.674687] igc 0000:04:00.0 eth2: igc: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Thu Aug 24 19:55:02 2023 user.notice post-tracking-post-tracking: wan2 (eth2) switched up
Thu Aug 24 19:55:46 2023 user.notice post-tracking-post-tracking: Check API configuration...
Thu Aug 24 19:55:52 2023 user.notice post-tracking-post-tracking: Check API configuration... Done

So I guess Thu Aug 24 19:54:57 2023 user.notice mptcp: Reloading mptcp config due to ifup of wan2 (eth2) is the problem.

I checked ip r before restarting the interface and after bringing it back up. All routes the the same but how can I tell MPTCP to use both wan1 and wan2 after wan2 comes back online?

ioogithub avatar Aug 25 '23 00:08 ioogithub

try to reload v2ray? only some VPN use MPTCP.

fareign avatar Aug 25 '23 01:08 fareign

try to reload v2ray? only some VPN use MPTCP. I tried to reload with /etc/init.d/openmptcprouter-vps restart but it executes and exits and doesn't restart anything that I can see in the logs.

If I reload with /etc/init.d/v2ray reload it doesn't seem to fix the problem, traffic still comes in over only 1 wan and I see these errors:

[info] DNS disabled: main_dns
[info] Setting transparent proxy on port: 1234
[info] Transparent proxy mode: default
Flush terminated
ss-rules6: unknown option def
iptables-restore: line 2 failed
iptables-restore: line 2 failed
iptables-restore: line 2 failed
iptables-restore: line 2 failed
iptables-restore: line 2 failed
iptables-restore: line 2 failed
iptables-restore: line 2 failed
iptables-restore: line 2 failed
Warning: Section 'zone_vpn' cannot resolve device of network 'omr6in4'
Warning: Option @redirect[3].v2ray is unknown
Warning: Option @redirect[4].v2ray is unknown
Warning: Section @redirect[4] (19224 on v2ray) does not specify a protocol, assuming TCP+UDP

If I restart with /etc/init.d/v2ray restart then it does work but it creates a new problem, all clients on the network get disconnected so this is too disruptive as omr-tracker can restart an interface at any time.

MPTCP is working perfectly until an interface is restarted or omr-tracker tracker bring an interface down and up. After that this interface cannot be used for multi path.

There has to be a way to tell v2ray that the multi path is available to be used again. When I bring the interface down, OMR knows how to fail over to one interface successfully, it should be able to recover when the interface comes back up.

ioogithub avatar Aug 25 '23 03:08 ioogithub

Do you have same issue using Shadowsocks ? And do you have same issue using 6.1 snapshot image ?

Ysurac avatar Aug 25 '23 07:08 Ysurac

Do you have same issue using Shadowsocks ? I just did 3 tests with v2ray and 3 tests with shadowsocks. The problem affects both v2ray and shadowsocks proxies.

Test:

  1. reboot OMR, start file upload transfer, observe normal multi-path
  2. restart wan2 interface
  3. transfer a file (upload), observe traffic on only wan1 interface.

I can repeat the test by restating wan1 and same result. After interface restarts the proxy does not use it for multi-path.

Here is the result:

Normal multi-path: before restart

After wan2 interface is reset, multi-path is broken: after restart

I am looking at the logs after restarting an interface trying to find out what is not running. I see one log entry: Fri Aug 25 13:10:49 2023 daemon.notice netifd: wan2 (23088): Command failed: Permission denied but it doesn't say what command failed.

Please help me to troubleshoot:

  • Which script tells the proxy that multi-path is available and proxy should use multi-path?
  • Is there a uci setting that is not being set properly?
  • Maybe something is failing on restart but there is no log showing failure.

I think this is a serious problem because omr-tracker can restart interface at any time and that disables multipath on OMR router, user will not realize that OMR is running in degraded performance.

If OMR can successfully degrade the router to single path it should be able to upgrade it back to multi-path as well right?

ioogithub avatar Aug 25 '23 17:08 ioogithub

It's not a know problem and I don't have this issue on my 0.59.1 (and still not with latest snapshot). What is the result of multipath, ip a and ip r commands ?

Ysurac avatar Aug 25 '23 17:08 Ysurac

Did you test it with an upload so it would use the port forward and the proxy, either shadowsocks or v2ray same result for me?

I think I saw download is working once but I am not sure because testing uploads now.

I am looking at ip a and ip r and ip rule and multipath. I will compare results of working fresh reboot of OMR and results after interface is restarted and incoming transfers not using multi-oath.

ioogithub avatar Aug 25 '23 17:08 ioogithub

I did lots of tests:

  1. restart OMR, capture ip r, ip a, ip rule, multipath settings
  2. restart wan2
  3. capture all settings
  4. compare settings from step 1 and 3.

The routes, rules, interfaces and multipath settings are all identical when its working and when it is broken after restarting an interface.

It's not a know problem and I don't have this issue on my 0.59.1 (and still not with latest snapshot). What is the result of multipath, ip a and ip r commands ?

Here are the outputs, everything looks normal there is nothing to me:

ip r

default via 10.255.255.1 dev tun0 
default metric 1 
        nexthop via 10.6.0.1 dev eth1 weight 2 
        nexthop via 10.0.0.1 dev eth2 weight 1 
default via 10.6.0.1 dev eth1 metric 3 
default via 10.0.0.1 dev eth2 metric 4 
default via 10.255.255.1 dev tun0 metric 1200 
10.0.0.0/24 via 10.0.0.1 dev eth2 
10.0.0.0/24 dev eth2 scope link metric 4 
10.255.255.1 dev tun0 proto kernel scope link src 10.255.255.2 
10.255.255.2 dev tun0 scope link metric 1200 
6.1.2.2 dev eth1 proto static scope link src 10.6.8.1 metric 3 
10.6.0.0/10 dev eth1 scope link metric 3 
127.0.0.0/8 dev lo proto static scope link metric 6 
x.x.x.x metric 1 
        nexthop via 10.6.0.1 dev eth1 weight 10 
        nexthop via 10.0.0.1 dev eth2 weight 1

multipath:

bond0 is deactivated
erspan0 is deactivated
eth0 is deactivated
eth1 is in default mode
eth2 is in default mode
eth3 is deactivated
gre0 is deactivated
gretap0 is deactivated
ifb4tun0 is deactivated
ip6gre0 is deactivated
ip6tnl0 is deactivated
lo is deactivated
sit0 is deactivated
teql0 is deactivated
tun0 is deactivated

ip a:

1: lo: <LOOPBACK,UP,LOWER_UP,80000> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ip6tnl0@NONE: <NOARP,80000> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
3: sit0@NONE: <NOARP,80000> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
4: gre0@NONE: <NOARP,80000> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
5: gretap0@NONE: <BROADCAST,MULTICAST,80000> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
6: erspan0@NONE: <BROADCAST,MULTICAST,80000> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
7: ip6gre0@NONE: <NOARP,80000> mtu 1448 qdisc noop state DOWN group default qlen 1000
    link/gre6 :: brd ::
8: bond0: <BROADCAST,MULTICAST,MASTER,80000> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b1:b1:b1:b1:b1:b1 brd ff:ff:ff:ff:ff:ff
9: teql0: <NOARP,80000> mtu 1500 qdisc noop state DOWN group default qlen 100
    link/void 
10: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP,80000> mtu 1500 qdisc mq state UP group default qlen 100
    link/ether a:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff
    inet 192.168.x.x/24 brd 192.168.200.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 bbbb:bbbb:bbbb::1/60 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 cccc:cccc:cccc:cccc:cccc/64 scope link 
       valid_lft forever preferred_lft forever
11: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 100
    link/ether dd:dd:dd:dd:dd:dd brd ff:ff:ff:ff:ff:ff
    inet 10.6.8.1/10 brd 100.127.255.255 scope global eth1
       valid_lft forever preferred_lft forever
12: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP group default qlen 100
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global eth2
       valid_lft forever preferred_lft forever
13: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP,80000> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether a1:a1:a1:a1:a1:a1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.a.b/24 brd 192.168.200.255 scope global eth3
       valid_lft forever preferred_lft forever
21: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP,80000> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    link/none 
    inet 10.255.255.2 peer 10.255.255.1/32 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 a2a2:a2a2:a2a2:a2a2:a2a2/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
24: ifb4tun0: <BROADCAST,NOARP,80000> mtu 1500 qdisc noop state DOWN group default qlen 32
    link/ether a3:a3:a3:a3:a3:a3 brd ff:ff:ff:ff:ff:ff

Do you have any ideas of what else I can look at, its like the proxy get stuck on a single path or it doesn't get notified that multi-path is available again.

Questions:

  • What script or settings tell the proxy to use a single path when an interface is down?
  • What scripts or settings tell the proxy to use multi-path when all interfaces are up?

If ip r, ip rule etc. are the same with it is working and not working it must be something else, what else should I look for?

ioogithub avatar Aug 25 '23 19:08 ioogithub

Only mptcp init script is used, to set route table and multipath command to set multipath status of an interface. I don't understand exactly the issue, it's only with port forwarding ? In this case with shadowsocks as Proxy only VPN is used and with v2ray it's used if you have the checkbox enabled in firewall configuration. It's aggregated in download and not on port forwarding upload in both case ?

What is the MPTCP scheduler used ?

Ysurac avatar Aug 25 '23 19:08 Ysurac

mptcp init script is used, to set route table

this script sets route table: /etc/init.d/mptcp

multipath command to set multipath status of an interface

Is this command inside the mptcp init script?

But if the route table and multipath status is the same before and after the interface is restarted then the problem is somewhere else.

Does this mean that only the routes and multipath status of interface determines if the proxy will use multipath? So the proxy pushes data out and the kernel uses the routing table to route over multiple interfaces? Does this mean the proxy doesn't know about MPTCP? There are no other settings that notify the proxy?

I don't understand exactly the issue, it's only with port forwarding ?

I think it is only uploads, when a client from outside the network requests data from a computer inside the network after an interface has been restarted, it does not use that interface. It sends data only on the other interface. In this case port forwarding is used. I don't know if port forwarding is involved but downloads do not use that port forwarding.

In this case with shadowsocks as Proxy only VPN is used

For this I made used used different port forwarding rules which swtching to shadowsocks.

with v2ray it's used if you have the checkbox enabled in firewall configuration.

Yes I am using the v2ray checkbox port forwarding rule.

All of this works normally before an interface is stopped and started or restarted. After an interface is restarted there is no way to get OMR to use that interface for uploads.

I see the problem with both v2ray and shadowsocks:

  • After I restart wan1 and try a transfer, OMR only uses wan2 (broken)
  • After I restart wan2 and try a transfer, OMR only uses wan1 (broken)
  • If I reboot OMR, it uses wan1 and wan2 and it aggregates the bandwidth (working).

It seems like the proxy gets stuck, it doesn't get notified that both wan1 and wan2 are available.

I will start a fresh test:

  1. reboot omr (known working state)
  2. restart wan1
  3. start an upload and look at bandwidth for aggregate.
  4. start a download and look at bandwidth for aggregate.

What is the MPTCP scheduler used ?

scheduler: BLEST congestion control: reno path manager: full mesh TCP sync: 2 MPTCP version: 0 MPTCP checksum: disabled

There is an MPTCP debug if I set it will we get more data in the logs?

If you have any more ideas please let me know.

ioogithub avatar Aug 25 '23 19:08 ioogithub

I have done a lot of testing today, here is the result:

v2ray multi-path

uploads

  • after OMR reboot: working
  • after one interface restart: NOT working traffic never returns to the restarted interface.

downloads

  • after OMR reboot: working
  • after one interface restart: working

shadowsocks multi-path

uploads

  • after fresh OMR reboot: working
  • after fresh OMR reboot: working *but with unusual behavior.

downloads

  • after fresh OMR reboot: working
  • after one interface restart: working

Conclusion

I initially thought the problem was uploads for shadowsocks and v2ray but I was wrong. When shadowsocks, sometimes it starts working only after a delay of up to 2 minutes. Sometimes it starts right away. In my previous tests I did not wait long enough. I did see it working today.

So the problem is isolated to a specific use case: v2ray uploads (traffic initiated from outside the network using the v2ray port forwarding) only. I have never seen this work.

I have also tried the following:

  • all different combinations of path schedulers, congestion control, and other MPTCP settings.
  • I put MPTCP into debug mode and watched the kernel sync messages, everything looks normal, at least it looks the same when mptcp is aggregating versus not aggregating.
  • I switched MPTCP master from wan1 to wan2, same result.

Nothing I can do will restore aggregate multi-path for uploads with v2ray once an interface is restarted. Traffic always flows though the one interface that was not restarted.

  • If I restart wan1 then that moment all upload traffic stops going out over wan1 and never recovers
  • If I restart wan1 then the moment all upload traffic stops going out over wan2 and never recovers

The only way to recover from this is to click Save and Apply on the Wizard screen or reboot OMR.

Question: how is the MPTCP actually working on OMR? The v2ray config.json doesn't have MPTCP set in the stream settings, it is just TCP so I guess v2ray doesn't even know about MPTCP so how does it even work?

It's not a know problem and I don't have this issue on my 0.59.1 (and still not with latest snapshot). What is the result of multipath, ip a and ip r commands ?

You tested tested uploads with port forwarding using v2ray as the proxy with 0.59.1 and it is working for you correct?

  1. You are able to start an upload
  2. see traffic flowing over multiple interfaces in the bandwidth graph
  3. click on restart for one interface
  4. observe the interface is back online on the status page and logread
  5. observe that traffic stops flowing over the restarted interface.
  6. wait 10 minutes
  7. start another upload, observe traffic still only flowing on the unstarted interface.

If this works for you and it does not work for me that suggests a configuration problem. Tomorrow I will setup a VPS with the 6.1 upstream kernel and boot my router with openmptcprouter-v0.59.2alpha-6.1-r0+23789-ce6ad123e7-x86-64-generic-ext4-combined-efi, configure v2ray and port forwarding and repeat the tests above. I will report the results back here.

ioogithub avatar Aug 27 '23 05:08 ioogithub

Here is the result of testing with openmptcprouter v0.59.2alpha-6.1 r0+23789-ce6ad123e7.

  1. create new VPS: wget -O - http://www.openmptcprouter.com/server-test/debian-x86_64.sh | UPSTREAM6="yes" sh
  2. book router with image: openmptcprouter-v0.59.2alpha-6.1-r0+23789-ce6ad123e7-x86-64-generic-ext4-combined-efi.img.gz

Configure openwrt:

  1. Set LAN IP, set wan1 and wan2 to default and set force link
  2. Set server IP and server key.
  3. Create one port forwarding rule
  4. Everything else defaults.

shadowsocks upload: multipath working shadowsocks upload after interface reset: multipath working

Configure V2ray:

  1. Set v2ray vless
  2. Create port forwarding, no v2ray check.
  3. Edit port forward and add v2ray check.
  4. Check /etc/shorewall/rules, correct.
  5. Confirm proxy is receiving traffic on OMR status page. Known issue: No proxy displayed here.
  6. Confirm traffic on port 65228 (tcpdump -i eth port 65228: yes

Test v2ray upload after interface restart.

  1. Start an upload using v2ray: multi-path working v2rayeforerestart

  2. Reset wan1 interface: multi-path not working v2rayafterrestartwan1

Same results as the latest stable release. Uploads do not use aggregate multi-path with v2ray as the proxy after an interface is restarted.

There is currently no way to recover from this.

  • Restarting v2ray (/etc/init.d/v2ray restart) does not work.
  • Pressing Save and Apply on the wizard page did not work and it break the port forward.
  • Reboot OMR, still lost access to the port forward. So not way to test this again.

With stable, at lease Save and Apply or reboot OMR will fix the problem. In 6.1 this does not work. So the this issue at this time it is better to stay with stable until a work around is found.

ioogithub avatar Aug 27 '23 20:08 ioogithub

This is an example of an event that kills aggregate uploads:

Aug 28 03:33:19 OpenMPTCProuter user.notice post-tracking-post-tracking: wan2 (eth2) switched off because check error and ping from x.x.x.x error (1.0.0.1,114.114.115.115,1.2.4.8)
Aug 28 03:33:19 OpenMPTCProuter user.notice post-tracking-post-tracking: Delete default route to y.y.y.y via x.x.x.x dev eth2
Aug 28 03:41:15 OpenMPTCProuter daemon.notice netifd: Network device 'eth2' link is down
Aug 28 03:41:15 OpenMPTCProuter daemon.notice netifd: Interface 'wan2' has link connectivity loss
Aug 28 03:41:22 OpenMPTCProuter daemon.notice netifd: Network device 'eth2' link is up
Aug 28 03:41:22 OpenMPTCProuter daemon.notice netifd: Interface 'wan2' has link connectivity 
Aug 28 03:41:25 OpenMPTCProuter user.notice post-tracking-post-tracking: wan2 (eth2) switched up
Aug 28 03:41:25 OpenMPTCProuter user.notice post-tracking-post-tracking: Interface route not yet set, set route ip r add default via x.x.x.x dev eth2 metric 4
Aug 28 03:41:26 OpenMPTCProuter user.notice post-tracking-post-tracking: New public ip detected for wan2 (eth2): a.a.a.a
Aug 28 03:41:26 OpenMPTCProuter user.notice post-tracking-post-tracking: Reload MPTCP for eth2

The OMR-tracker brings the interface down and up again. After this, OMR is in single mode and will not use wan2 again until the server is restarted so this issue has a big impact on OMR stability and reliability.

There has to be a way to have uploads recover without manual user intervention after these interface reset events.

ioogithub avatar Aug 28 '23 05:08 ioogithub

On latest snapshot, I'm not able to reproduce the issue for now using "iperf3" to test upload speed with V2Ray VLESS. How do you test upload speed ? By downloading externally using port forward ?

Ysurac avatar Aug 28 '23 10:08 Ysurac

Yes, I am doing a real world test like this:

external client -> VPS (public IP) -> port forward OMR via v2ray vless -> host on LAN.

What exact test are you running? omr-iperf on OMR? If I do that right now this is what I get but I only see one wan connection use so maybe this test is not correct test: https://ibb.co/L5W55Tg

For my test, after OMR is restarted, If I do a download from an external client (upload) in a good state, I get this: https://ibb.co/LvLDFn8

If I do a download from an external client (upload) after an interface was restarted (by omr-tracker) or manually, I get this: https://ibb.co/tDfRGD6

It never recovers from this. omr-tracker reset wan2 8 hours ago: Aug 28 07:43:41 OpenMPTCProuter user.notice post-tracking-post-tracking: wan2 (eth2) switched off because check error and ping from x.x.x.x error (1.0.0.1,114.114.115.115,1.2.4.8), so OMR has been sitting in single path mode for uploads since this time.

If I do this: /etc/init/d/v2ray restart or wizard validate on the stable version, it will return to normal aggregate however this operation is disruptive, it kills all client connections and is manual and it only lasts until omr-tracker resets one of the interfaces again.

ioogithub avatar Aug 28 '23 16:08 ioogithub

here is a new upload test as discussed using iperf and a new port foward setup for the test.

iperf test:

  • on client outside network: iperf -c VPN_Public_IP -p 19223 -t 180 -i 180
  • on server behind OMR: iperf -s -p 19223
  • firewall rule: ACCEPT net $FW tcp 19223 # OMR openmptcprouter open router 19223 port tcp --- V2Ray to 192.168.x.x:19223
  • wan2 restarted on interface page.

result:

result, bandwidth never return to wan2: https://ibb.co/x8tJqQf start a new file transfer: https://ibb.co/PY1vNsz you can see, on the first screenshot, when wan2 goes down, omr switches to wan1 but after wan2 comes back up it never recovers.

The problem is with stable /etc/init.d/v2ray restart would fix the problem but disconnect all clients on the network. With 6.1 testing, after running /etc/init.d/v2ray restart packets do not leave the vps and get to omr. The only way to recover and start a new test is to restart both OMR and VPS. Validating wizard and v2ray restart, packets do not get past vps with tcpdump -i eth0 port 19223

ioogithub avatar Aug 28 '23 17:08 ioogithub

I've tested, using V2RAY VLESS protocol redirect port to internal http server and downloading file from external and no problem, when I remove a connection and put it back it's used again. I have also no problem with v2ray restart.

Can you give me full log when you test disconnect and then reconnect a connection ? I will reset my install to a fresh config for more tests.

Ysurac avatar Aug 28 '23 19:08 Ysurac

Can you give me full log when you test disconnect and then reconnect a connection ?

Yes, please let me know exactly which log you are looking for ? logread from OMR, journalctl -f from vps or something else? Is there any debug mode I can enable to get more info?

ioogithub avatar Aug 28 '23 19:08 ioogithub

I would need system log from the router, so the result of logread.

I tested with a new VPS and router installation and I have no issue. What is the result of uci show network via SSH on router ?

Ysurac avatar Aug 29 '23 13:08 Ysurac

I have been testing with the 6.1 kernel and switched back to 0.51.9 recently as it is easier for me to recover.

What is the result of uci show network

network.loopback=interface
network.loopback.proto='static'
network.loopback.ipaddr='127.0.0.1'
network.loopback.netmask='255.0.0.0'
network.loopback.multipath='off'
network.loopback.device='lo'
network.loopback.metric='6'
network.globals=globals
network.globals.mptcp_subflows='3'
network.globals.mptcp_add_addr_accepted='1'
network.globals.mptcp_add_addr_timeout='120'
network.globals.mptcp_fullmesh_num_subflows='1'
network.globals.mptcp_fullmesh_create_on_err='1'
network.globals.mptcp_ndiffports_num_subflows='1'
network.globals.ula_prefix='abac:4c32:5cbb::/48'
network.globals.mptcp_version='0'
network.globals.mptcp_rr_cwnd_limited='Y'
network.globals.mptcp_rr_num_segments='1'
network.globals.multipath='enable'
network.globals.mptcp_syn_retries='2'
network.globals.mptcp_scheduler='blest'
network.globals.congestion='reno'
network.globals.mptcp_path_manager='fullmesh'
network.globals.mptcp_checksum='0'
network.globals.mptcp_debug='0'
network.lan=interface
network.lan.proto='static'
network.lan.ipaddr='192.168.x.x'
network.lan.netmask='255.255.255.0'
network.lan.device='eth0'
network.lan.ifname='eth0'
network.lan.delegate='0'
network.lan.addlatency='0'
network.lan.multipath='off'
network.lan.ip4table='lan'
network.lan.metric='7'
network.lan.label='toMesh'
network.lan.defaultroute='0'
network.lan.peerdns='0'
network.lan.ip6assign='60'
network.lan.ipv6='0'
network.lan.force_link='1'
network.lan_rule=rule
network.lan_rule.lookup='lan'
network.lan_rule.priority='100'
network.wan1=interface
network.wan1.device='eth1'
network.wan1.ip4table='wan'
network.wan1.defaultroute='0'
network.wan1.addlatency='0'
network.wan1.ipv6='0'
network.wan1.peerdns='0'
network.wan1.proto='dhcp'
network.wan1.label='starlink'
network.wan1.force_link='1'
network.wan1.metric='3'
network.wan1.multipath='master'
network.wan1_dev=device
network.wan1_dev.name='eth1'
network.wan1_dev.txqueuelen='20'
network.wan2=interface
network.wan2.device='eth2'
network.wan2.ip4table='wan'
network.wan2.defaultroute='0'
network.wan2.addlatency='0'
network.wan2.ipv6='0'
network.wan2.peerdns='0'
network.wan2.proto='dhcp'
network.wan2.force_link='1'
network.wan2.metric='4'
network.wan2.label='lte'
network.wan2.multipath='on'
network.wan2_dev=device
network.wan2_dev.name='eth2'
network.wan2_dev.txqueuelen='20'
network.omrvpn=interface
network.omrvpn.device='tun0'
network.omrvpn.ip4table='vpn'
network.omrvpn.multipath='off'
network.omrvpn.leasetime='12h'
network.omrvpn.type='tunnel'
network.omrvpn.txqueuelen='100'
network.omrvpn.metric='1200'
network.omrvpn.proto='none'
network.omr6in4=interface
network.omr6in4.proto='6in4'
network.omr6in4.ip4table='vpn'
network.omr6in4.multipath='off'
network.omr6in4.ipaddr='10.255.255.2'
network.omr6in4.peeraddr='10.255.255.1'
network.omr6in4.ip6addr='fe80::a00:2/126'
network.omr6in4.gateway='fe80::a00:1/126'
network.omr6in4.force_link='1'
network.omr6in4.metric='1201'
network.lan_dev=device
network.lan_dev.name='eth0'
network.wan3_dev=device
network.wan3_dev.type='macvlan'
network.wan3_dev.mode='vepa'
network.wan3_dev.ifname='eth0'
network.wan3_dev.name='wan3'
network.wan3_dev.txqueuelen='20'
network.localNIC=interface
network.localNIC.proto='static'
network.localNIC.device='eth3'
network.localNIC.netmask='255.255.255.0'
network.localNIC.multipath='off'
network.localNIC.addlatency='0'
network.localNIC.metric='11'
network.localNIC.ipaddr='192.168.y.y'
network.localNIC.ipv6='0'
network.localNIC.defaultroute='0'
network.localNIC.peerdns='0'
network.localNIC_dev=device
network.localNIC_dev.name='eth3'
network.@route[2]=route
network.@route[2].interface='lan'
network.@route[2].target='192.168.a.a'
network.@route[2].netmask='255.255.255.0'
network.@route[2].gateway='192.168.x.xyy'

I would need system log from the router, so the result of logread.

Here is the log from the moment I clicked reset on the wan2 interface. After this I just see expected lease renewal messages, nothing else related to connections and the log is quiet.

Thu Aug 31 14:13:42 2023 daemon.notice netifd: Interface 'wan2' is now down
Thu Aug 31 14:13:42 2023 kern.info kernel: [ 3351.120795] igc 0000:04:00.0 eth2: igc: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Thu Aug 31 14:13:42 2023 daemon.notice netifd: Interface 'wan2' is disabled
Thu Aug 31 14:13:42 2023 daemon.notice netifd: Interface 'wan2' is enabled
Thu Aug 31 14:13:42 2023 daemon.notice netifd: Interface 'wan2' is setting up now
Thu Aug 31 14:13:42 2023 daemon.notice netifd: Network device 'eth2' link is down
Thu Aug 31 14:13:42 2023 daemon.notice netifd: Interface 'wan2' has link connectivity loss
Thu Aug 31 14:13:42 2023 daemon.notice netifd: wan2 (29374): udhcpc: started, v1.33.2
Thu Aug 31 14:13:42 2023 daemon.notice netifd: wan2 (29374): udhcpc: sending discover
Thu Aug 31 14:13:42 2023 kern.err kernel: [ 3351.487765] __mptcp_init4_subsockets: token 0xa463fb6a bind() to y.y.y.y index 12 failed, error -99
Thu Aug 31 14:13:43 2023 user.notice MPTCP: Flush route cache
Thu Aug 31 14:13:44 2023 user.notice post-tracking-post-tracking: wan2 (eth2) switched off because wan2 may have ip issues, interface have no IPv4, interface have no IPv4 gateway
Thu Aug 31 14:13:44 2023 user.notice post-tracking-post-tracking: Delete default route to x.x.x.x via  dev eth2
Thu Aug 31 14:13:45 2023 daemon.notice netifd: wan2 (29374): udhcpc: sending discover
Thu Aug 31 14:13:46 2023 daemon.notice netifd: Network device 'eth2' link is up
Thu Aug 31 14:13:46 2023 daemon.notice netifd: Interface 'wan2' has link connectivity
Thu Aug 31 14:13:46 2023 kern.info kernel: [ 3354.700767] igc 0000:04:00.0 eth2: igc: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Thu Aug 31 14:13:49 2023 daemon.notice netifd: wan2 (29374): udhcpc: sending discover
Thu Aug 31 14:13:52 2023 daemon.notice netifd: wan2 (29374): udhcpc: sending select for y.y.y.y
Thu Aug 31 14:13:52 2023 daemon.notice netifd: wan2 (29374): udhcpc: lease of y.y.y.y obtained, lease time 172800
Thu Aug 31 14:13:52 2023 daemon.notice netifd: Interface 'wan2' is now up
Thu Aug 31 14:13:52 2023 daemon.notice netifd: Network device 'eth2' link is down
Thu Aug 31 14:13:52 2023 daemon.notice netifd: Interface 'wan2' has link connectivity loss
Thu Aug 31 14:13:52 2023 user.notice firewall: Reloading firewall due to ifup of wan2 (eth2)
Thu Aug 31 14:13:52 2023 user.notice firewall.omr-server: Firewall reload, set server part firewall reloading
Thu Aug 31 14:13:53 2023 user.notice post-tracking-post-tracking: Set firewall on server vps
Thu Aug 31 14:13:54 2023 daemon.err glorytun[14226]: read: Operation timed out
Thu Aug 31 14:13:54 2023 daemon.info glorytun[14226]: STOPPED tun0
Thu Aug 31 14:13:55 2023 daemon.err glorytun[14226]: x.x.x.x.65001: connected
Thu Aug 31 14:13:55 2023 daemon.info glorytun[14226]: STARTED tun0
Thu Aug 31 14:13:55 2023 user.notice MPTCP: Add route z.z.z.z/24 via z.z.z.b dev eth2
Thu Aug 31 14:13:55 2023 user.notice MPTCP: Add route a.a.a.a via a.a.a.b dev eth0
Thu Aug 31 14:13:55 2023 user.notice mptcp: Reloading mptcp config due to ifup of wan2 (eth2)
Thu Aug 31 14:13:55 2023 user.notice MPTCP: Add route z.z.z.z/24 via z.z.z.b dev eth2
Thu Aug 31 14:13:56 2023 user.notice MPTCP: Add route a.a.a.a via a.a.a.b dev eth0
Thu Aug 31 14:13:56 2023 daemon.notice netifd: Network device 'eth2' link is up
Thu Aug 31 14:13:56 2023 daemon.notice netifd: Interface 'wan2' has link connectivity
Thu Aug 31 14:13:56 2023 kern.info kernel: [ 3364.671214] igc 0000:04:00.0 eth2: igc: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Thu Aug 31 14:13:56 2023 user.notice MPTCP: Flush route cache
Thu Aug 31 14:13:56 2023 user.notice MPTCP: Add route z.z.z.z/24 via z.z.z.b dev eth2
Thu Aug 31 14:13:56 2023 user.notice MPTCP: Add route a.a.a.a via a.a.a.b dev eth0
Thu Aug 31 14:13:56 2023 user.notice MPTCP: Flush route cache
Thu Aug 31 14:13:58 2023 user.notice post-tracking-post-tracking: wan2 (eth2) switched up
Thu Aug 31 14:14:01 2023 user.notice MPTCP: Add route z.z.z.z/24 via z.z.z.b dev eth2
Thu Aug 31 14:14:01 2023 user.notice MPTCP: Add route a.a.a.a via a.a.a.b dev eth0
Thu Aug 31 14:14:45 2023 user.notice post-tracking-post-tracking: Check API configuration...
Thu Aug 31 14:14:53 2023 user.notice post-tracking-post-tracking: Check API configuration... Done

After this point wan2 does not recover: Screenshot_20230831_141709

  • I can come back in 6 hours and see the same bandwidth graph, only wan1 traffic for uploads
  • I can repeat the test but restarting wan1, same result, only traffic on wan2 now.
  • I also tried resetting the lan interface and ensuring that any existing connections timed out before starting a new transfer, same results.
  • tested file transfers with sftp, curl from http server, ierf etc.
  • only manual intervention /etc/init.d/vray restart will fix the problem and allow upload traffic to use both wan1 and wan2 but this causes other problems because it kills all active connection on the network.

ioogithub avatar Aug 31 '23 18:08 ioogithub

Can you try to do multipath eth2 off && multipath eth2 on to check if this change something ?

Ysurac avatar Aug 31 '23 19:08 Ysurac

multipath eth2 off && multipath eth2 on

  1. start a transfer after interface reset when OMR is in single wan state
  2. multipath eth2 off && multipath eth2 on
  3. No interruption or change to the transfer
  4. Also no notice in logread -f from this command.

Result: I cannot see any effect from this command.

There was one error after i reset wan2: Thu Aug 31 14:13:42 2023 kern.err kernel: [ 3351.487765] __mptcp_init4_subsockets: token 0xa463fb6a bind() to y.y.y.y index 12 failed, error -99

Could this be the problem it happens when wan2 is brought down.

I am going to reboot OMR to a known good state and watch the logs closely again.

ioogithub avatar Aug 31 '23 20:08 ioogithub

Here is a log with two question I have.

  1. OMR rebooted at 15:58. At 16:14 this message:

Thu Aug 31 16:14:21 2023 kern.info kernel: [ 1079.764399] igc 0000:04:00.0: changing MTU from 1500 to 1460 and this change causes eth2 to go down and up:

Thu Aug 31 16:14:21 2023 daemon.notice netifd: Network device 'eth2' link is down
Thu Aug 31 16:14:21 2023 daemon.notice netifd: Interface 'wan2' has link connectivity loss
Thu Aug 31 16:14:25 2023 daemon.notice netifd: Network device 'eth2' link is up
Thu Aug 31 16:14:25 2023 daemon.notice netifd: Interface 'wan2' has link connectivity
Thu Aug 31 16:14:25 2023 kern.info kernel: [ 1083.381921] igc 0000:04:00.0 eth2: igc: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

Is there an OMR process that is monitoring something and trying to optimize MTU and changes it while the interface is up and running? The consequence is that it restarts the interface after and then the interface isn't used for uploads because of this issue.

  1. I also see these messages repeated:
Thu Aug 31 16:02:06 2023 user.notice MPTCP: Flush route cache
Thu Aug 31 16:02:06 2023 user.notice MPTCP: Add route 10.x.x.x/24 via 10.x.x.y dev eth2
Thu Aug 31 16:02:06 2023 user.notice omr-bypass: Starting OMR-ByPass...
Thu Aug 31 16:02:07 2023 user.notice omr-bypass: OMR-ByPass is running
Thu Aug 31 16:02:07 2023 daemon.info glorytun: starting glorytun vpn instance vpn
Thu Aug 31 16:02:07 2023 user.notice omr-tracker: Launching...
Thu Aug 31 16:02:14 2023 user.notice omr-tracker: Launched
Thu Aug 31 16:02:15 2023 daemon.info omr-tracker-v2ray: V2Ray is up (can contact via http 1.0.0.1)

Sometimes I see these message every few minutes and sometimes I do not see them for 1 hour in the log. What are these for and what determines how often they will run? Is this normal?

Also, is glorytun actually retarting everytime this is logged: Thu Aug 31 16:02:07 2023 daemon.info glorytun: starting glorytun vpn instance vpn

ioogithub avatar Aug 31 '23 20:08 ioogithub

I rebooted OMR at 17:00, same issue, the kernel or some OMR process changed the MTU again at 17:13:

Thu Aug 31 17:14:35 2023 kern.info kernel: [  966.328211] igc 0000:04:00.0: changing MTU from 1500 to 1460
Thu Aug 31 17:14:36 2023 daemon.notice netifd: Network device 'eth2' link is down
Thu Aug 31 17:14:36 2023 daemon.notice netifd: Interface 'wan2' has link connectivity loss
Thu Aug 31 17:14:39 2023 user.notice post-tracking-post-tracking: Check API configuration...
Thu Aug 31 17:14:39 2023 daemon.notice netifd: Network device 'eth2' link is up
Thu Aug 31 17:14:39 2023 daemon.notice netifd: Interface 'wan2' has link connectivity
Thu Aug 31 17:14:39 2023 kern.info kernel: [  969.825546] igc 0000:04:00.0 eth2: igc: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

and then the log file is quite, no omr-tracker or glorytun log messages for 2 hours.

What is the purpose of resetting the MTU and why don't I see the omr-tracking, glorytun, and mtcp flush cache messages any longer?

ioogithub avatar Aug 31 '23 23:08 ioogithub

As soon as I initiate 1 file upload, these messages appear again, every few minutes. Are they related to connections or file transfers? Are they related to this issue?

Thu Aug 31 19:29:27 2023 user.notice omr-tracker: Launching...
Thu Aug 31 19:29:34 2023 user.notice omr-tracker: Launched
Thu Aug 31 19:29:34 2023 daemon.info glorytun: starting glorytun vpn instance vpn
Thu Aug 31 19:29:34 2023 user.notice MPTCP: Flush route cache
Thu Aug 31 19:29:34 2023 user.notice MPTCP: Add route 10.a.a.a/24 via 10.0.0.1 dev eth2
Thu Aug 31 19:29:34 2023 user.notice omr-bypass: Starting OMR-ByPass...
Thu Aug 31 19:29:35 2023 daemon.info omr-tracker-v2ray: V2Ray is up (can contact via http 1.0.0.1)
Thu Aug 31 19:29:35 2023 user.notice omr-bypass: OMR-ByPass is running
Thu Aug 31 19:29:35 2023 user.notice omr-tracker: Launching...
Thu Aug 31 19:29:42 2023 user.notice omr-tracker: Launched
Thu Aug 31 19:29:42 2023 daemon.info glorytun: starting glorytun vpn instance vpn
Thu Aug 31 19:29:42 2023 user.notice MPTCP: Flush route cache
Thu Aug 31 19:29:43 2023 user.notice MPTCP: Add route 10.a.a.a/24 via 10.0.0.1 dev eth2
Thu Aug 31 19:29:43 2023 user.notice omr-bypass: Starting OMR-ByPass...
Thu Aug 31 19:29:43 2023 daemon.info omr-tracker-v2ray: V2Ray is up (can contact via http 1.0.0.1)
Thu Aug 31 19:29:43 2023 user.notice omr-bypass: OMR-ByPass is running
Thu Aug 31 19:29:51 2023 user.notice omr-tracker: Launching...
Thu Aug 31 19:29:58 2023 user.notice omr-tracker: Launched
Thu Aug 31 19:29:58 2023 daemon.info glorytun: starting glorytun vpn instance vpn
Thu Aug 31 19:29:58 2023 user.notice MPTCP: Flush route cache
Thu Aug 31 19:29:58 2023 user.notice MPTCP: Add route 10.a.a.a/24 via 10.0.0.1 dev eth2
 Aug 31 19:29:58 2023 user.notice omr-bypass: Starting OMR-ByPass...
Thu Aug 31 19:29:59 2023 user.notice omr-bypass: OMR-ByPass is running
Thu Aug 31 19:29:59 2023 daemon.info omr-tracker-v2ray: V2Ray is up (can contact via http 1.0.0.1)

ioogithub avatar Aug 31 '23 23:08 ioogithub

You can fix MTU in Network->Interfaces for each interfaces, then it will not try to calculate MTU (but changing MTU should not put the interface down).

Ysurac avatar Sep 01 '23 06:09 Ysurac

You can fix MTU in Network->Interfaces for each interfaces, then it will not try to calculate MTU (but changing MTU should not put the interface down).

Are you sure? I always see a reset after an MTU change:

I get this:

Fri Sep 1 01:08:08 2023 kern.info kernel: [ 3654.871161] igc 0000:04:00.0: changing MTU from 1500 to 1460

Fri Sep  1 01:08:08 2023 daemon.notice netifd: Network device 'eth2' link is down
Fri Sep  1 01:08:08 2023 daemon.notice netifd: Interface 'wan2' has link connectivity loss
Fri Sep  1 01:08:11 2023 daemon.notice netifd: Network device 'eth2' link is up
Fri Sep  1 01:08:11 2023 daemon.notice netifd: Interface 'wan2' has link connectivity
Fri Sep  1 01:08:11 2023 kern.info kernel: [ 3658.468358] igc 0000:04:00.0 eth2: igc: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

I can also reproduce it with this:

ip link set dev eth0 mtu 1460

it will always reset the interface. Fri Sep 1 01:08:08 2023 daemon.notice netifd: Network device 'eth0' link is down

but this MTU just contributes to the problem, it doesn't fix the root cause of aggregate broken after interface restarted.

What about this error error after i reset wan2: Thu Aug 31 14:13:42 2023 kern.err kernel: [ 3351.487765] __mptcp_init4_subsockets: token 0xa463fb6a bind() to y.y.y.y index 12 failed, error -99 could this be causing the issue or is this just from the interface going down?

ioogithub avatar Sep 01 '23 15:09 ioogithub

Seems that changing MTU put interface down in some case, I will change the code. The mptcp error is due to interface down yes. I will try to reproduce the issue by using DHCP on interfaces.

Ysurac avatar Sep 01 '23 15:09 Ysurac

I am looking closely at the logs. When I reboot OMR I see these errors related to v2ray, could any of these cause the issue?

Fri Sep  1 12:40:06 2023 daemon.notice procd: /etc/rc.d/S99v2ray: ss-rules6: unknown option def
Fri Sep  1 12:40:06 2023 user.notice v2ray: Reload omr-bypass rules
Fri Sep  1 12:40:06 2023 daemon.notice procd: /etc/rc.d/S99v2ray: iptables-restore: line 2 failed
Fri Sep  1 12:40:06 2023 daemon.notice procd: /etc/rc.d/S99v2ray: iptables-restore: line 2 failed
Fri Sep  1 12:40:06 2023 daemon.notice procd: /etc/rc.d/S99v2ray: iptables-restore: line 2 failed
Fri Sep  1 12:40:06 2023 daemon.notice procd: /etc/rc.d/S99v2ray: iptables-restore: line 2 failed
Fri Sep  1 12:40:06 2023 daemon.notice procd: /etc/rc.d/S99v2ray: iptables-restore: line 2 failed
Fri Sep  1 12:40:07 2023 daemon.notice procd: /etc/rc.d/S99v2ray: Warning: Section 'zone_vpn' cannot resolve device of network 'omr6in4'
Fri Sep  1 12:40:07 2023 daemon.notice procd: /etc/rc.d/S99v2ray: Warning: Option @redirect[3].v2ray is unknown
Fri Sep  1 12:40:07 2023 daemon.notice procd: /etc/rc.d/S99v2ray: Warning: Option @redirect[4].v2ray is unknown
Fri Sep  1 12:40:07 2023 daemon.notice procd: /etc/rc.d/S99v2ray: Warning: Section @redirect[4] (19223 on v2ray) does not specify a protocol, assuming TCP+UDP
Fri Sep  1 12:40:07 2023 daemon.info v2ray[26149]: V2Ray 4.43.0 (V2Fly, a community-driven edition of V2Ray.) OpenWrt R1 (go1.17.3 linux/amd64)
Fri Sep  1 12:40:07 2023 daemon.info v2ray[26149]: A unified platform for anti-censorship.
Fri Sep  1 12:40:07 2023 daemon.info v2ray[26149]: 2023/09/01 16:40:07 [Info] main/jsonem: Reading config: /var/etc/v2ray/v2ray.main.json

The redirect 3 and 4 are the port forwards. At the end of the boot however aggregate does work so maybe this is normal?

From all my testing is seems aggregate uploads stop working always after an interface reset so booting OMR or reloading v2ray fixes it but there are several automated processes like MTU or omr-tracker which reset the interface.

I manually set MTU so that will prevent DHCP from restarting the interfaces. Is there anything I can do to prevent omr-tracker or restart, I don't want to disable it because it is necessary right?

Is there anything else I can adjust or configure to prevent automated interface restarts?

ioogithub avatar Sep 01 '23 16:09 ioogithub

The errors doesn't cause issues. For omr-tracker, you can change settings in Services->OMR-Tracker. OMR-Tracker is necessary to set routes, check services, get some connection info,... The problem is more why connection are no more in V2ray in your case.

Ysurac avatar Sep 01 '23 19:09 Ysurac