Skynet blocking GitHub access by "forgetting" existing whitelist entry

[TheStork239]

Brief Description Of Issue

Every few days Skynet blocks access to Github.com (e.g. IP 140.82.121.3) due to Github IPs erroneously ending on an IP block list, but somehow the existing whitelisting of that IP does not get checked or executed by Skynet.

Logs contain blocks of e.g. 140.82.121.3, but when trying to add that IP to the whitelist Skynet notifies that it has already been added.

Therefore it seems like Skynet is somehow missing putting the whitelist entries for GitHub into action.

(I am not the only one with the issue; there seems to be a number of people with this issue on discussion forums) I am using an RT-AC88U router with Merlin 386.3_2 firmware, with Diversion running in addition to Skynet.

Steps To Reproduce Issue

Seems to occur randomly every few days. Issue seems to be temporarily fixed by either 1) disabling Skynet off, 2) manually refreshing whitelist entries, or 3) rebooting router. However, that is only a temporary fix and the issue seems to recur.

Expected Behaviour

Github.com and related IPs are already on the built-in Skynet whitelist, so access to GitHub.com should not be blocked by Skynet.

Output of ( sh /jffs/scripts/firewall debug info )

[Adamm00]

I had some ideas for whitelist improvements but it got put on the back-burner. This is nothing more then an (annoying) false positive from firehol lvl3. You can temporarily remove this from your filter list. I’ll see about other whitelist improvements when I get a chance.

[p3yot3]

Thanks for reporting this issue - I thought it was just me !

[Adamm00]

I've pushed v7.2.6 https://github.com/Adamm00/IPSet_ASUS/commit/f0cea071acd30b03af3173fd9dc75e7a21311492

This will add GitHubs asn to Skynet's whitelist to avoid any IP changes. I have plans to make the whitelist more accurate with domains, but it will take some time to implement as my free time is limited.

As for some of those commenting on SNB, I find it distasteful. I've provided free support and the groundwork for countless addons for 7+ years. Rather then take shots at someone whose not around to defend themselves, create a pull request and remember this is an "open source" project.

[TheStork239]

Thank you @Adamm00 ; really appreciate your help and support. I’ll give this fix a try later on.

[dave14305]

As for some of those commenting on SNB, I find it distasteful. I've provided free support and the groundwork for countless addons for 7+ years. Rather then take shots at someone whose not around to defend themselves, create a pull request and remember this is an "open source" project.

Will do!

[p3yot3]

Updated & so far so good. Thanks for all your hard work - it is very much appreciated by most of us!

[TheStork239]

Updated and seems to do the trick. Thanks! I’ll close the issue.

[entrop1x]

I've updated Skynet to v7.2.6 and the 140.82.121.3 ip still ends up getting blocked.

I had to add it to Whitelist, but even I don't know how long that'll work. I think this issue should remain open.

[TheStork239]

Ok. I’ll re-open the issue as it still seems to be causing trouble for some.

[TheStork239]

I can confirm that even with v7.2.6 GitHub.com does from time to time still get blocked. It seems to be fixed (temporarily) with a manual refresh of entries in the whitelist, through Skynet command line. It appears as if somehow Skynet “forgets” the whitelist until the entries are refreshed.

[dave14305]

I believe the problem is some kind of rate limiting by bgpview.io. If you remove the & at the end of line 879, it works more reliably, albeit slower. https://github.com/Adamm00/IPSet_ASUS/blob/f0cea071acd30b03af3173fd9dc75e7a21311492/firewall.sh#L879

[Adamm00]

I believe the problem is some kind of rate limiting by bgpview.io. If you remove the & at the end of line 879, it works more reliably, albeit slower.

https://github.com/Adamm00/IPSet_ASUS/blob/f0cea071acd30b03af3173fd9dc75e7a21311492/firewall.sh#L879

Thanks, thought we got past this months ago but I guess bpgview increased their rate limiting again. Will look into it when I get some free time this week.

[macexx]

Though even if bgpview has a harsh rate limit there will still be two issues.

1, BGPview only pulls the registered CIDR´s from IANA for given AS.

So in githubs case there is quite a lot of addresses used that is not covered(different Data centers, subsidiaries, ISPs, sub contractors, remote APIs etc,.)

For an example using an vpn to US endpoint virgina(cloudflare dns) and trying to build a docker container(to not get IP of a CDN) a lookup gave me "185.199.108.0/22 AS54113" which is not covered by bgpview but by githubs own list.

So should probably be pulled from github itself with something similar to what is done for amazon, Will add a lot of duplicates but ipset will sort that out anyways.

curl -fsL --retry 3 --connect-timeout 3 https://api.github.com/meta | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}' | awk '{printf "add Skynet-Whitelist %s comment \"CDN-Whitelist: Github\"\n", $1 }'

2, Why is Github blacklisted in the first place? Firehol lvl3 was suppose to be a 30 day list but firehol as a hole is not maintained any moore couple that with some sources it pulls from that is not maintained and more importantly it pulls from vxvault, a superbad combo.

So if if some code is uploaded to vxvault for testing from a source github, discord etc., and it can check that the file is "hosted"., and it contains malicious code etc., it will blacklist that IP. (and this list is not actively checked at all so a false positive wont get removed until it falls from the list which seem to be along long time).

This is probably why github is on the list and a huge amount of discord IPs.

BAD BAD VERY BAD Vxvault uses "X-Forwarded-For" from the upload, to determinate the IP to block.

As long as your packet doesn't travel through a load-balancer/proxy that changes the value of "X-Forwarded-For" header, it will remain.

"X-Forwarded-For" can easily be changed by any client, so you can easily set "8.8.8.8(google dns)" upload some malicious code and get google dns banned from the lvl3 list.

Well you need a login on vxvault to do all this which probably explains why there is quite a low number of banned IPs on that list.(though found an addon for discord that has this pre entered which probably explains all the discord IPs on that list)

So this list should probably be removed and add ciarmy http://cinsscore.com/list/ci-badguys.txt since its what the list pulls its majority from any ways.

[Adamm00]

Though even if bgpview has a harsh rate limit there will still be two issues.

1, BGPview only pulls the registered CIDR´s from IANA for given AS.

So in githubs case there is quite a lot of addresses used that is not covered(different Data centers, subsidiaries, ISPs, sub contractors, remote APIs etc,.)

For an example using an vpn to US endpoint virgina(cloudflare dns) and trying to build a docker container(to not get IP of a CDN) a lookup gave me "185.199.108.0/22 AS54113" which is not covered by bgpview but by githubs own list.

So should probably be pulled from github itself with something similar to what is done for amazon, Will add a lot of duplicates but ipset will sort that out anyways.

curl -fsL --retry 3 --connect-timeout 3 https://api.github.com/meta | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}' | awk '{printf "add Skynet-Whitelist %s comment \"CDN-Whitelist: Github\"\n", $1 }'

2, Why is Github blacklisted in the first place? Firehol lvl3 was suppose to be a 30 day list but firehol as a hole is not maintained any moore couple that with some sources it pulls from that is not maintained and more importantly it pulls from vxvault, a superbad combo.

So if if some code is uploaded to vxvault for testing from a source github, discord etc., and it can check that the file is "hosted"., and it contains malicious code etc., it will blacklist that IP. (and this list is not actively checked at all so a false positive wont get removed until it falls from the list which seem to be along long time).

This is probably why github is on the list and a huge amount of discord IPs.

BAD BAD VERY BAD Vxvault uses "X-Forwarded-For" from the upload, to determinate the IP to block.

As long as your packet doesn't travel through a load-balancer/proxy that changes the value of "X-Forwarded-For" header, it will remain.

"X-Forwarded-For" can easily be changed by any client, so you can easily set "8.8.8.8(google dns)" upload some malicious code and get google dns banned from the lvl3 list.

Well you need a login on vxvault to do all this which probably explains why there is quite a low number of banned IPs on that list.(though found an addon for discord that has this pre entered which probably explains all the discord IPs on that list)

So this list should probably be removed and add ciarmy http://cinsscore.com/list/ci-badguys.txt since its what the list pulls its majority from any ways.

Thanks https://github.com/Adamm00/IPSet_ASUS/commit/ce999d8278718b0fc0baf857d9e5ad53b810467b

Will keep the rest in mind.

[dave14305]

add Skynet-Whitelist 140.82.112.0/20 comment "CDN-Whitelist: Github" This should be capturing the 140.82.121.3 address.

[dave14305]

There are overlapping ranges (many smaller CIDRs within the /20 I mentioned above). Does that matter?

[Adamm00]

I can confirm that with the latest v7.2.7, Github is still getting blocked. As is DuckDuckGo. @Adamm00 I hope you have the time and persistence to make this go away once and for all. Best wishes. thelonelycoder

Mind giving me examples of blocked addresses so we can identify what’s going on?

[decoderman]

I can confirm that with the latest v7.2.7, Github is still getting blocked. As is DuckDuckGo. @Adamm00 I hope you have the time and persistence to make this go away once and for all. Best wishes. thelonelycoder

Mind giving me examples of blocked addresses so we can identify what’s going on?

nslookup only gives me one IP. And since I restarted Skynet it will not be blocked until it happens next time: Non-authoritative answer: Name: duckduckgo.com Address: 40.114.177.156

Updated to your attempted fix and it immediately blocked DDG. Now I get: Non-authoritative answer: Name: duckduckgo.com Address: 191.235.123.80

[Adamm00]

So...

https://i.imgur.com/QaugUKJ.png https://i.imgur.com/wUnkZla.png

But this will take some time to get right. It is linked to many moving pieces and depends on how reliable dnsmasq is.

[macexx]

I can confirm that with the latest v7.2.7, Github is still getting blocked. As is DuckDuckGo. @Adamm00 I hope you have the time and persistence to make this go away once and for all. Best wishes. thelonelycoder

Mind giving me examples of blocked addresses so we can identify what’s going on?

nslookup only gives me one IP. And since I restarted Skynet it will not be blocked until it happens next time: Non-authoritative answer: Name: duckduckgo.com Address: 40.114.177.156

Updated to your attempted fix and it immediately blocked DDG. Now I get: Non-authoritative answer: Name: duckduckgo.com Address: 191.235.123.80

Well this is expected behavior, you will get a different IP, kind of how the internet works with CDN´s / Load balancers. (Thou if you use something like stubby with an DNS list this will exponentially give you more IP lookups since the DNS servers are always called from different locations)

So if you white list a domain it will whitelist the "DNS look up IP" when added, so if this changes on your next DNS lookup(haven't checked the code for when it updates) it "might" get blocked if the new IP is in a block list.

This is why whitelisting of domains get super tricky with an "block list" which Skynet pretty much is. (Sure you could have some code hammering DNS lookups to a list but would be resource heavy and plain stupid since there is nothing to verify against) And this is how it should work. (So it will work on and off)

And in your case with Duckduck. They don't have their own dedicated IP´s, but are hosted in this case on Azure(Microsofts server farm).

So you cant go about to whitelist the entire Azure since there definitely is malicious stuff on a big server cluster as that. (Guess that ship has kinda sailed thou since AWS is white listed)

So some options:

Check what list is blocking the IP/Range and remove it (Do your own custom list, option exists in Skynet). Thou try to find out why, might be a good reason why its blocked in the first place!!!!

White list CIDRs (or the built in AS option "check bgpview.io for AS-number" )

Stop using Skynet, (or any custom rules) Firewalls are maintenance that's why network engineers exists(if it was setup and forget Asus would have implemented it along time ago)

,

[dave14305]

v7.3.0 breaks dnsmasq, presumably because the ipset line in dnsmasq.conf.add is too long to be handled correctly, depending on the length of the shared whitelists.

[alexeyza]

Not sure if it is related to what @dave14305 said, but as soon as I upgraded to v7.3.0 (from v7.2.8), none of my devices were able to connect to the router. I couldn't even ssh to it or stop Skynet (had no way to see any errors or logs). I ended up factory resetting the router, and reinstalling everything. Later, a couple minutes after re-installing Skynet v7.3.0, same thing happened again. (I'm using RT-AC86U)

[TheStork239]

Same here @alexeyza; shortly after upgrading Skynet to 7.3.0 my router (AC88U) went totally unresponsive, so had to factory reset and restore to a set-up prior to 7.3.0. Now works again, but don't dare to apply 7.3.0.

[Adamm00]

So apparently people have huge shared whitelists? Beyond what diversion and Skynet add by default. I’ll revert this one when I get access to a computer in a few hours unless someone wants to make a pull request in the mean time… 🤦‍♂️

[alexeyza]

Actually, my whitelists have about 20 items in them (13 items in diversion + a few in Skynet), and whatever is the default Diversion + Skynet lists. So, not sure if that's the issue.

Thanks for looking into this.

Edit:

# wc -l /jffs/addons/shared-whitelists/*
52 /jffs/addons/shared-whitelists/shared-Diversion-whitelist

[Adamm00]

Anyone with the issue what’s the output of;

wc -l /jffs/addons/shared-whitelists/*

You can then follow @dave14305 instructions to temporarily fix dnsmasq until I can revert the commit (GitHub web somehow doesn’t allow reverting commits and I’m stuck on a train for another hour or two).

https://www.snbforums.com/threads/whats-appending-this-line-in-dnsmasq-conf-add.75457/post-721350

My apologies to anyone it affected, this has been working locally for weeks without issue.

[TheStork239]

Anyone with the issue what’s the output of;

wc -l /jffs/addons/shared-whitelists/*

I have only added 5 domains manually to the whitelist, in addition to default Skynet+Diversion. The output I get from the above command (having restored from a backup to 7.2.5) is:

45 /jffs/addons/shared-whitelists/shared-Diversion-whitelist
       13 /jffs/addons/shared-whitelists/shared-Skynet-whitelist
       22 /jffs/addons/shared-whitelists/shared-Skynet2-whitelist
       80 total

[Adamm00]

Going to try tackle this again soon. From a quick test it seems a line around 1000 characters (all domains from /jffs/addons/shared-whitelists/* combined) causes dnsmasq to fail.

Jan 13 21:34:40 Skynet: [i] Restarting Firewall Service Jan 13 21:34:40 rc_service: service 21463:notify_rc restart_firewall Jan 13 21:34:40 rc_service: waitting "start_dnsmasq" via watchdog ... Jan 13 21:34:51 Diversion: auto-added IPv6 entries to blocking list(s) and blacklist, this might have taken some time Jan 13 21:34:51 Diversion: restarted Dnsmasq to apply settings Jan 13 21:34:51 stubby[21596]: Stubby version: Stubby 0.4.0 Jan 13 21:34:51 dnsmasq[21598]: bad option at line 56 of /etc/dnsmasq.conf Jan 13 21:34:51 dnsmasq[21598]: FAILED to start up Jan 13 21:34:51 custom_script: Running /jffs/scripts/service-event-end (args: start dnsmasq)

I welcome anyone else to take a look as my free time is very limited recently.

[dave14305]

OpenWrt looks like it will go with the equivalent of one domain per line. https://github.com/openwrt/openwrt/pull/4667

[Adamm00]

https://github.com/Adamm00/IPSet_ASUS/commit/7e374be7a133bfb305462e06dcea3eb9f772ca29

For those of you wanting to try this out, there's a new branch to avoid a situation like last time. I'd appreciate any feedback.