nextcloudpi
nextcloudpi copied to clipboard
nextcloud
ncp-diag: port checks fail, if customer has a dual stack lite (DS Lite) connection
This is a bug, which is independent from issue #1142, which was due to a change in the portchecker.co site setup. This new issue also is falsely reporting the ports closed while they are in fact open.
My provider gives me a set of unique IPv6 addresses, but he does not provide a unique IPv4 address! Instead, I share my IPv4 address with up to 60 other users, and the address translation for incoming and outgoing traffic is executed at the provider level. This is called dual stack lite or DS Lite in Germany.
Accordingly, I am only reachable from the internet via my IPv6 addresses, but not via my IPv4 address! Nevertheless, my IPv4 address can normally be accessed from the internet - only I cannot be reached.
Result from ncp-diag:
>>> You should open your ports for Lets Encrypt and external Access <<<
NextCloudPi version|v1.48.2
OS|Debian GNU/Linux 11. 5.10.0-16-amd64 (x86_64)
automount|yes
USB devices|sda sdb
datadir|/media/USB***
data in SD|yes
data filesystem|btrfs
data disk usage|213G/932G
rootfs usage|19G/25G
swapfile|/dev/mmcblk1p3
dbdir|/media/USB***
Nextcloud check|ok
Nextcloud version|23.0.7.2
HTTPD service|up
PHP service|up
MariaDB service|up
Redis service|up
HPB service|up
Postfix service|up
internet check|ok
>>> port check 80|closed <<<
>>> port check 443|closed <<<
IP|192.168.***
gateway|192.168.***
interface|enp1s0
certificates|***
NAT loopback|no
uptime|1:11
I took a look at ncp-diag. In lines 67 til 87 the ports 80 and 443 are tested, if they are open:
function is_port_open()
{
local port=$1
local public_ip
public_ip="$(curl -m4 -4 https://icanhazip.com 2>/dev/null)" || { echo "closed"; return 1; }
if [[ "${public_ip}" == "" ]]; then
public_ip="$(curl -m4 -6 https://icanhazip.com 2>/dev/null)" || { echo "closed"; return 1; }
fi
local tmp_file=$(mktemp)
local token=$(wget -T2 -t1 -qO- --keep-session-cookies --save-cookies $tmp_file https://portchecker.co | grep -oP "_csrf\" value=\"\K.*\"")
if [[ "${token}" != "" ]]; then
wget -T2 -t1 -qO- --load-cookies $tmp_file https://portchecker.co --post-data "target_ip=${public_ip}&port=${port}&_csrf=${token::-1}" \
| grep -q '<span class="green">open</span>' && { echo "open"; return 1; }
fi
rm $tmp_file
echo "closed"
}
echo "port check 80|$( is_port_open 80 )"
echo "port check 443|$( is_port_open 443 )"
If I understand this code correctly, in case no IPv4 address can be retrieved, i.e. icanhazip.com throws an error, it will report "closed". With other words: if the server has no public IPv4 address, the ports will falsely be reported as closed, since no check occurs for a valid IPv6 address! The same in my case: icanhazip.com retrieves a valid IPv4 address, but obviously the port checks will report as closed. The same here: no check occurs for a valid IPv6 address.
In order to cover all possibilities i believe it is necessary to check for a valid IPv6 address and execute the consequent port checks in case, the IPv4 checks run into an error or report closed!
This could be done with the following lines, which would replace the lines 67 to 87 from above. (In this new code the suggested change from issues #1475 and #1486 is included also.)
function portcheck()
{
local public_ip=$1
local port=$2
local tmp_file=$(mktemp)
local token=$(wget -T2 -t1 -qO- --keep-session-cookies --save-cookies $tmp_file https://portchecker.co | grep -oP "_csrf\" value=\"\K.*\"")
if [[ "${token}" != "" ]]; then
wget -T2 -t1 -qO- --load-cookies $tmp_file https://portchecker.co/check --post-data "target_ip=${public_ip}&port=${port}&_csrf=${token::-1}" \
| grep -q '<span class="green">open</span>' && { echo "open"; return 1; }
fi
rm $tmp_file
echo "closed"
}
function is_port_open()
{
local port=$1
local answer
local public_ip
public_ip="$(curl -m4 -4 https://icanhazip.com 2>/dev/null)" || { echo ""; }
if [[ "${public_ip}" != "" ]]; then
answer=$( portcheck ${public_ip} ${port} )
fi
if [[ "${answer}" == "closed" ]]; then
public_ip="$(curl -m4 -6 https://icanhazip.com 2>/dev/null)" || { echo "closed"; return 1; }
if [[ "${public_ip}" != "" ]]; then
answer=$( portcheck ${public_ip} ${port} )
{ echo "${answer}"; return 1; }
fi
else
{ echo "${answer}"; return 1; }
fi
}
echo "port check 80|$( is_port_open 80 )"
echo "port check 443|$( is_port_open 443 )"
Since I am a beginner with bash this code needs review and is definitely not the most elegant, but I hope you get my idea.
Thank you for the detailed report and solution proposal. I'll look into it soon
Hi Calcaholic,
thank you for taking care of NextcloudPi now - I really appreciate it!
I have checked my bash code above, and it contains an error. I have corrected it in the following version:
function portcheck()
{
local public_ip=$1
local port=$2
local tmp_file=$(mktemp)
local token=$(wget -T2 -t1 -qO- --keep-session-cookies --save-cookies $tmp_file https://portchecker.co | grep -oP "_csrf\" value=\"\K.*\"")
if [[ "${token}" != "" ]]; then
wget -T2 -t1 -qO- --load-cookies $tmp_file https://portchecker.co/check --post-data "target_ip=${public_ip}&port=${port}&_csrf=${token::-1}" \
| grep -q '<span class="green">open</span>' && { echo "open"; return 1; }
fi
rm $tmp_file
echo "closed"
}
function is_port_open()
{
local port=$1
local answer
local public_ip
public_ip="$(curl -m4 -4 https://icanhazip.com 2>/dev/null)" || public_ip=""
if [[ "${public_ip}" != "" ]]; then
answer=$( portcheck ${public_ip} ${port} )
else
answer="closed"
fi
if [[ "${answer}" == "closed" ]]; then
public_ip="$(curl -m4 -6 https://icanhazip.com 2>/dev/null)" || { echo "closed"; return 1; }
if [[ "${public_ip}" != "" ]]; then
answer=$( portcheck ${public_ip} ${port} )
{ echo "${answer}"; return 1; }
fi
else
{ echo "${answer}"; return 1; }
fi
}
echo "port check 80|$( is_port_open 80 )"
echo "port check 443|$( is_port_open 443 )"
Thank you. However, it won't be that easy to integrate this, because portcheck.co, our service for checking if ports are open, does not support ipv6 (and I've not yet found a replacement that does and doesn't require an API key)
Hi @theCalcaholic , firstly, I would like to thank you very much for doing all this work for the ones like me, who would be lost without it! I really, really appreciate it.
And thanks again for looking up this issue. I believe I found out what happens when you are utilizing portcheck.co with an ipv6 address. So, my setup:
- Public ipv4 address behind NAT at my DSL provider = ports 80 and 443 are closed; my router is NOT reachable via ipv4.
- Public individual ipv6 address = ports 80 and 443 are opened by me
If I navigate to portcheck.co, and I enter my public ipv4 address (or get it inserted by clicking on "Use current IP") I get the following result (ipv4 address masked):
If I enter my public ipv6 address now, I get the following result (ipv6 address masked):
Now I did two more checks:
- Entering a false ipv6 address by changing the 2a02... to a 1a02... Result: Port 80 is closed
- Closing my port 80 at my router and checked it again. Result: Internal Server Error !!!
I also ran the modified bash file for ncp-diag (see above) again on my server: It reports 'open' correctly.
My conclusions from this are:
a) portcheck.co works with ipv6 addresses and reports the port correctly as 'open'.
b) In case the port at the valid ipv6 address is closed, there is a program bug in portcheck.co internally, which leads to an internal server error.
c) The case of an internal server error executing portcheck.co is handled inside ncp-diag in such a way, that this will be reported as 'closed', Therefore, I do believe it is safe to allow the checks of ipv6 addresses in ncp-diag. If this server error is corrected, portcheck.co would just correctly report 'closed', as it is now correctly reporting 'open'.
If you or anybody else has an individual ipv6 address besides the public ipv4 address, it should be possible to verify this by entering this ipv6 address in portcheck.co and shortly opening/closing a port .
I would really love to support in solving this issue in ncp-diag. I will just elaborate on this in my following comment.
Hi @theCalcaholic, I would like to support in solving this issue. Especially, since it is not a major bug which has any relevance for the functioning of NCP. It is rather a little annoyance of displaying a wrong result. So it would be good for me to improve my proficiency.
If the conclusions in my previous comment are verified successfully, I would like to do a pull request to modify ncp-diag. Since issue #1509 is related to ncp-diag also, I will address them both in this pull request.
Since this is the first time I am doing a pull request, I am uncertain how to set it up correctly:
- What kind of branch is the correct one to choose? devel? master-copy?
- What else is important to know?
Please, bear with me, if this is somewhere written for newbies. Somehow I failed to find it.
@LittleAlf The best is to send a pull request towards Devel, @theCalcaholic is currently on a train and won't be available for a little bit :)
I've already created a solution for this and tested it for IPv4 & IPv6, pasted a copy below of my draft code, right now I'm just waiting for @theCalcaholic to have some time over to check it out and see what he thinks before sending a PR about it :slightly_smiling_face:
# CHECK PORTS IPV4
IPV4="$(dig +short txt ch whoami.cloudflare @1.0.0.1)"; \
DAPORT="443"; \
tmp_file=$(mktemp); \
token=$(wget -T2 -t1 -qO- \
--keep-session-cookies \
--save-cookies $tmp_file \
https://portchecker.co | grep -oP "_csrf\" value=\"\K.*\""); \
wget -T2 -t1 -qO- --load-cookies $tmp_file https://portchecker.co/check \
--post-data "target_ip=${IPV4}&port=${DAPORT}&_csrf=${token::-1}" | \
grep -q '<span class="green">open</span>' && \
{ echo "IPv4 Port ${DAPORT} is open"; } || { echo "IPv4 Port ${DAPORT} is closed"; }; \
rm "$tmp_file"
# CHECK PORTS IPV6
IPV6="$(curl ifconfig.co)"; \
DAPORT="443"; \
tmp_file=$(mktemp); \
token=$(wget -T2 -t1 -qO- \
--keep-session-cookies \
--save-cookies $tmp_file \
https://portchecker.co | grep -oP "_csrf\" value=\"\K.*\""); \
wget -T2 -t1 -qO- --load-cookies $tmp_file https://portchecker.co/check \
--post-data "target_ip=${IPV6}&port=${DAPORT}&_csrf=${token::-1}" | \
grep -q '<span class="green">open</span>' && \
{ echo "IPv6 Port ${DAPORT} is open"; } || { echo "IPv6 Port ${DAPORT} is closed"; }; \
rm "$tmp_file"
@ZendaiOwl: Thank you for your solution. I see, that you replaced the determination of the ip addresses, and report now both: port status at ipv4 and ipv6.
Well, here is my code now. I tried to get rid of the dublicated code segments, and it still gives just one report for each port:
function portcheck()
{
local public_ip=$1
local port=$2
local tmp_file token
tmp_file=$(mktemp)
token=$(wget -T2 -t1 -qO- --keep-session-cookies --save-cookies "$tmp_file" https://portchecker.co | \
grep -oP "_csrf\" value=\"\K.*\"")
if [[ "${token}" != "" ]]; then
wget -T2 -t1 -qO- --load-cookies "$tmp_file" https://portchecker.co/check --post-data \
"target_ip=${public_ip}&port=${port}&_csrf=${token::-1}" \
| grep -q '<span class="green">open</span>' && { echo "open"; return 1; }
fi
rm "$tmp_file"
echo "closed"
}
function is_port_open()
{
local port=$1
local answer public_ip
local -i ip_ver=4
while [ $ip_ver -le 6 ]; do
public_ip="$(curl -m4 -"${ip_ver}" https://icanhazip.com 2>/dev/null)" || public_ip=""
answer="closed"
if [[ "${public_ip}" != "" ]]; then
answer=$( portcheck "${public_ip}" "${port}" )
fi
if [[ "${answer}" == "closed" ]]; then
ip_ver+=2
else
ip_ver=8
fi
done
{ echo "${answer}"; [[ "$answer" == "open" ]] && return 0 || return 1; }
}
echo "port check 80|$( is_port_open 80 )"
echo "port check 443|$( is_port_open 443 )"
Please, allow me a few newbie questions:
- What happens, if there is no ip address, either ipv4 or ipv6? Will it run into an error?
- Do you think it is advantageous to report both ip versions? I think most users won't care. They just wanna see, that everything is 'OK'.
- If I read it correctly, you are putting everything on a single line using the backslash: What is the advantage of this?
- And finally: how did you get the colored &&, ||, semicolons and 'echo' into your code block? How could you get the background of mentioning me with a light yellow background?
Many thanks for your help.
@LittleAlf and @ZendaiOwl Thanks for checking this again. I don't know what went wrong when I attempted to use portchecker.co for an ipv6 address, but it's obviously working so we should fix this.
Regarding ipv4 and ipv6 status, I didn't think about this before, but it's probably not a bad idea to show both status (ipv4 and ipv6) separately, because debugging ipv6 connections to my router was a pain in the ass in my experience and this status might be useful for people who attempt to set this up. I could imagine something like port check 80|ok (ipv4) or port check 80|ok (ipv4/ipv6)
@LittleAlf You can get syntax highlighting in code blocks for shell scripts like this:
```sh your_code_here() { ; } ```
that will be rendered like
your_code_here() { ; }
@theCalcaholic You almost read my mind! :smile: haha
You're welcome :pray:
Here is the PR: https://github.com/nextcloud/nextcloudpi/pull/1537
The new is_port_open function, that checks for both IPv4 & IPv6 and tells you which one or both is open/closed, also if an external IP was N/A or if the token was unable to be fetched.
function is_port_open()
{
local port public_ipv4 public_ipv6 tmp_file token ipv4_check ipv4_port ipv6_check ipv6_port
port="$1"
ipv4_check=""
ipv4_port=""
ipv6_check=""
ipv6_port=""
public_ipv4="$(curl -s -m4 -4 https://icanhazip.com 2>/dev/null)" || { ipv4_check="1"; }
public_ipv6="$(curl -s -m4 -6 https://icanhazip.com 2>/dev/null)" || { ipv6_check="1"; }
if [[ "$ipv4_check" == "1" ]] && [[ "$ipv6_check" == "1" ]]
then
echo "Error - IPv4 & IPv6: N/A, couldn't get external IP."; return 1;
fi
tmp_file=$(mktemp)
token=$(wget -T2 -t1 -qO- --keep-session-cookies --save-cookies $tmp_file https://portchecker.co | grep -oP "_csrf\" value=\"\K.*\"")
if [[ "${token}" != "" ]]
then
if [[ "$ipv4_check" != "1" ]] && [[ "$ipv6_check" == "1" ]]
then
wget -T2 -t1 -qO- --load-cookies $tmp_file https://portchecker.co/check \
--post-data "target_ip=${public_ipv4}&port=${port}&_csrf=${token::-1}" | grep -q '<span class="green">open</span>' && \
{ echo "open IPv4 (IPv6: N/A)"; return 1; } || { echo "closed IPv4 (IPv6: N/A)"; return 1; }
elif [[ "$ipv4_check" == "1" ]] && [[ "$ipv6_check" != "1" ]]
then
wget -T2 -t1 -qO- --load-cookies $tmp_file https://portchecker.co/check \
--post-data "target_ip=${public_ipv6}&port=${port}&_csrf=${token::-1}" | grep -q '<span class="green">open</span>' && \
{ echo "open IPv6 (IPv4: N/A)"; return 1; } || { echo "closed IPv6 (IPv4: N/A)"; return 1; }
else
wget -T2 -t1 -qO- --load-cookies $tmp_file https://portchecker.co/check \
--post-data "target_ip=${public_ipv4}&port=${port}&_csrf=${token::-1}" | grep -q '<span class="green">open</span>' && \
{ ipv4_port="open"; } || { ipv4_port="closed"; }
wget -T2 -t1 -qO- --load-cookies $tmp_file https://portchecker.co/check \
--post-data "target_ip=${public_ipv6}&port=${port}&_csrf=${token::-1}" | grep -q '<span class="green">open</span>' && \
{ ipv6_port="open"; } || { ipv6_port="closed"; }
if [[ "$ipv4_port" == "open" ]] && [[ "$ipv6_port" == "open" ]]
then
echo "open IPv4 & IPv6"; return 1;
fi
if [[ "$ipv4_port" == "open" ]] && [[ "$ipv6_port" == "closed" ]]
then
echo "open IPv4 (IPv6: closed)"; return 1;
elif [[ "$ipv4_port" == "closed" ]] && [[ "$ipv6_port" == "open" ]]
then
echo "open IPv6 (IPv4: closed)"; return 1;
else
echo "closed IPv4 & IPv6"; return 1;
fi
fi
else
echo "Error - Couldn't obtain a token for port check"; return 1;
fi
rm "$tmp_file"
}
Please, allow me a few newbie questions:
Of course ^^ I consider myself a newbie still tbh :smile:
What happens, if there is no ip address, either ipv4 or ipv6? Will it run into an error?
I've rewritten the function properly now so it checks for both IPv4 & IPv6 and should it fail to fetch a public IP for both it will give an error saying that was the problem, it also gives an error depending on what part of the script went wrong, either fetching a public IPv4/6 or the token for the port check. It will then tell you if the port is open or closed on IPv4 and IPv6 ^^
No external IP error message: echo "Error - IPv4 & IPv6: N/A, couldn't get external IP."; return 1;
Do you think it is advantageous to report both ip versions? I think most users won't care. They just wanna see, that everything > is 'OK'.
I don't know 🙏 but yes I agree, it could be so, that some users won't care about these details, however there are also users who can only access their instance via IPv6 and so it becomes more of a necessity for them to be able to check or see if their IPv6 port is open as well.
If I read it correctly, you are putting everything on a single line using the backslash: What is the advantage of this?
That was merely some draft code that was capable of being executed directly in the terminal to try out a "proof of concept" so to speak, the ( ";" ) is used to execute for the command before the semi-colon and the backslash makes it continue reading on the next line for the next command
And finally: how did you get the colored &&, ||, semicolons and 'echo' into your code block? How could you get the background of mentioning me with a light yellow background?
theCalcoholic already answered that one in the post above mine :cat:
Many thanks for your help.
@LittleAlf Of course :pray: and you're welcome
You can see the PR I made in the post above, with the new function as well :smile:
@ZendaiOwl Great! I really like it. 👍
I saw that the PR is opened now. I have put in there a suggestion for a small change: I do not like so much duplicated code sections. 😉