kiwix-serve does not support IPv6

[ghost]

root@hertz-ms ~ # kiwix-serve kiwix/* -i 2a0f:9400:7a0a::5
Ip address 2a0f:9400:7a0a::5  is not a valid ip address

root@hertz-ms ~ # kiwix-serve kiwix/* -i "[2a0f:9400:7a0a::5]"
Ip address [2a0f:9400:7a0a::5]  is not a valid ip address

[kelson42]

I confirm, this is a current limitation ; but ipv6 is pretty rare in local networks.

[ghost]

I confirm, this is a current limitation ; but ipv6 is pretty rare in local networks.

No IPv6 is the latest IP version and is widely used already. Please consider adding IPv6 support, shouldn't be complex.

[kelson42]

It seem libmicrohttp can support both ipv4 and ipv6 at the same time or only ipv6 or only ipv4. See https://www.gnu.org/software/libmicrohttpd/manual/html_node/microhttpd_002dconst.html.

We have to choose the way how to go forward.

Being cautious, I tend to think that having first an opion —ipv6 or -6 would be the best approach.

[aryanA101a]

@kelson42 Can I work on this?

[kelson42]

@kelson42 Can I work on this?

Yes, if you have a plan how to implement it.

[aryanA101a]

@kelson42 Can I work on this?

Yes, if you have a plan how to implement it.

I'll proceed by adding ipv6 support in libkiwix, then proceed with kiwix-tools.

[aryanA101a]

It seem libmicrohttp can support both ipv4 and ipv6 at the same time or only ipv6 or only ipv4. See https://www.gnu.org/software/libmicrohttpd/manual/html_node/microhttpd_002dconst.html.

We have to choose the way how to go forward.

Being cautious, I tend to think that having first an opion —ipv6 or -6 would be the best approach.

MHD_USE_IPv6
Run using the IPv6 protocol (otherwise, MHD will just support IPv4). If you specify MHD_USE_IPV6 and the local platform does not support it, MHD_start_daemon will return NULL.

If you want MHD to support IPv4 and IPv6 using a single socket, pass MHD_USE_DUAL_STACK, otherwise, if you only pass this option, MHD will try to bind to IPv6-only (resulting in no IPv4 support).

MHD_USE_DUAL_STACK
Use a single socket for IPv4 and IPv6. Note that this will mean that IPv4 addresses are returned by MHD in the IPv6-mapped format (the ’struct sockaddr_in6’ format will be used for IPv4 and IPv6).

If I understood it correctly MHD_USE_DUAL_STACK flag will allow us to receive both ipv4 and ipv6 clients. And on testing it I found that it allows us to listen on ipv6 addr.

Regarding, plan of action:

• To serve both ipv4 and ipv6 traffic MHD_USE_DUAL_STACK has to be turned on.
• And I don't think there is a need for a new ipv6 flag, since we can check the format of user provided addr with inet_pton(AF_INET...) and inet_pton(AF_INET6...)

What do you think?

[kelson42]

@aryanA101a Have you considered all options? How they shoukd work?

[aryanA101a]

It seem libmicrohttp can support both ipv4 and ipv6 at the same time or only ipv6 or only ipv4. See https://www.gnu.org/software/libmicrohttpd/manual/html_node/microhttpd_002dconst.html. We have to choose the way how to go forward. Being cautious, I tend to think that having first an opion —ipv6 or -6 would be the best approach.

MHD_USE_IPv6
Run using the IPv6 protocol (otherwise, MHD will just support IPv4). If you specify MHD_USE_IPV6 and the local platform does not support it, MHD_start_daemon will return NULL.

If you want MHD to support IPv4 and IPv6 using a single socket, pass MHD_USE_DUAL_STACK, otherwise, if you only pass this option, MHD will try to bind to IPv6-only (resulting in no IPv4 support).

MHD_USE_DUAL_STACK
Use a single socket for IPv4 and IPv6. Note that this will mean that IPv4 addresses are returned by MHD in the IPv6-mapped format (the ’struct sockaddr_in6’ format will be used for IPv4 and IPv6).

If I understood it correctly MHD_USE_DUAL_STACK flag will allow us to receive both ipv4 and ipv6 clients. And on testing it I found that it allows us to listen on ipv6 addr.

Regarding, plan of action:

• To serve both ipv4 and ipv6 traffic MHD_USE_DUAL_STACK has to be turned on.
• And I don't think there is a need for a new ipv6 flag, since we can check the format of user provided addr with inet_pton(AF_INET...) and inet_pton(AF_INET6...)

I think that's not an appropriate solution. Since, we cannot assume that everyone has ipv6 support in their system as MHD_USE_DUAL_STACK causes MHD_start_daemon(...) to fail if there is no ipv6 support in their system.

The better plan of action would be: Add a -6 flag to kiwix-serve.

in libkiwix/src/internalServer.cpp

if -6 flag == true:
   add MHD_USE_DUAL_STACK to server flags
   if explicit ip addr not empty:
      create a socket with explicit ip addr
   else:
      listen on all ipv6 interfaces

[aryanA101a]

libkiwix/src/server/internalServer.cpp

if (m_addr.empty()) {
    if (0 != INADDR_ANY) {
      sockAddr.sin_addr.s_addr = htonl(INADDR_ANY);
    }
    m_addr = kiwix::getBestPublicIp();
  }

Q1. If INADDR_ANY is defined as 0, why are we checking if it is unequal to zero? Q2. if m_addr.empty() then the webserver serves on 0.0.0.0 and m_addr = kiwix::getBestPublicIp(); causes printed message on the terminal after kiwix-serve is called without an explicit ip addr to show that it is serving on one of the network interfaces present on the sys.

aryanarora@aryanarora:~/kiwix-tools$ kiwix-serve vikidia_en_all_maxi_2023-09.zim -p 2021
The Kiwix server is running and can be accessed in the local network at: http://192.168.122.210:2021

Is it the intended behavior ?

[aryanA101a]

libkiwix/src/server/internalServer.cpp

if (m_addr.empty()) {
    if (0 != INADDR_ANY) {
      sockAddr.sin_addr.s_addr = htonl(INADDR_ANY);
    }
    m_addr = kiwix::getBestPublicIp();
  }

Q1. If INADDR_ANY is defined as 0, why are we checking if it is unequal to zero? Q2. if m_addr.empty() then the webserver serves on 0.0.0.0 and m_addr = kiwix::getBestPublicIp(); causes printed message on the terminal after kiwix-serve is called without an explicit ip addr to show that it is serving on one of the network interfaces present on the sys.

aryanarora@aryanarora:~/kiwix-tools$ kiwix-serve vikidia_en_all_maxi_2023-09.zim -p 2021
The Kiwix server is running and can be accessed in the local network at: http://192.168.122.210:2021

Is it the intended behavior ?

@mgautierfr ?

[mgautierfr]

Q1. If INADDR_ANY is defined as 0, why are we checking if it is unequal to zero?

I would say that we want to support potential system where INADDR_ANY would not be defined to 0. In this case, we have to use this value, what ever it is. Which system has INADDR_ANY not equal to 0 ? I don't know. (Even if it is me who has written (or copied 🤫) this part of code)

Is it the intended behavior ?

We want to tell to the user on which IP address they can connect. 0.0.0.0 is not a valid address (as the define name implies (or not), we are listening on ALL interfaces of the system). And localhost or 127.0.0.1 will not work from another computer. We have to give the user the public address of their computer.

[aryanA101a]

how to build kiwix-tools for windows?

[kelson42]

how to build kiwix-tools for windows?

@mgautierfr ? With kiwix build?

[aryanA101a]

kiwix-build kiwix-tools --working-dir ~/build-kiwix-tools/ --config win32_static

kiwix-build is giving error. This is the log from /home/aryanarora/build-kiwix-tools/BUILD_win32_static/pugixml-1.2/meson-logs/meson-log.txt

OS: Fedora 39

Build started at 2024-04-17T22:08:55.429428
Main binary: /home/aryanarora/experiment/kb/bin/python
Build Options: -Dprefix=/home/aryanarora/build-kiwix-tools/BUILD_win32_static/INSTALL -Dlibdir=lib64 -Dbuildtype=release -Ddefault_library=static --cross-file=/home/aryanarora/build-kiwix-tools/BUILD_win32_static/meson_cross_file.txt
Python system: Linux
The Meson build system
Version: 1.4.0
Source dir: /home/aryanarora/build-kiwix-tools/SOURCE/pugixml-1.2
Build dir: /home/aryanarora/build-kiwix-tools/BUILD_win32_static/pugixml-1.2
Build type: cross build
Project name: pugixml
Project version: undefined
-----------
Detecting compiler via: `/usr/lib64/ccache/i686-w64-mingw32-g++ --version` -> 0
stdout:
i686-w64-mingw32-g++ (GCC) 13.2.1 20230728 (Fedora MinGW 13.2.1-5.fc39)
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-----------
Running command: /usr/lib64/ccache/i686-w64-mingw32-g++ -E -dM -
-----
-----------
Detecting linker via: `/usr/lib64/ccache/i686-w64-mingw32-g++ -Wl,--version -lwinmm -lshlwapi -lws2_32 -lssp` -> 0
stdout:
GNU ld version 2.40-4.fc39
Copyright (C) 2023 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
-----------
stderr:
collect2 version 13.2.1 20230728 (Fedora MinGW 13.2.1-5.fc39)
/usr/lib/gcc/i686-w64-mingw32/13.2.1/../../../../i686-w64-mingw32/bin/ld -plugin /usr/libexec/gcc/i686-w64-mingw32/13.2.1/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/i686-w64-mingw32/13.2.1/lto-wrapper -plugin-opt=-fresolution=/tmp/ccTTqpxy.res -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -plugin-opt=-pass-through=-lkernel32 -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-ladvapi32 -plugin-opt=-pass-through=-lshell32 -plugin-opt=-pass-through=-luser32 -plugin-opt=-pass-through=-lkernel32 -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -plugin-opt=-pass-through=-lkernel32 --sysroot=/usr/i686-w64-mingw32/sys-root -m i386pe -Bdynamic -u ___register_frame_info -u ___deregister_frame_info /usr/i686-w64-mingw32/sys-root/mingw/lib/../lib/crt2.o /usr/lib/gcc/i686-w64-mingw32/13.2.1/crtbegin.o -L/usr/lib/gcc/i686-w64-mingw32/13.2.1 -L/usr/lib/gcc/i686-w64-mingw32/13.2.1/../../../../i686-w64-mingw32/lib/../lib -L/usr/i686-w64-mingw32/sys-root/mingw/lib/../lib -L/usr/lib/gcc/i686-w64-mingw32/13.2.1/../../../../i686-w64-mingw32/lib -L/usr/i686-w64-mingw32/sys-root/mingw/lib --version -lwinmm -lshlwapi -lws2_32 -lssp -lstdc++ -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt -lkernel32 -lpthread -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt -lkernel32 /usr/lib/gcc/i686-w64-mingw32/13.2.1/crtend.o
-----------
Sanity testing C++ compiler: /usr/lib64/ccache/i686-w64-mingw32-g++
Is cross compiler: True.
Sanity check compiler command line: /usr/lib64/ccache/i686-w64-mingw32-g++ sanitycheckcpp.cc -o sanitycheckcpp_cross.exe -DWIN32 -I/home/aryanarora/build-kiwix-tools/BUILD_win32_static/INSTALL/include -D_FILE_OFFSET_BITS=64 -lwinmm -lshlwapi -lws2_32 -lssp
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----
Running test binary command:  /usr/bin/wine /home/aryanarora/build-kiwix-tools/BUILD_win32_static/pugixml-1.2/meson-private/sanitycheckcpp_cross.exe

meson.build:1:0: ERROR: Executables created by cpp compiler /usr/lib64/ccache/i686-w64-mingw32-g++ are not runnable.

[aryanA101a]

Do I proceed on to open a pr to add -liphlpapi flag to kiwixbuild/configs/win*.py ?

[kelson42]

@aryanA101a Not sure honestly, do like you think is right. What I'm sure, is that this is a bad idea to discuss in length Kiwix build topics here.

[veloman-yunkan]

Copying my comment from the PR review:

My feeling is that the option --ipv6 is not a good way to instruct the kiwix server to support IPv6. Is it supposed to enforce IPv6 or merely enables it?

An alternative may be to use special values for the -i/--address option. For example,

• all - listen on all available IP addresses (IPv4 and IPv6)
• ipv4 - listen on all available IPv4 addresses
• ipv6 - listen on all available IPv6 addresses
• auto
• network interface name (e.g. eth0, wlan0) - will be resolved to the IP address of the named interface

I think that we have to close in on the use model of this enhancement before proceeding with the PR.