glusterfs
glusterfs copied to clipboard
gluster
Compatibility with Alpine Linux
Hi, Alpine Linux use the musl library instead of the glibc. Is it possible to make a musl compatible version of glusterfs ? It would be awesome to have glusterfs in alpine linux.
Doesn't the musl-gcc wrapper provide the necessary magic to link to the platform specific 'C' Library on Alpine Linux ?
I think you're right, the musl-gcc wrapper provide some magic to link glic functions, but all the functions must be POSIX compliant. I read that glusterfs use not POSIX functions : https://bugs.alpinelinux.org/issues/2834 .
Portablexdr doesn't seem to work with glusterfs (usually rpcgen is provided by glibc but musl doesn't have it):
Making all in rpc/xdr/gen
glusterfs3-xdr.x:39: syntax error
Both 3.10.7 and master produce the same error.
rpcgen is in its own separate package. At least in alpine 3.9.2. (Similar to Fedora28+ and RHEL8 where rpcgen has been removed from glibc. I expect other distributions will do the same eventually.)
I didn't look in any older versions.
apk update && apk add rpcgen
But there are other issues with this rpcgen that alpine ships. It's not the same as the unbundled rpcgen in Fedora and doesn't like the "hyper" data type in, e.g., glusterfs4-xdr.x
apk info rpcgen says it's built from the source from/git.linux-nfs.org which I presume is really git://git.linux-nfs.org/projects/steved/nfs-utils.git, and rpcgen in that repo is deprecated according to steved.
Alpine should probably use the rpcgen that's in Fedora from https://github.com/thkukuk/rpcsvc-proto.git, which supports the "hyper" data type (i.e. quad_t).
Without support for hyper, nobody is going to get very far building glusterfs.
Someone with a vested interest in Alpine should get them to update to a modern/current rpcgen. In the mean time anyone wanting to build gluster will absolutely have to build their own rpcgen from source.
Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.
Two things here. I tried to get this done last year, and the changes in xdr lib was little too much, and there were discussions about newer network layer, and I left it off at that.
Considering we from kadalu.io would like to have a smaller footprint of glusterfs in container image, I will see if we can get this picked up. If anyone wants to volunteers, (with protobuf experience), happy to coordinate.
Gave this a try... stuck with below error, and not sure how to go about that:
/usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: ../../libglusterfs/src/.libs/libglusterfs.so: undefined reference to `swapcontext'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: ../../libglusterfs/src/.libs/libglusterfs.so: undefined reference to `getcontext'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: ../../libglusterfs/src/.libs/libglusterfs.so: undefined reference to `makecontext'
collect2: error: ld returned 1 exit status
@xhernandez According to this link below is mentioned:
With the incorporation of the ISO/IEC 9899:1999 standard into this specification it was found that the ISO C standard (Subclause 6.11.6) specifies that the use of function declarators with empty parentheses is an obsolescent feature. Therefore, using the function prototype:
void makecontext(ucontext_t *ucp, void (*func)(), int argc, ...);is making use of an obsolescent feature of the ISO C standard. Therefore, a strictly conforming POSIX application cannot use this form. Therefore, use of
getcontext(),makecontext(), andswapcontext()is marked obsolescent.
Looks like its not supported in musl-dev at all, and hence glusterfs is not able to link fine.
To get to this, I needed to do few changes in code, including not building gfapi (more errors seen). I will update later about all dependencies and changes done. The rpcgen issues etc are gone now, and libtirpc-dev also is available. So all good there.
With below dockerfile
FROM alpine:latest
RUN apk add git automake autoconf libtool build-base rpcgen flex bison openssl-dev
RUN apk add libuuid util-linux-dev acl-dev zlib-dev python3-dev libxml2-dev argp-standalone
RUN apk add libtirpc-dev userspace-rcu-dev fuse-dev
RUN git clone --depth 1 https://github.com/amarts/glusterfs_fork --branch alpine_compile --single-branch glusterfs
RUN cd glusterfs && ./autogen.sh && GF_LDFLAGS="/usr/lib/libargp.a" ./configure --without-gfapi && make
# Debugging, Comment the above line and
# uncomment below line
ENTRYPOINT ["tail", "-f", "/dev/null"]
and docker build -t glusterfs-alpine -f ./Dockerfile.alpine . as command I was able to test this.
Gave this a try... stuck with below error, and not sure how to go about that:
/usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: ../../libglusterfs/src/.libs/libglusterfs.so: undefined reference to `swapcontext' /usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: ../../libglusterfs/src/.libs/libglusterfs.so: undefined reference to `getcontext' /usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: ../../libglusterfs/src/.libs/libglusterfs.so: undefined reference to `makecontext' collect2: error: ld returned 1 exit status@xhernandez According to this link below is mentioned:
With the incorporation of the ISO/IEC 9899:1999 standard into this specification it was found that the ISO C standard (Subclause 6.11.6) specifies that the use of function declarators with empty parentheses is an obsolescent feature. Therefore, using the function prototype:
void makecontext(ucontext_t *ucp, void (*func)(), int argc, ...);is making use of an obsolescent feature of the ISO C standard. Therefore, a strictly conforming POSIX application cannot use this form. Therefore, use ofgetcontext(),makecontext(), andswapcontext()is marked obsolescent.Looks like its not supported in musl-dev at all, and hence glusterfs is not able to link fine.
Without support for makecontext() and similar functions, it will be hard to make gluster run on this platform. One possibility is to use a setjmp()/longjmp() with some tricks for stack allocation to simulate those functions, but creation of synctasks will probably be slow if we want to use a portable method.
Another way is to implement those functions ourselves, but it's not trivial because we would need to do it at very low level (probably assembly) and we'll need to do it for each machine type we want to be able to run Gluster. Probably this is not the right approach.
Ack, I agree. Currently a lot of logic depends on syncop* logic. I wouldn't pursue this further now, but happy if anyone has a way to get this resolved in alpine. As per basic searches, even openssl had issue, but they run in minimal no-async mode on alpine. We can't even run our server only version too.
I will keep the issue open, and keep the branch alive, so anyone interested can pick it up and take it forward. For now, focusing on few other things.
Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.
Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.
The problem wit make/swap/getcontext and a similar with argp_parse functions etc. can be solved by the following commands:
apk add libucontext-dev
apk add argp_standalone
export LIBS="-lucontext -largp "
./configure
Though there are some more issues, when compiling on alpine, for example that __offt64 is not defined and _PART_MOUNTED is not defined and one more i don't remember. I will try to make some work arounds for this and post it here again ...
I tested this with the release-10 branch
Ok i could now compile glusterfs with musl-libc on alpine 3.15 and gcc 10.3. I made some minor changes to files, where something was undefined.
At least i added
#define _PATH_MOUNTED "/etc/mtab" in some files, idk why it wasn't defined.
also in /contrib/fuse-lib/mount-common.c strncmp was undefined. so i added #include <string.h>
I think i did one more thing i don't remember at moment, but i'll make a diff later, to find it.
Next step is to test on my little cluster, if the setup of a filesystem works also.
So here are all changes:
--- ../glusterfs-release-10/contrib/fuse-lib/mount-common.c
+++ contrib/fuse-lib/mount-common.c
@@ -8,7 +8,12 @@
*/
#include "mount-gluster-compat.h"
+#ifndef _PATH_MOUNTED
+#define _PATH_MOUNTED "/etc/mtab"
+#endif
+#include <string.h>
--- ../glusterfs-release-10/api/src/glfs.h
+++ api/src/glfs.h
@@ -67,6 +67,7 @@
* is false and __USE_FILE_OFFSET64 is true, then go on to define
* off64_t using __off64_t.
*/
+#define __off64_t_defined
#ifndef GF_BSD_HOST_OS
#if defined(__USE_FILE_OFFSET64) && !defined(__off64_t_defined)
typedef __off64_t off64_t;
--- ../glusterfs-release-10/contrib/fuse-util/fusermount.c
+++ contrib/fuse-util/fusermount.c
@@ -6,6 +6,9 @@
See the file COPYING.
*/
/* This program does the mounting and unmounting of FUSE filesystems */
+#ifndef _PATH_MOUNTED
+#define _PATH_MOUNTED "/etc/mtab"
+#endif
#include <config.h>
--- ../glusterfs-release-10/libglusterfs/src/glusterfs/dict.h
+++ libglusterfs/src/glusterfs/dict.h
@@ -320,6 +320,7 @@
dict_set_uint64(dict_t *this, char *key, uint64_t val);
/* POSIX-compliant systems requires the 'time_t' to be a signed integer. */
+#define __WORDSIZE 64
#if __WORDSIZE == 64
#define dict_get_time(dict, key, val) dict_get_int64((dict), (key), (val))
#define dict_set_time(dict, key, val) dict_set_int64((dict), (key), (val))
--- ../glusterfs-release-10/xlators/mgmt/glusterd/src/glusterd-utils.c
+++ xlators/mgmt/glusterd/src/glusterd-utils.c
@@ -7,6 +7,9 @@
later), or the GNU General Public License, version 2 (GPLv2), in all
cases as published by the Free Software Foundation.
*/
+#ifndef _PATH_MOUNTED
+#define _PATH_MOUNTED "/etc/mtab"
+#endif
#include <inttypes.h>
#if defined(GF_LINUX_HOST_OS)
Maybe some of them make no sense, i just added them so that it can compile and to show where the problems are, not to make it work correctly
Hint: The target-triplet of the gcc was aarch64-alpine-linux-musl
Update: in musl-libcs unistd.h they define off64_t as following:
#if defined(_LARGEFILE64_SOURCE) || defined(_GNU_SOURCE)
....
#define off64_t off_t
#endif
so in api/src/glfs.h maybe that could be checked too? like:
#if (defined(_LARGEFILE64_SOURCE) || defined(_GNU_SOURCE)) && !defined(__off64_t_defined)
#define __off64_t_defined
#endif
Update2:
_PATH_MOUNTED is in glibc defined by <mntent.h>, which includes <path.h> where it is defined. But musl's <mntent.h> doesn't include <path.h>. So two ways of solving this are:
- use the MOUNTED macro but i think thats deprecated?
- replace <mntent.h> with/add #include <paths.h>
rpic4:~# uname -a
Linux rpic4 5.15.4-0-rpi4 #1-Alpine SMP PREEMPT Mon Nov 22 10:43:31 UTC 2021 aarch64 GNU/Linux
rpic4:~# gluster peer status
Number of Peers: 3
Hostname: rpic2
Uuid: 2d5904f3-3dfe-47a8-b293-b373349721de
State: Peer in Cluster (Connected)
Hostname: rpic1
Uuid: 5e3b90cf-7fb0-4340-b09f-ebee503f0e95
State: Peer in Cluster (Connected)
Hostname: rpic3
Uuid: d2c27958-88de-4447-bc87-ba1eeaa88b4c
State: Peer in Cluster (Connected)
rpic4:~# gluster volume info
Volume Name: gluster1
Type: Replicate
Volume ID: 1e98d5ff-f28a-47b3-8ad0-64c2633055da
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: rpic1:/cluster/gluster1/brick1/volume
Brick2: rpic2:/cluster/gluster1/brick2/volume
Brick3: rpic3:/cluster/gluster1/brick3/volume
Brick4: rpic4:/cluster/gluster1/brick4/volume
Options Reconfigured:
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
rpic4:~# gcc -dumpmachine
aarch64-alpine-linux-musl
Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.
Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.
@amarts We are doing a triage of the bugs that we are going to fix. Do you have bandwidth to work on this? Otherwise we can go ahead and close it. If you have partial solution somewhere, please publish it, so that we can refer to this in the future if we want to pick it up.
