Swift backend support

[constantinius]

trafficstars

Adding OpenStack Swift cache backend using modified keystone-client and swift-client libraries.

[constantinius]

I used these libraries as a baseline to access swift/keystone:

• https://github.com/ukyg9e5r6k7gubiekd6/swift-client
• https://github.com/ukyg9e5r6k7gubiekd6/keystone-client

I'd love to have some feedback on how I integrated it into mapcache. Currently this functionality is not really tested in production, so this PR stays a draft.

keystone-client uses json-c (added as cmake dependency), I'm not sure if this is done correctly. I'm aware, that mapcache ships with its own JSON implementation (cJSON), but I did not yet make an effort to replace either.

[constantinius]

@tbonfort

I updated the source code and now support keystone v3 API. In principle, this works, we already use it in a staging environment. Unfortuately, we run into realloc(): invalid next size when using with mapcache_seed (and sometimes in the Apache module too).

With gdb, we got this stack trace:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fde75254801 in __GI_abort () at abort.c:79
#2  0x00007fde7529d897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fde753cab9a "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007fde752a490a in malloc_printerr (str=str@entry=0x7fde753c8ed3 "realloc(): invalid next size") at malloc.c:5350
#4  0x00007fde752a99b4 in _int_realloc (av=av@entry=0x7fde5c000020, oldp=oldp@entry=0x7fde5c024080, oldsize=oldsize@entry=64, nb=nb@entry=64)
    at malloc.c:4534
#5  0x00007fde752acf9b in __GI___libc_realloc (oldmem=0x7fde5c024090, bytes=56) at malloc.c:3230
#6  0x00007fde7719376e in swift_set_object () from /usr/lib/x86_64-linux-gnu/libmapcache.so.1
#7  0x00007fde7716fb62 in ?? () from /usr/lib/x86_64-linux-gnu/libmapcache.so.1
#8  0x00007fde77165f4e in mapcache_cache_tile_exists () from /usr/lib/x86_64-linux-gnu/libmapcache.so.1
#9  0x00005602271e7d7d in examine_tile ()
#10 0x00005602271e807b in cmd_recurse ()
#11 0x00005602271e866c in feed_worker ()
#12 0x00005602271e872b in ?? ()
#13 0x00007fde6dbb96db in start_thread (arg=0x7fde6257a700) at pthread_create.c:463
#14 0x00007fde7533588f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

This points to an issue in swift_set_object, where I basically duplicate a string (the re-allocation). From my perspective it looks clean but I'm suspecting, that there is some issue with connection pooling (construction, destruction), but I can't figure it out. Could you maybe have a look at it and tell me if I use the interfaces correctly?

[constantinius]

I think I found the issue, a classic buffer overflow (re-)allocating strlen bytes, but strcpy writes a final '\0' character out of the allocated area.

This PR is good to go from my perspective. @tbonfort please let me know if need anything in addition.

[jbo-ads]

Have you figured out the causes of failed checks?

[constantinius]

@jbo-ads

Yes, I think I solved the issues

[jbo-ads]

I've checked your updates following @tbonfort's comments and it looks OK in my opinion.

About licenses, I think there is nothing wrong with using LGPLv3 source code in a bigger MIT licensed project.

I don't know the prefered way of mentioning it: copy license text at the beginning of related code (only swift-client.h and swift-client.c, since no license is mentioned in keystone-client repo), or include it in MapCache LICENSE file with a note indicating that only swift-client is concerned with LGPLv3. @jmckenna any advice on this?

[jmckenna]

@jbo-ads I prefer the second option, but I think it should be noted in several places:

• source LICENSE file
• source README.rst (so it displays nicely on Github)
• add a 'copyright.txt' file in the MapCache documentation, similar to the one for MapServer at https://mapserver.org/copyright.html (so web users can see the full license for MapCache)

[jbo-ads]

@constantinius - Please don't forget to update the documentation with this new feature: https://mapserver.org/mapcache/caches.html

[Schpidi]

Unfortunately the latest commit breaks building for me:

[ 95%] Linking C executable mapcache.fcgi
cd /build/mapcache-1.10.0/obj-x86_64-linux-gnu/cgi && /usr/bin/cmake -E cmake_link_script CMakeFiles/mapcache.fcgi.dir/link.txt --verbose=1
/usr/bin/cc -g -O2 -fdebug-prefix-map=/build/mapcache-1.10.0=. -fstack-protector-strong -Wformat -Werror=format-security -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Werror=declaration-after-statement  -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -rdynamic CMakeFiles/mapcache.fcgi.dir/mapcache.c.o  -o mapcache.fcgi -Wl,-rpath,/build/mapcache-1.10.0/obj-x86_64-linux-gnu: ../libmapcache.so.1.10.0 -lfcgi -lpng -lz -ljpeg -lcurl -lapr-1 -laprutil-1 -lpixman-1 -lgdal -lpcre -lsqlite3 -lpq -ltiff -ljson-c -ldl -lm 
../libmapcache.so.1.10.0: undefined reference to `keystone_get_service_url'
../libmapcache.so.1.10.0: undefined reference to `keystone_set_auth_version'
../libmapcache.so.1.10.0: undefined reference to `keystone_start'
../libmapcache.so.1.10.0: undefined reference to `keystone_end'
../libmapcache.so.1.10.0: undefined reference to `keystone_authenticate'
../libmapcache.so.1.10.0: undefined reference to `keystone_get_auth_token'
../libmapcache.so.1.10.0: undefined reference to `keystone_set_debug'
collect2: error: ld returned 1 exit status