guru Organicmaps fixes

@vitaly-zdanevich

May 06 '24 20:05 gerion0

Can organicmaps and the other ebuild be versioned instead of being live version only? I think organicmaps has tags/releases.

May 06 '24 23:05 stkw0

Can organicmaps and the other ebuild be versioned instead of being live version only? I think organicmaps has tags/releases.

fast_double_parser cannot be a released version, yet. There exist releases but only in the git version an install target exists (the older versions just compile the sources but do not install any header/library), see https://github.com/lemire/fast_double_parser/commit/2a3319d6987dac627d235c5f2a9ccc569d84ddfc.

For organicmaps, there could be a release version. I will try to add it.

May 07 '24 00:05 gerion0

The last version is from 2022, I guess it would make sense to open an issue upstream to see how is the situation and if a new version could be released. If not it would make more sense to git the snapshot of a given date and use the date as version in the ebuild.

May 07 '24 09:05 stkw0

Thankfully, @lemire released a new version, so organicmaps now depends on that version.

May 07 '24 15:05 gerion0

Big thanks for fixes, today or tomorrow I will try to run this.

May 07 '24 19:05 vitaly-zdanevich

Can organicmaps and the other ebuild be versioned instead of being live version only? I think organicmaps has tags/releases.

There are releases but none of them seem to be downloadable for the desktop version. One could tie this to a specific commit using EGIT_COMMIT and make it a 20240504 or similar like the ones found here: https://github.com/organicmaps/organicmaps/releases/

May 07 '24 19:05 hrnick

There are releases but none of them seem to be downloadable for the desktop version.

The releases are linked to a specific git commit and the assets contain a source code tarball besides the APK. I would just use this as "desktop release" (Gentoo needs the source tarball anyway).

May 07 '24 19:05 gerion0

Cannot compile organicmaps, logs ends with:

   repository:               https://github.com/lemire/fast_double_parser.git
   at the commit:            efec03532ef65984786e5e32dbc81f6e6a55a115
 * Skipping submodule 3party/fast_double_parser/benchmark/dependencies/abseil-cpp
 * Skipping submodule 3party/fast_double_parser/benchmark/dependencies/double-conversion
 * Checking out https://github.com/thisistherk/fast_obj to /var/tmp/portage/gui-apps/organicmaps-9999/work/organicmaps-9999/3party/fast_obj ...
git checkout --quiet 1a8060257a96401a9555a476bd13c3a87502c9b5
GIT update -->
   repository:               https://github.com/thisistherk/fast_obj
   at the commit:            1a8060257a96401a9555a476bd13c3a87502c9b5
 * Repository id: organicmaps_world_feed_integration_tests_data.git
 * To override fetched repository properties, use:
 *   EGIT_OVERRIDE_REPO_ORGANICMAPS_WORLD_FEED_INTEGRATION_TESTS_DATA
 *   EGIT_OVERRIDE_BRANCH_ORGANICMAPS_WORLD_FEED_INTEGRATION_TESTS_DATA
 *   EGIT_OVERRIDE_COMMIT_ORGANICMAPS_WORLD_FEED_INTEGRATION_TESTS_DATA
 *   EGIT_OVERRIDE_COMMIT_DATE_ORGANICMAPS_WORLD_FEED_INTEGRATION_TESTS_DATA
 * 
 * Fetching https://github.com/organicmaps/world_feed_integration_tests_data.git ...
git fetch https://github.com/organicmaps/world_feed_integration_tests_data.git +HEAD:refs/git-r3/HEAD
git symbolic-ref refs/git-r3/gui-apps/organicmaps/0/__main__ refs/git-r3/HEAD
 * Checking out https://github.com/organicmaps/world_feed_integration_tests_data.git to /var/tmp/portage/gui-apps/organicmaps-9999/work/world_feed_integration_tests_data-9999 ...
git checkout --quiet refs/git-r3/HEAD
GIT update -->
   repository:               https://github.com/organicmaps/world_feed_integration_tests_data.git
   at the commit:            3b66e59eaae85ebc583ce20baa3bdf27811349c4
>>> Source unpacked in /var/tmp/portage/gui-apps/organicmaps-9999/work
>>> Preparing source in /var/tmp/portage/gui-apps/organicmaps-9999/work/organicmaps-9999 ...
 * Source directory (CMAKE_USE_DIR): "/var/tmp/portage/gui-apps/organicmaps-9999/work/organicmaps-9999"
 * Build directory  (BUILD_DIR):     "/var/tmp/portage/gui-apps/organicmaps-9999/work/organicmaps-9999_build"
 * Applying fix-3party.patch ...                                                                                                                                    [ ok ]
 * Applying fix-jansson.patch ...                                                                                                                                   [ ok ]
 * Hardcoded definition(s) removed in 3party/osrm/osrm-backend/CMakeLists.txt:
 *    set(CMAKE_BUILD_TYPE Release)
 * Hardcoded definition(s) removed in 3party/fast_double_parser/CMakeLists.txt:
 *    set(CMAKE_BUILD_TYPE Release CACHE STRING "Choose the type of build." FORCE
 * Hardcoded definition(s) removed in CMakeLists.txt:
 *    set(CMAKE_BUILD_TYPE "Release")
>>> Source prepared.
>>> Configuring source in /var/tmp/portage/gui-apps/organicmaps-9999/work/organicmaps-9999 ...
 * Source directory (CMAKE_USE_DIR): "/var/tmp/portage/gui-apps/organicmaps-9999/work/organicmaps-9999"
 * Build directory  (BUILD_DIR):     "/var/tmp/portage/gui-apps/organicmaps-9999/work/organicmaps-9999_build"
cmake -C /var/tmp/portage/gui-apps/organicmaps-9999/work/organicmaps-9999_build/gentoo_common_config.cmake -G Ninja -DCMAKE_INSTALL_PREFIX=/usr -DWITH_SYSTEM_PROVIDED_3PARTY=yes -DBUILD_SHARED_LIBS=off -DTEST_DATA_REPO_URL=/var/tmp/portage/gui-apps/organicmaps-9999/work/world_feed_integration_tests_data-9999 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_TOOLCHAIN_FILE=/var/tmp/portage/gui-apps/organicmaps-9999/work/organicmaps-9999_build/gentoo_toolchain.cmake /var/tmp/portage/gui-apps/organicmaps-9999/work/organicmaps-9999
loading initial cache file /var/tmp/portage/gui-apps/organicmaps-9999/work/organicmaps-9999_build/gentoo_common_config.cmake
-- The C compiler identification is GNU 13.2.1
-- The CXX compiler identification is GNU 13.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/x86_64-pc-linux-gnu-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/x86_64-pc-linux-gnu-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using compiler GNU 13.2.1
-- Using Unity Build with batch 50, export UNITY_DISABLE=1 or use -DUNITY_DISABLE=ON to disable it.
-- export COLORS_DISABLE=1 or use -DCOLORS_DISABLE=ON to disable colored compiler output.
-- Build type: RelWithDebInfo
-- Using ld.ldd linker
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - no
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake/Modules/FindThreads.cmake:226 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:252 (find_package)

May 07 '24 21:05 vitaly-zdanevich

-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - no
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake/Modules/FindThreads.cmake:226 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:252 (find_package)

Strange error and should be unrelated to my changes. CMake does not find a compiler with pthread support. Is your gcc compiled with the openmp useflag?

May 08 '24 06:05 gerion0

Is your gcc compiled with the openmp useflag?

Yes.

$ emerge --info gcc
Portage 3.0.61 (python 3.11.9-final-0, default/linux/amd64/17.1/no-multilib, gcc-13, glibc-2.39-r4, 6.6.21-gentoo+ x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-6.6.21-gentoo+-x86_64-AMD_Ryzen_9_7950X_16-Core_Processor-with-glibc2.39
KiB Mem:    65056260 total,  44321032 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Mon, 29 Apr 2024 04:45:00 +0000
Head commit of repository gentoo: 0fe7590f5e2de5c73685fb9348d2853838cdbe7c
Timestamp of repository edgets: Fri, 19 Apr 2024 23:18:39 +0000
Head commit of repository edgets: fec6ca866c8c7414ea51cbc208fcdd58e711ccab

Head commit of repository guru: 149a5a329e4794068cd3f61d3379554bf644eac6

Timestamp of repository pentoo: Sun, 28 Apr 2024 05:19:07 +0000
Head commit of repository pentoo: 176494a8d9d2f2b8a49a93ddb29465c2920eeb8c

sh bash 5.1_p16-r6
ld GNU ld (Gentoo 2.41 p5) 2.41.0
app-misc/pax-utils:        1.3.7::gentoo
app-shells/bash:           5.1_p16-r6::gentoo
dev-build/autoconf:        2.13-r8::gentoo, 2.71-r7::gentoo
dev-build/automake:        1.16.5-r2::gentoo
dev-build/cmake:           3.28.3::gentoo
dev-build/libtool:         2.4.7-r4::gentoo
dev-build/make:            4.4.1-r1::gentoo
dev-build/meson:           1.3.2::gentoo
dev-java/java-config:      2.3.3-r1::gentoo
dev-lang/perl:             5.38.2-r2::gentoo
dev-lang/python:           3.9.19_p2::gentoo, 3.11.9::gentoo, 3.12.3::gentoo
dev-lang/rust:             1.75.0-r1::gentoo
sys-apps/baselayout:       2.14-r2::gentoo
sys-apps/openrc:           0.54::gentoo
sys-apps/sandbox:          2.38::gentoo
sys-devel/binutils:        2.41-r5::gentoo
sys-devel/binutils-config: 5.5::gentoo
sys-devel/clang:           17.0.6::gentoo
sys-devel/gcc:             13.2.1_p20240210::gentoo
sys-devel/gcc-config:      2.11::gentoo
sys-devel/lld:             17.0.6::gentoo
sys-devel/llvm:            17.0.6::gentoo
sys-kernel/linux-headers:  6.6-r1::gentoo (virtual/os-headers)
sys-libs/glibc:            2.39-r4::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: rsync
    sync-uri: rsync://rsync2.ru.gentoo.org/gentoo-portage/
    priority: -1000
    volatile: False
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-jobs: 1
    sync-rsync-extra-opts: 
    sync-rsync-verify-max-age: 24

edgets
    location: /var/db/repos/edgets
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/edgets.git
    masters: gentoo
    volatile: False

guru
    location: /var/db/repos/guru
    sync-type: git
    sync-uri: [email protected]:repo/proj/guru.git
    masters: gentoo
    volatile: False

overlay-of-vitaly-zdanevich
    location: /var/db/repos/overlay-of-vitaly-zdanevich
    masters: gentoo
    volatile: False

pentoo
    location: /var/db/repos/pentoo
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/pentoo.git
    masters: gentoo
    volatile: False

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2                -march=native -pipe -flto"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.9/conf"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php8.2/ext-active/ /etc/php/cgi-php8.2/ext-active/ /etc/php/cli-php8.2/ext-active/ /etc/php/fpm-php8.2/ext-active/ /etc/php/phpdbg-php8.2/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2                -march=native -pipe -flto"
DISTDIR="/var/cache/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2                -march=native -pipe -flto"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2                -march=native -pipe -flto"
GENTOO_MIRRORS="http://mirror.yandex.ru/gentoo-distfiles/ http://gentoo-mirror.alexxy.name/"
LANG="C.UTF8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LEX="flex"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
RUSTFLAGS="-C debuginfo=0 -C target-cpu=native -C opt-level=3 -C panic=abort"
SHELL="/bin/bash"
USE="X acl amd64 avif bluray bzip2 cli crypt dbus dri dvd egl elogind eme-free exif ffmpeg flac fortran gdbm gdk-pixbuf graphite grub heif iconv jpeg libtirpc lto mp3 ncurses nls ogg opengl openh264 openmp opus pam pcre pgo png pulseaudio rdp readline seccomp split-usr ssl svg svt-av1 test-rust tiff tkip udf unicode v4l vaapi vpx vulkan webp x264 x265 xattr xml zlib" ABI_X86="64" ADA_TARGET="gcc_12" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 avx512f avx512dq avx512cd avx512bw avx512vl avx512vbmi f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 ntrip navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LLVM_TARGETS="x64" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-1" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" RUBY_TARGETS="ruby31" VIDEO_CARDS="amdgpu radeonsi" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, MAKEOPTS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, SIZE, STRINGS, STRIP, YACC, YFLAGS

=================================================================
                        Package Settings
=================================================================

sys-devel/gcc-13.2.1_p20240210::gentoo was built with the following:
USE="(cxx) fortran graphite lto nls openmp pgo (pie) sanitize ssp -ada -cet (-custom-cflags) -d -debug -default-stack-clash-protection -default-znow -doc (-fixed-point) -go -hardened (-ieee-long-double) -jit (-libssp) -modula2 (-multilib) -objc -objc++ -objc-gc (-pch) -systemtap -test -valgrind -vanilla -vtv -zstd"
CFLAGS="-march=native -pipe -O2"
CXXFLAGS="-march=native -pipe -O2"
FEATURES="protect-owned ebuild-locks merge-sync assume-digests ipc-sandbox unmerge-logs pid-sandbox config-protect-if-modified unmerge-orphans preserve-libs fixlafiles pkgdir-index-trusted news multilib-strict binpkg-dostrip userpriv network-sandbox strict sfperms userfetch xattr distlocks buildpkg-live usersync binpkg-logs parallel-fetch qa-unresolved-soname-deps sandbox unknown-features-warn binpkg-docompress usersandbox"

May 08 '24 21:05 vitaly-zdanevich

There are releases but none of them seem to be downloadable for the desktop version.

The releases are linked to a specific git commit and the assets contain a source code tarball besides the APK. I would just use this as "desktop release" (Gentoo needs the source tarball anyway).

Using the source tarballs is not easy. I came up with this:

PYTHON_COMPAT=( python3_{7..12} )
inherit git-r3 python-r1 xdg cmake

# this URL is to make the tests compile since organicmaps usually dynamically clones the repo
# maybe a better way would be to skip the test
EGIT_WORLD_FEED_REPO_URI="https://github.com/${PN}/world_feed_integration_tests_data.git"

if [[ ${PV} == 9999 ]]; then
	EGIT_REPO_URI="https://github.com/${PN}/${PN}.git"
	# organicmaps gets more and more system libraries, we use as many
	# as currently possible, use submodules for the rest
	EGIT_SUBMODULES=(
		3party/just_gtfs
		3party/protobuf/protobuf # wait for https://github.com/organicmaps/organicmaps/pull/6310
		3party/fast_obj
	)
else
	# organicmaps itself and the bundled dependencies
	SRC_URI="
		https://github.com/organicmaps/organicmaps/archive/refs/tags/${PV}-9-android.tar.gz -> ${P}.tar.gz
		https://github.com/organicmaps/just_gtfs/archive/7516753825500f90ac2de6f18c256d5abec1ff33.tar.gz -> just_gtfs-75167538.tar.gz
		https://github.com/organicmaps/protobuf/archive/bda4bb90688a36e1311d798030894dcd5f6105ef.tar.gz -> organicmaps-protobuf-bda4bb906.tar.gz
		https://github.com/thisistherk/fast_obj/archive/724e09ff1e82679eeab5ae56652ecc3be593b5b0.tar.gz -> organicmaps-fast_obj-724e09ff1.tar.gz
	"
fi

Explanation:

OrganicMaps dynamically clones (as part of is build process) a Git repo with test data. In the live ebuild this is solved by specifying the repo as additional repo URL and let the build process clone a local repo. The tests are optional, though. We could deactivate it. In the tarball release all of this becomes a problem since git-r3 needs a REPO_URI and cannot be used just to fetch some random Git repo.
OrganicMaps has a few bundled dependencies that usually don't have tags/releases. Luckily, Github provides downloading a tarball of a random commit so this is the workaround I used here.

However, the whole ebuild does not look clean for me, here. OrganicMaps has done a lot in the past to unbundle dependencies. I would just wait some time to get a cleaner ebuild for a specific release.

May 08 '24 21:05 gerion0

Is your gcc compiled with the openmp useflag?

Yes.

Hmm, I don't have a good idea. Can you put this pthread test application into a file (// C program to show thread functions) and try to compile it with gcc yourfile.c? For me, it just works.

May 08 '24 22:05 gerion0

gcc yourfile.c produces no errors.

May 09 '24 10:05 vitaly-zdanevich

But still, I cannot compile this :(

May 14 '24 22:05 vitaly-zdanevich

guru guru copied to clipboard

Organicmaps fixes

guru
guru copied to clipboard