[travis-docker] Weird issue with ubuntu-20.10

[kit-ty-kate]

The Travis-docker script fail when DISTRO=ubuntu (which uses ubuntu-20.10)

Step 8/13 : RUN opam depext -ui travis-opam
 ---> Running in dabab12f6aef
# Detecting depexts using vars: arch=x86_64, os=linux, os-distribution=ubuntu, os-family=debian
m4
# The following system packages are needed:
The following command needs to be run through "sudo":
    apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu groovy InRelease
Get:2 http://security.ubuntu.com/ubuntu groovy-security InRelease [108 kB]
Get:3 http://archive.ubuntu.com/ubuntu groovy-updates InRelease [108 kB]
Hit:4 http://archive.ubuntu.com/ubuntu groovy-backports InRelease
Get:5 http://security.ubuntu.com/ubuntu groovy-security/universe amd64 Packages [22.6 kB]
Get:6 http://archive.ubuntu.com/ubuntu groovy-updates/restricted amd64 Packages [45.7 kB]
Get:7 http://security.ubuntu.com/ubuntu groovy-security/restricted amd64 Packages [26.8 kB]
Get:8 http://security.ubuntu.com/ubuntu groovy-security/main amd64 Packages [103 kB]
Get:9 http://archive.ubuntu.com/ubuntu groovy-updates/main amd64 Packages [127 kB]
Get:10 http://archive.ubuntu.com/ubuntu groovy-updates/universe amd64 Packages [32.7 kB]
Fetched 574 kB in 1s (518 kB/s)
Reading package lists...
The following command needs to be run through "sudo":
    apt-get install -qq -yy m4
# OS package update successful
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libsigsegv2:amd64.

(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 14882 files and directories currently installed.)

Preparing to unpack .../libsigsegv2_2.12-2build1_amd64.deb ...

Unpacking libsigsegv2:amd64 (2.12-2build1) ...

Selecting previously unselected package m4.

Preparing to unpack .../archives/m4_1.4.18-4_amd64.deb ...

Unpacking m4 (1.4.18-4) ...

Setting up libsigsegv2:amd64 (2.12-2build1) ...

Setting up m4 (1.4.18-4) ...

Processing triggers for libc-bin (2.32-0ubuntu3) ...

# OS packages installation successful
# Now letting OPAM install the packages
The following actions will be performed:
  - install dune             2.7.1  [required by travis-opam]
  - install ocamlbuild       0.14.0 [required by jsonm]
  - install conf-m4          1      [required by ocamlfind]
  - install opam-file-format 2.1.0  [required by travis-opam]
  - install uchar            0.0.2  [required by jsonm]
  - install ocamlfind        1.8.1  [required by jsonm]
  - install topkg            1.0.3  [required by jsonm]
  - install uutf             1.0.2  [required by jsonm]
  - install jsonm            1.0.1  [required by travis-opam]
  - install travis-opam      1.5.0
===== 10 to install =====

<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[ocamlbuild.0.14.0] downloaded from https://github.com/ocaml/ocamlbuild/archive/0.14.0.tar.gz
[dune.2.7.1] downloaded from https://github.com/ocaml/dune/releases/download/2.7.1/dune-2.7.1.tbz
[jsonm.1.0.1] downloaded from http://erratique.ch/software/jsonm/releases/jsonm-1.0.1.tbz
[opam-file-format.2.1.0] downloaded from https://github.com/ocaml/opam-file-format/archive/2.1.0.tar.gz
[ocamlfind.1.8.1] downloaded from http://download.camlcity.org/download/findlib-1.8.1.tar.gz
[travis-opam.1.5.0] downloaded from https://github.com/ocaml/ocaml-ci-scripts/releases/download/1.5.0/travis-opam-1.5.0.tbz
[uchar.0.0.2] downloaded from https://github.com/ocaml/uchar/releases/download/v0.0.2/uchar-0.0.2.tbz
[topkg.1.0.3] downloaded from http://erratique.ch/software/topkg/releases/topkg-1.0.3.tbz
[uutf.1.0.2] downloaded from http://erratique.ch/software/uutf/releases/uutf-1.0.2.tbz

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
-> installed conf-m4.1
-> installed ocamlfind.1.8.1
-> installed ocamlbuild.0.14.0
-> installed uchar.0.0.2
[ERROR] The compilation of topkg failed at "/home/opam/.opam/4.09/bin/ocaml pkg/pkg.ml build --pkg-name topkg --dev-pkg false".
-> installed dune.2.7.1
-> installed opam-file-format.2.1.0

#=== ERROR while compiling topkg.1.0.3 ========================================#
# context              2.0.7 | linux/x86_64 | ocaml-base-compiler.4.09.1 | file:///home/opam/opam-repository
# path                 ~/.opam/4.09/.opam-switch/build/topkg.1.0.3
# command              ~/.opam/4.09/bin/ocaml pkg/pkg.ml build --pkg-name topkg --dev-pkg false
# exit-code            1
# env-file             ~/.opam/log/topkg-6-52c494.env
# output-file          ~/.opam/log/topkg-6-52c494.out
### output ###
# ocamlfind: Package `threads' not found
# pkg.ml: [WARNING] OCaml host-os conf:  cmd ['ocamlfind' 'ocamlc' '-config']: exited with 2
# pkg.ml: [WARNING] OCaml host-os conf: key native: undefined, stdlib dir not found for discovery
#                   using false
# pkg.ml: [WARNING] OCaml host-os conf: key natdynlink: undefined, stdlib dir not found for discovery
#                   using false
# pkg.ml: [WARNING] OCaml host-os conf: key supports_shared_libraries: undefined, stdlib dir not found for discovery
#                   using false
# pkg.ml: [WARNING] OCaml host-os conf: key ext_obj: undefined, using ".o"
# pkg.ml: [WARNING] OCaml host-os conf: key ext_lib: undefined, using ".a"
# pkg.ml: [WARNING] OCaml host-os conf: key ext_dll: undefined, using ".so"
# pkg.ml: [WARNING] OCaml host-os conf: key ext_exe: undefined and
#                   no C toolchain detected, using ""
# ocamlfind ocamldep -modules src/topkg.mli > src/topkg.mli.depends
# + ocamlfind ocamlc -where > /home/opam/.opam/4.09/.opam-switch/build/topkg.1.0.3/_build/ocamlc.where
# ocamlfind: Package `threads' not found
# Command exited with code 2.
# pkg.ml: [ERROR] cmd ['ocamlbuild' '-use-ocamlfind' '-classic-display' '-j' '4' '-tag' 'debug'
#      '-build-dir' '_build' 'CHANGES.md' 'LICENSE.md' 'README.md' 'pkg/META'
#      'topkg.opam' 'src/topkg.cma' 'src/topkg.cmi' 'src/topkg.mli']: exited with 10

<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
+- The following actions failed
| - build topkg 1.0.3
+- 
+- The following changes have been performed (the rest was aborted)
| - install conf-m4          1
| - install dune             2.7.1
| - install ocamlbuild       0.14.0
| - install ocamlfind        1.8.1
| - install opam-file-format 2.1.0
| - install uchar            0.0.2

Full log: https://travis-ci.org/github/hannesm/albatross/jobs/743374084

I tested it on my test server with the exact same dockerfile and I didn't get any issues. However on Travis the issue is reproducible...

Currently my guess is that the kernel version used on Travis' machines is too old (to my knowledge they use ubuntu-16.04 whereas my test server uses ubuntu 18.04) and some kind of desynching between host kernel and containered distribution is happening. However if anybody has other suggestion I'd take it. cc @avsm @dbuenzli @talex5 ?

[talex5]

The Travis log says Build dist: xenial, but you can use a different one by adding e.g. dist: focal to the .travis.yml.

[hannesm]

Thanks for the issue report, and your discussion. With dist: focal in .travis.yml - as suggested by @talex5 - the compilation of topkg succeeds -- see https://travis-ci.org/github/hannesm/albatross/builds/743769505 (which fails because <sys/sysctl.h> is not present, but this is clearly a different issue).

[dbuenzli]

# pkg.ml: [WARNING] OCaml host-os conf: cmd ['ocamlfind' 'ocamlc' '-config']: exited with 2

I suggest looking at why this happens.

[edwintorok]

It looks like the container has a very high number of parallel jobs set (127), probably based on how many cores were detected at the time the CI docker container got built (and thus different from the number of cores the Travis environment has when it runs the CI container):

podman run -it ocaml/opam:ubuntu-20.10-ocaml-4.09
$ cat .opam/config
opam-version: "2.0"
repositories: "default"
installed-switches: "4.09"
switch: "4.09"
jobs: 127
download-jobs: 3
eval-variables: [
  sys-ocaml-version
  ["ocamlc" "-vnum"]
  "OCaml version present on your system independently of opam, if any"
]
default-compiler: [
  "ocaml-system" {>= "4.02.3"}
  "ocaml-base-compiler"
]

That is not very friendly towards a Travis environment, which has a limited number of cores and memory. Trying to spawn 127 processes may hit some limits. (and the error messages you get from OOM may not be obvious if you don't have access to dmesg, and also opam doesn't print the full build log, they may be more informative messages earlier on in ~/.opam/log/topkg-6-52c494.out that we can't see). Try explicitly setting a limit: ENV OPAMJOBS=4

[Risto-Stevcev]

I'm getting the same issue: https://travis-ci.org/github/Risto-Stevcev/bastet/jobs/759334248

I tried the suggestion to set OPAMJOBS=4 but no success: https://travis-ci.org/github/Risto-Stevcev/bastet/jobs/759386879