package e2e tests in rpm

[edsantiago]

Allow other parties to run e2e tests against an rpm-installed podman.

Much trickier than I'd predicted. Split into three commits. Please review those separately for your sanity.

Judgment call: I'm shoehorning these into the existing podman-tests rpm which until now has only had system tests. If there's any objection, or any strong argument for breaking out yet another new subpackage, please speak now.

e2e tests are now included in the podman-tests rpm

[openshift-ci[bot]]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ~~OWNERS~~ [edsantiago]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[packit-as-a-service[bot]]

Ephemeral COPR build failed. @containers/packit-build please check.

[packit-as-a-service[bot]]

Ephemeral COPR build failed. @containers/packit-build please check.

[edsantiago]

Blue-robot failures look real:

+ ./test/tools/build/ginkgo build test/e2e
Failed to compile e2e:

go build github.com/containers/storage/pkg/devicemapper:
# pkg-config --cflags  -- devmapper
Package devmapper was not found in the pkg-config search path.
Perhaps you should add the directory containing `devmapper.pc'
to the PKG_CONFIG_PATH environment variable
Package 'devmapper', required by 'virtual:world', not found
pkg-config: exit status 1
# github.com/mattn/go-sqlite3
../../vendor/github.com/mattn/go-sqlite3/sqlite3.go:85:1: warning: ‘_sqlite3_exec’ defined but not used [-Wunused-function]
   85 | _sqlite3_exec(sqlite3* db, const char* pcmd, long long* rowid, long long* changes)
      | ^~~~~~~~~~~~~

ginkgo build failed

...but it's not something I'm going to look into today.

[Luap99]

Blue-robot failures look real:

+ ./test/tools/build/ginkgo build test/e2e
Failed to compile e2e:

go build github.com/containers/storage/pkg/devicemapper:
# pkg-config --cflags  -- devmapper
Package devmapper was not found in the pkg-config search path.
Perhaps you should add the directory containing `devmapper.pc'
to the PKG_CONFIG_PATH environment variable
Package 'devmapper', required by 'virtual:world', not found
pkg-config: exit status 1
# github.com/mattn/go-sqlite3
../../vendor/github.com/mattn/go-sqlite3/sqlite3.go:85:1: warning: ‘_sqlite3_exec’ defined but not used [-Wunused-function]
   85 | _sqlite3_exec(sqlite3* db, const char* pcmd, long long* rowid, long long* changes)
      | ^~~~~~~~~~~~~

ginkgo build failed

...but it's not something I'm going to look into today.

MAke sure you are using the proper build tags, devmapper should not be used.

[edsantiago]

Why package these? That's the big question. I'm not completely sold on it; I just want to see if it can be done. And if it can, offer it to the FuSa people and see if they find it useful.

I do believe there is value in test packages that track released packages. For instance, two years from now someone is tracking down a failure in 5.2, installs test rpm on latest Fedora, it fails, that could point to a different component (something else got upgraded, maybe systemd or kernel). With an rpm-maintained test suite, it can be easier to bisect the responsible component. I'm a big fan of bundling tests with builds.

What I really hate is bundling a binary without sources. Yuk. That makes it much harder, probably impossible, for a future maintainer to instrument failing tests. I don't see this as a fixable problem, because Go compilers change so frequently and beyond our control.

Thank you for your feedback!

[Luap99]

I do believe there is value in test packages that track released packages. For instance, two years from now someone is tracking down a failure in 5.2, installs test rpm on latest Fedora, it fails, that could point to a different component (something else got upgraded, maybe systemd or kernel). With an rpm-maintained test suite, it can be easier to bisect the responsible component. I'm a big fan of bundling tests with builds.

Fair but I guess this is where my disconnect is, for me as upstream developer I can just as well checkout v5.2 branch/tag and run the suite that way.

[edsantiago]

rpm-build jobs succeeded. I chased the rabbit down to a page with .repo files, set up the rawhide repo on a VM, and ran:

# dnf install podman-tests podman-remote slirp4netns
...
# /usr/share/podman/test/e2e/run-tests &> /var/tmp/e2e-tests.root.01.log
# echo $?
0    <---- yay!

[edsantiago]

Rootless:

# loginctl enable-linger fedora
# su - fedora
$ /usr/share/podman/test/e2e/run-tests &> /var/tmp/e2e-tests.rootless.01.log
$ echo $?
1
...failed the expected three tests, which I choose not to bother with right now:
Summarizing 3 Failures:
  [FAIL] podman system connection sshd and API services required [It] add ssh:// socket path using connection heuristic
  /builddir/build/BUILD/podman-5.1.0-dev/test/e2e/system_connection_test.go:350
  [FAIL] Podman run with --cgroup-parent [It] no --cgroup-parent
  /builddir/build/BUILD/podman-5.1.0-dev/test/e2e/run_cgroup_parent_test.go:45
  [FAIL] Podman systemd [It] podman run container with systemd PID1
  /builddir/build/BUILD/podman-5.1.0-dev/test/e2e/systemd_test.go:115

[ypu]

This is very helpful for user to run e2e related tests without setup a compile env especially for some env that leak of related packages and resources. But if we want to make sure all tests are passed, there still lots of work to do.

[edsantiago]

@containers/podman-maintainers I think this is ready

[EDIT: this is impossible to review in toto. Please be sure to review commit-by-commit]

[edsantiago]

Hearing no demand for this...