[Feature Request] `make check` out of racket-N.M-src or similar should run automatic tests

[riastradh]

Is your feature request related to a problem? Please describe. When I have just built Racket from source, I would like to run some reasonably fast but extensive tests to verify that the build works. Nothing that will take hours of exhaustive testing or randomized property testing, but ideally it would be a little more extensive than just launching the REPL and evaluating (+ 1 2).

Describe the solution you'd like Either:

1. make check just runs the tests
2. another tarball distributed alongside racket-N.M-src (or racket-minimal-N.M-src), pegged to the same version, can be extracted so that there's a one-liner that triggers a reasonable selection of automatic tests running racket out of the build directory (or perhaps a destdir)

Describe alternatives you've considered I started reading src/README.txt which mentioned a test suite but it wasn't obvious where to find it from src/README.txt, and after struggling with build issues my attention span is too short to go further at the moment. (If I missed something that already does this, please feel free to summarily close this!)

Do you want to contribute to this feature Probably not, sorry!

[samth]

Here's a two-liner:

raco pkg install racket-test-core
raco test -l tests/racket/test

[riastradh]

Are the files already included in racket-N.M-src, or does this talk to the network? Having the files distributed in advance is a hard requirement.

[samth]

That talks to the network. If you don't want to do that, you could try the following:

wget https://download.racket-lang.org/releases/8.10/pkgs/racket-test-core.zip # this talks to the network, of course
raco pkg install -t file racket-test-core.zip
raco test -l tests/racket/test

[riastradh]

Great, that's better! Of course we need to talk to the network at some point; it's just that it has to be finished before any of the build starts.

Can it run out of the build directory, or does it work only the final installation location?

[samth]

Those commands are all totally independent of CWD.

[riastradh]

What I'm getting at is: does it require that the installation prefix (which might be somewhere privileged like /usr/pkg) be readable and writable, or can it be used strictly out of the build tree (which might be somewhere unprivileged like /home/pkgbuild/work) before the installation prefix is populated and without writing to it?

[samth]

You can't run that (easily) before running make install. You should be able to run

raco pkg install --user -t ...

to install it to a user-local directory if the installation prefix is not writable.

[LiberalArtist]

One thing we could consider is, for future releases, adding an additional source tarball to https://download.racket-lang.org/releases/8.10/ along the lines the lines of the "Racket plus Tests" builds that the Northwestern snapshots include.

At least one test in racket-test-core does currently try to write the installation-wide collections directory. I'm planning to fix it soon-ish to skip that test if the current user doesn't have write permissions. I've also been working very slowly at running tests without racket-test-core. It's still very experimental, but I've made some progress since https://github.com/racket/racket/issues/4458: for the packages listed in https://gitlab.com/philip1/guix-racket-experiment/-/blob/e2a96fb/wip-gnu/packages/racket.scm that don't have #:tests? #f, the build system basically runs raco test -j $PARALLEL_JOB_COUNT --heartbeat --package $PACKAGE_NAME successfully.

[riastradh]

That talks to the network. If you don't want to do that, you could try the following:

wget https://download.racket-lang.org/releases/8.10/pkgs/racket-test-core.zip # this talks to the network, of course
raco pkg install -t file racket-test-core.zip
raco test -l tests/racket/test

$ curl -fsSL -O https://download.racket-lang.org/releases/8.10/pkgs/racket-test-core.zip
$ raco pkg install --user -t file racket-test-core.zip
The following uninstalled packages are listed as dependencies of racket-test-core: 
   zo-lib
   at-exp-lib
   serialize-cstruct-lib
   dynext-lib
   sandbox-lib
Would you like to install these dependencies? [Y/n/a/c/?] 

This makes it sound like it still needs network access? (Haven't yet tested in a contained environment without network access, though.)

[riastradh]

I answered y to the question (and followups about transitive dependencies), and it seemed to download things from various URLs.

Then:

$ raco test -l tests/racket/test
/home/riastradh/pkgsrc/current/pkg/bin/raco: Unrecognized command: test

Usage: raco <command> <option> ... <arg> ...

Frequently used commands:
  setup   install and build libraries and documentation
  pkg     manage packages

A command can be specified by an unambiguous prefix.
See `raco help' for a complete list of commands.
See `raco help <command>' for help on a command.

[samth]

I had thought you were installing the full Racket distribution, if not you will need those packages too. They're available as similar zip files from the same place.

The raco test command is in the compiler-lib package.

[riastradh]

Interested in the answer for both racket-N.M-src and racket-minimal-N.M-src. Sorry, should've said that up front!

[samth]

For the full distribution you should only need racket-test-core (that's the point of the package). For the minimal distribution then you will need the listed dependencies which should be the ones you saw.