secp256k1 icon indicating copy to clipboard operation
secp256k1 copied to clipboard

Published 20 hours ago • verified publisher icon bitcoin-core

tests: allow user to select tests via command line args

Open jonasnick opened this issue 2 years ago • 2 comments

This PR introduces the ability for users to select specific tests to run via command line arguments. The aim is to eliminate the need to comment out parts of the tests.c file in order to speed up testing. The implementation uses similar command line flags as the benchmarks, for example:

./tests ecmult schnorrsig

The approach taken in this PR may not be the most straightforward, but it ensures backwards compatibility. The shell script located at https://gist.github.com/jonasnick/5e37248e3fa5911cd41e1da2f5f4e395 can be used to test the changes introduced in this PR. I'd be happy to add more flags if needed.

jonasnick avatar Feb 09 '23 16:02 jonasnick

Concept ACK

I still think it's an interesting idea for the future to look into existing test frameworks, which typically support running a subset of tests. (But that's no reason not to merge this, of course.)

real-or-random avatar Feb 09 '23 19:02 real-or-random

Concept ACK

sipa avatar Feb 10 '23 19:02 sipa

A bit late to comment here, but I ended up implementing this feature differently in #1734 (built on top of the new unit test framework introduced there).

Originally, I just wanted to parallelize test execution because the suite was taking too long on my end. Since I couldn’t find a good C89-compatible unit test framework without dynamic memory allocations, I built one from scratch; supporting parallelization from the start and adding an easy to expand named-args command-line interface. Then, once the framework was in place, adding new options was easy, so I also added support for running specific targets or groups of targets (along with a few other useful options). That’s how I ended up approaching this problem from another angle. I hope I didn’t step on your toes with that. The main idea behind the new framework is to make the testing infrastructure scalable and development smoother.

furszy avatar Sep 05 '25 19:09 furszy

I hope I didn’t step on your toes with that.

Definitely not. The approaches in both PRs seem to be very different. The goal of this PR is to allow running specific tests with minimal changes to the testing code.

jonasnick avatar Sep 07 '25 15:09 jonasnick

Concept ACK, with the hope that build system changes will be picked up (perhaps in a follow-up).

hebasto avatar Sep 10 '25 13:09 hebasto

Closing because I prefer #1734:

  • parallelism and the resulting speedup
  • much more extensible, while only adding a reasonable number of locs
  • easier to add and change granularity of tests
  • this PR has a bug where the second command line argument is always interpreted as a test target and as the seed.

jonasnick avatar Sep 23 '25 12:09 jonasnick