Parallel testing in Azure DevOps Pipelines

[kummerer94]

Hi,

first of all: thank you for your work on this package. It has made testing code in R so much more pleasant.

We upgraded to testthat edition 3 a few days ago to use the parallel test runner. To make this work, we followed the tutorial and locally (Windows) everything works fine.

Once we changed our Azure DevOps pipeline under Linux to use the parallel runner, we ran into the following error:

ℹ Testing xxx
Starting 2 test processes
✔ | F W  S  OK | Context

⠋ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                          
⠙ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                          
⠹ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                          
⠸ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                          
⠼ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                          
⠴ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                          
⠦ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                          
⠧ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                          
⠇ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                          
⠏ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                          
⠋ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                          
⠙ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                          
⠹ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...            

I start my tests with devtools::test().

The Starting up... is the only log output that I see. No tests will be run and the pipeline task will timeout after however many minutes are configured as max time for a task.

It's difficult for me to setup a completely reproducible example as CI/CD pipelines tend to be quite custom, so my question / issue is: Is there a way to enable advanced debugging / logging for testthat? It would help if I could get a sense of what is happening when the new processes start.

I suspect that the process that should start in the background to run the tests fails on startup and so we are in this endless loop of trying to start a new process to run the tests but this process exits immediately.

[hadley]

@gaborcsardi any ideas here?

[gaborcsardi]

Only the obvious ones. Create a simple package, add tests and bells and whistles to it gradually, one by one. Try to reproduce it locally. Debug locally using the usual tools, browser(), debug(), etc. Add logging from your setup and test files, log to files that you can download from the CI.

Also obviously, if you can't reproduce it locally, then the easiest is just to turn off parallel tests on the CI.

[kummerer94]

Thank you for your help! I did use a rather simple package for my tests. Also, I was not able to reproduce this locally which may be due to the specifics of the host in our pipeline.

Whenever I will make another attempt and figure out how to reproduce this, I will let you know about the findings.