coreos-assembler icon indicating copy to clipboard operation
coreos-assembler copied to clipboard

kola: `--multiply` on a test with subtests doesn't work

Open dustymabe opened this issue 1 year ago • 1 comments

When calling kolet the test name gets passed in mangled. I was able to see this with a small patch:

$ git diff mantle/kola/cluster/cluster.go
diff --git a/mantle/kola/cluster/cluster.go b/mantle/kola/cluster/cluster.go
index 24f3bce03..ce1898628 100644
--- a/mantle/kola/cluster/cluster.go
+++ b/mantle/kola/cluster/cluster.go
@@ -75,6 +75,7 @@ func (t *TestCluster) RunLogged(name string, f func(c TestCluster)) bool {
 // RunNative runs a registered NativeFunc on a remote machine
 func (t *TestCluster) RunNative(funcName string, m platform.Machine) bool {
        command := fmt.Sprintf("./kolet run %q %q", t.H.Name(), funcName)
+       t.Logf("command is %s", command)
        return t.Run(funcName, func(c TestCluster) {
                client, err := m.SSHClient()
                if err != nil {

and then running a simple --multiply=2 basic:

kola -p qemu run --multiply=2 basic --output-dir tmp/kola                                                             
⏭️   Skipping kola test pattern "fcos.internet":           
  👉 https://github.com/coreos/coreos-assembler/pull/1478                                                             
  👉 https://github.com/coreos/coreos-assembler/pull/1478                                                             
⏭️   Skipping kola test pattern "podman.workflow":                                                                    
  👉 https://github.com/coreos/coreos-assembler/pull/1478                                                             
🕒  Snoozing kola test pattern "ext.config.kdump.crash" until Sep 16 2024                                             
  👉 https://github.com/coreos/fedora-coreos-tracker/issues/1791                                                      
=== RUN   basic0                                           
=== RUN   basic1                                           
=== RUN   basic0/ReadOnly                                  
=== RUN   basic0/Useradd                                   
=== RUN   basic0/MachineID                                 
=== RUN   basic0/FCOSGrowpart                              
=== RUN   basic0/PortSSH                                   
=== RUN   basic0/DbusPerms                                 
=== RUN   basic0/ServicesActive                            
--- FAIL: basic0 (27.21s)                                  
        cluster.go:78: command is ./kolet run "basic0" "ReadOnly"                                                     
        cluster.go:96: kolet:                              
Usage:                                                     
  kolet run [test] [func] [flags]                          
  kolet run [command]                                      

Available Commands:                                        
  basic 

Note that kolet should have been called with ./kolet run "basic" "ReadOnly" but the multiplied test name got the appended 0, which appears to be breaking things.

dustymabe avatar Sep 12 '24 21:09 dustymabe

This seems to arise from harness.go:

https://github.com/coreos/coreos-assembler/blob/701df4ab60ed79f48c16e73842166aa0146793b9/mantle/kola/harness.go#L788

The name is the map key, though, as seen in register.go:

https://github.com/coreos/coreos-assembler/blob/701df4ab60ed79f48c16e73842166aa0146793b9/mantle/kola/register/register.go#L133-L142

I can offer two quick first-round options:

  1. Add a field to the Test struct for iteration and use it as a compound key in registering
  2. Add a known unique suffix format to the name and strip it in the test run

nbspsemicolon avatar May 17 '25 03:05 nbspsemicolon