stacks-core icon indicating copy to clipboard operation
stacks-core copied to clipboard

Mutants Optimisations: big-packages, timeout multiplier, stacks-signer case

Open ASuciuX opened this issue 1 year ago • 5 comments

Updated the workflow to include the following changes:

  1. split big packages into stackslib and stacks-node
  2. update cargo mutants to use specific version from github in order to include a timeout multiplier feature
  3. add a specific run case for stacks-signer

This PR needs to be merged beforehand: https://github.com/stacks-network/actions/pull/23.

The reason why I'm using cargo nextest before running the stacks-signer mutants is that on this specific package, the run has a bug: the test runs indefinitely, and then the workflow timeouts if cargo nextest is not run before the cargo mutants command.

Debug runs that showcase the issue that appeared when running stacks-signer mutants:

  1. Run working with cargo build and cargo nextest run before cargo mutants: Action run: https://github.com/ASuciuX/stacks-core/actions/runs/7901429286/attempts/2?pr=57 Action file: https://github.com/stacks-network/actions/blob/8a7cb7ed0249b473d241b57228cd718d2bd1e3ef/stacks-core/mutation-testing/pr-differences/action.yml#L[…]9
  2. Run not working without cargo build and cargo nextest run: Action run: https://github.com/ASuciuX/stacks-core/actions/runs/7901429286/attempts/3?pr=57 Action file: https://github.com/stacks-network/actions/blob/801b21e8294c032686dab54b74cb579074141764/stacks-core/mutation-testing/pr-differences/action.yml#L[…]3
  3. Run working with only cargo nextest run before cargo mutants: Action run: https://github.com/ASuciuX/stacks-core/actions/runs/7901429286?pr=57 Action file: https://github.com/stacks-network/actions/blob/b9971c8063623e9f4dee82164652264cce703216/stacks-core/mutation-testing/pr-differences/action.yml#L[…]6

ASuciuX avatar Feb 14 '24 22:02 ASuciuX

3. https://github.com/ASuciuX/stacks-core/actions/runs/7901429286?pr=57

it looks like this build failed on 4: unmutated build failed - should this be expected to pass?

wileyj avatar Feb 19 '24 17:02 wileyj

  1. https://github.com/ASuciuX/stacks-core/actions/runs/7901429286?pr=57

it looks like this build failed on 4: unmutated build failed - should this be expected to pass?

Thanks for pointing that, it shouldn't be expected to fail... The cargo nextest passed it and then it failed on cargo mutants running nextest. I've messaged Martin from cargo mutants regarding this because something is not working as expected.

I'll disable stacks-signer mutants till this is fixed on cargo mutants side as this is the only one having this issue.

ASuciuX avatar Feb 20 '24 14:02 ASuciuX

This is the workflow that was run before. Now it is running as expected, without stacks-signer, even though there are changes on that package.

https://github.com/ASuciuX/stacks-core/actions/runs/8009445597/job/21878164232?pr=59

ASuciuX avatar Feb 22 '24 18:02 ASuciuX

This is the workflow that was run before. Now it is running as expected, without stacks-signer, even though there are changes on that package.

https://github.com/ASuciuX/stacks-core/actions/runs/8009445597/job/21878164232?pr=59

and the output of "unviable mutants" is expected here?

wileyj avatar Feb 22 '24 19:02 wileyj

Yes, there are unviable mutants in this run. If it returns unviable, it means it finished the workflow as the problem was keeping the run till the whole github workflow timed out.

ASuciuX avatar Feb 22 '24 19:02 ASuciuX