False positive when running `mxpy contract test`

[ctindogaru]

See https://github.com/ctindogarus4f/elrond-smart-contracts/tree/mxpy-false-positive/staking for reproducing the issue.

The test passes when running mxpy contract test, but fails when running cargo test.

The test should always fail, but for some reason, when running it with mxpy contract test, the test passes, giving us a false positive and making the smart contracts error-prone.

[andreibancioiu]

Hello @ctindogaru, thank you for the report :pray:

Currently, mxpy contract test only runs the JSON-base scenarios (previously called Mandos Tests). On the other hand, cargo test only runs the Rust-based unit tests. Thus, currently, the two commands run different sets of tests.

In your case, a temporary workaround, suggested by @psorinionut, is to write an additional Rust unit test to execute the JSON scenarios, as well:

#[test]
fn init_go() {
    elrond_wasm_debug::mandos_go("mandos/init.scen.json");
}

Of course, we have to also improve the mxpy contract test command, to also run the unit tests.

[ctindogaru]

That's not the problem.

The same scenario is run by both mxpy and cargo, the difference is that with mxpy the test is passing. mxpy doesn't skip the test, mxpy runs the test and reports a false result.

[andreibancioiu]

Hello @ctindogaru,

Currently, mxpy does not run any Rust unit test. Also, cargo test does not run any scenario (ex-mandos). Does it answer the question?

We will adjust mxpy to run unit tests as well.

Thank you :pray:

[ctindogaru]

Hey @andreibancioiu.

I think there is a misunderstanding here.

I'm saying that I have a mandos test that is run for sure by both cargo test and mxpy contract test. With cargo, it fails, with mxpy it doesn't.

I'm 100% confident that the mandos test is run by both cargo and mxpy. If I'm adding a false assertion on purpose inside the test, the test fails on bothcargo test and mxpy contract test. If I remove the assertion, the test only fails on cargo test. Is that a strong enough argument that the test is run by mxpy contract test?

[andreibancioiu]

Thank you, we will check using the provided repository, and return with an answer :pray:

mxpy contract test never runs Rust tests (yet, in the current version):

https://github.com/multiversx/mx-sdk-py-cli/blob/main/multiversx_sdk_cli/projects/project_base.py#L138

[ctindogaru]

I'm not having any rust tests, only JSON-base scenarios.

[andreibancioiu]

We've reproduced the issue with the provided code, thank you!

While cargo test gives an error (actually, a panic):

test staking_rs ... FAILED

failures:

---- staking_rs stdout ----
thread 'staking_rs' panicked at 'bad storage value. Address: sc:staking. Key: 0x7061636b616765496e666f0000000662726f6e7a65. Want: "". Have: 0x01000000000000016d000000000000000a00000000000151800000000203e8000000030591c800000000000d2f000000000000000005', ~/multiversx-sdk/vendor-rust/registry/src/github.com-1ecc6299db9ec823/elrond-wasm-debug-0.29.3/src/mandos_step/check_state.rs:75:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    staking_rs

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

... the tool that executes the JSON scenarios, ~/multiversx-sdk/vmtools/run-scenarios has the following output:

elrond-smart-contracts/staking$ ~/multiversx-sdk/vmtools/run-scenarios "mandos"

Scenario: staking.scen.json ...   ok
Done. Passed: 1. Failed: 0. Skipped: 0.
SUCCESS

Thus, mxpy is not informed by any error here:

https://github.com/multiversx/mx-sdk-py-cli/blob/main/multiversx_sdk_cli/projects/project_base.py#L138

I'll forward the issue to vmtools/run-scenarios. Thank you for reporting it :pray:

[ctindogaru]

No problem.

Is vmtools/run-scenarios open source? If so, can you subscribe me to the issue, please?