elrond-sdk-erdpy
elrond-sdk-erdpy copied to clipboard
False positive when running `mxpy contract test`
See https://github.com/ctindogarus4f/elrond-smart-contracts/tree/mxpy-false-positive/staking for reproducing the issue.
The test passes when running mxpy contract test
, but fails when running cargo test
.
The test should always fail, but for some reason, when running it with mxpy contract test
, the test passes, giving us a false positive and making the smart contracts error-prone.
Hello @ctindogaru, thank you for the report :pray:
Currently, mxpy contract test
only runs the JSON-base scenarios (previously called Mandos Tests). On the other hand, cargo test
only runs the Rust-based unit tests. Thus, currently, the two commands run different sets of tests.
In your case, a temporary workaround, suggested by @psorinionut, is to write an additional Rust unit test to execute the JSON scenarios, as well:
#[test]
fn init_go() {
elrond_wasm_debug::mandos_go("mandos/init.scen.json");
}
Of course, we have to also improve the mxpy contract test
command, to also run the unit tests.
That's not the problem.
The same scenario is run by both mxpy
and cargo
, the difference is that with mxpy
the test is passing. mxpy
doesn't skip the test, mxpy
runs the test and reports a false result.
Hello @ctindogaru,
Currently, mxpy
does not run any Rust unit test. Also, cargo test
does not run any scenario (ex-mandos). Does it answer the question?
We will adjust mxpy
to run unit tests as well.
Thank you :pray:
Hey @andreibancioiu.
I think there is a misunderstanding here.
I'm saying that I have a mandos test that is run for sure by both cargo test
and mxpy contract test
. With cargo, it fails, with mxpy it doesn't.
I'm 100% confident that the mandos test is run by both cargo and mxpy. If I'm adding a false assertion on purpose inside the test, the test fails on bothcargo test
and mxpy contract test
. If I remove the assertion, the test only fails on cargo test
. Is that a strong enough argument that the test is run by mxpy contract test
?
Thank you, we will check using the provided repository, and return with an answer :pray:
mxpy contract test
never runs Rust tests (yet, in the current version):
https://github.com/multiversx/mx-sdk-py-cli/blob/main/multiversx_sdk_cli/projects/project_base.py#L138
I'm not having any rust tests, only JSON-base scenarios.
We've reproduced the issue with the provided code, thank you!
While cargo test
gives an error (actually, a panic):
test staking_rs ... FAILED
failures:
---- staking_rs stdout ----
thread 'staking_rs' panicked at 'bad storage value. Address: sc:staking. Key: 0x7061636b616765496e666f0000000662726f6e7a65. Want: "". Have: 0x01000000000000016d000000000000000a00000000000151800000000203e8000000030591c800000000000d2f000000000000000005', ~/multiversx-sdk/vendor-rust/registry/src/github.com-1ecc6299db9ec823/elrond-wasm-debug-0.29.3/src/mandos_step/check_state.rs:75:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
staking_rs
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
... the tool that executes the JSON scenarios, ~/multiversx-sdk/vmtools/run-scenarios
has the following output:
elrond-smart-contracts/staking$ ~/multiversx-sdk/vmtools/run-scenarios "mandos"
Scenario: staking.scen.json ... ok
Done. Passed: 1. Failed: 0. Skipped: 0.
SUCCESS
Thus, mxpy
is not informed by any error here:
https://github.com/multiversx/mx-sdk-py-cli/blob/main/multiversx_sdk_cli/projects/project_base.py#L138
I'll forward the issue to vmtools/run-scenarios
. Thank you for reporting it :pray:
No problem.
Is vmtools/run-scenarios
open source? If so, can you subscribe me to the issue, please?