Break down ockam command bats tests

[mrinalwadhwa]

Current

Currently we have some Ockam command bats tests here

https://github.com/build-trust/ockam/blob/develop/implementations/rust/ockam/ockam_command/tests/commands.bats

These tests are run in CI here https://github.com/build-trust/ockam/blob/develop/.github/workflows/ockam_command.yml#L45-L77

Desired

We want to break these down into many small bats test files, and have a way to run them as a group. It would be good to have one bats file per command so we can tests various commands more ehaustively. https://github.com/build-trust/ockam/blob/develop/implementations/rust/ockam/ockam_command/src/lib.rs#L284-L305

We could keep the bigger integration tests separate from tests that are testing a specific command.

---

We love helping new contributors! If you have questions or need help as you work on your first Ockam contribution, please leave a comment on this discussion. If you're looking for other issues to contribute to, checkout this discussion and labels - https://github.com/build-trust/ockam/labels/good%20first%20issue or https://github.com/build-trust/ockam/labels/help%20wanted

[thaodt]

Hi, is there anyone working on this? Otherwise, I will give it a try.

[mrinalwadhwa]

No, all yours. Let us know if you have any questions

[thaodt]

No, all yours. Let us know if you have any questions

I think I got it. But lemme clarify my understanding before implementation. :smile: Basically we want to separate tests in 1 bats file into smaller bats files and can reuse some of them to run as a group test. Am i right?

Also, I noticed both setup() and teardown() fn called $OCKAM node delete --all || true, is this duplication necessary?

[mrinalwadhwa]

Am i right?

yes exactly.

Also, I noticed both setup() and teardown() fn called $OCKAM node delete --all || true, is this duplication necessary?

yes, currently we don't always run the tests in a new environment so it does help.

[mrinalwadhwa]

@thaodt checking in to see if you still have time to work on this or if you need any help. Totally okay if you don't have time we can open it up for other contributors. thank you.

[thaodt]

@thaodt checking in to see if you still have time to work on this or if you need any help. Totally okay if you don't have time we can open it up for other contributors. thank you.

hi @mrinalwadhwa yeah still want to work on this. Sorry im just back from holiday, I'm checking my list of issues one by one :sweat_smile:

[mrinalwadhwa]

Great, let us know if you need help.

[thaodt]

@mrinalwadhwa it seems some new tests didn't pass after some weeks i didn't touch to ockam code. maybe im missing something, could you please point it out? The errors i got below:

Error details

✗ create an inlet (with implicit secure channel creation) / outlet pair with relay through a forwarder in an orchestrator project and move tcp traffic through it
   (in test file implementations/rust/ockam/ockam_command/tests/commands.bats, line 327)
     `$OCKAM tcp-inlet create --at /node/green --from 127.0.0.1:7000 --to /project/default/service/forward_to_blue/secure/api/service/outlet' failed with status 2
   
   Node:
     Name: blue
     Status: UP
     Services:
       Service:
         Type: TCP Listener
         Address: /ip4/127.0.0.1/tcp/46855
       Service:
         Type: Secure Channel Listener
         Address: /service/api
         Route: /ip4/127.0.0.1/tcp/46855/service/api
         Identity: Pc31c3e79f3835b702c80dc03e0fbf9e77e07bb7ea7c1a62c6797a0ac5e40ab29
         Authorized Identities:
           - Pc31c3e79f3835b702c80dc03e0fbf9e77e07bb7ea7c1a62c6797a0ac5e40ab29
       Service:
         Type: Uppercase
         Address: /service/uppercase
       Service:
         Type: Echo
         Address: /service/echo
     Secure Channel Listener Address: /service/api
   
   /service/outlet
   /service/forward_to_blue
   
   Node:
     Name: green
     Status: UP
     Services:
       Service:
         Type: TCP Listener
         Address: /ip4/127.0.0.1/tcp/45065
       Service:
         Type: Secure Channel Listener
         Address: /service/api
         Route: /ip4/127.0.0.1/tcp/45065/service/api
         Identity: Pc31c3e79f3835b702c80dc03e0fbf9e77e07bb7ea7c1a62c6797a0ac5e40ab29
         Authorized Identities:
           - Pc31c3e79f3835b702c80dc03e0fbf9e77e07bb7ea7c1a62c6797a0ac5e40ab29
       Service:
         Type: Uppercase
         Address: /service/uppercase
       Service:
         Type: Echo
         Address: /service/echo
     Secure Channel Listener Address: /service/api
   
   error: Invalid value "/project/default/service/forward_to_blue/secure/api/service/outlet" for '--to <ROUTE>': unregistered protocol prefix "secure"
   
   For more information try --help
 ✗ inlet/outlet example with credentials, not provided
   (from function `assert_success' in file /home/thaodt/projects/bats-setup/test/test_helper/bats-assert/src/assert.bash, line 114,
    in test file implementations/rust/ockam/ockam_command/tests/commands.bats, line 340)
     `assert_success' failed
   
   -- command failed --
   status : 2
   output (8 lines):
     error: Found argument '--no-shared-identity' which wasn't expected, or isn't valid in this context
   
        If you tried to supply `--no-shared-identity` as a value rather than a flag, use `-- --no-shared-identity`
   
     USAGE:
         ockam node create <NODE_NAME>
   
     For more information try --help
   --
   
 ✗ inlet (with implicit secure channel creation) / outlet example with credentials, not provided
   (from function `assert_success' in file /home/thaodt/projects/bats-setup/test/test_helper/bats-assert/src/assert.bash, line 114,
    in test file implementations/rust/ockam/ockam_command/tests/commands.bats, line 374)
     `assert_success' failed
   
   -- command failed --
   status : 2
   output (8 lines):
     error: Found argument '--no-shared-identity' which wasn't expected, or isn't valid in this context
   
        If you tried to supply `--no-shared-identity` as a value rather than a flag, use `-- --no-shared-identity`
   
     USAGE:
         ockam node create <NODE_NAME>
   
     For more information try --help
   --
   
 ✗ inlet/outlet example with credentials
   (from function `assert_success' in file /home/thaodt/projects/bats-setup/test/test_helper/bats-assert/src/assert.bash, line 114,
    in test file implementations/rust/ockam/ockam_command/tests/commands.bats, line 406)
     `assert_success' failed
   
   -- command failed --
   status : 2
   output (8 lines):
     error: Found argument '--enable-credential-checks' which wasn't expected, or isn't valid in this context
   
        If you tried to supply `--enable-credential-checks` as a value rather than a flag, use `-- --enable-credential-checks`
   
     USAGE:
         ockam node create <NODE_NAME>
   
     For more information try --help
   --
   
 ✗ inlet (with implicit secure channel creation) / outlet example with credentials
   (from function `assert_success' in file /home/thaodt/projects/bats-setup/test/test_helper/bats-assert/src/assert.bash, line 114,
    in test file implementations/rust/ockam/ockam_command/tests/commands.bats, line 438)
     `assert_success' failed
   
   -- command failed --
   status : 2
   output (8 lines):
     error: Found argument '--enable-credential-checks' which wasn't expected, or isn't valid in this context
   
        If you tried to supply `--enable-credential-checks` as a value rather than a flag, use `-- --enable-credential-checks`
   
     USAGE:
         ockam node create <NODE_NAME>
   
     For more information try --help
   --
   
 ✗ project requiring credentials
   (from function `assert_success' in file /home/thaodt/projects/bats-setup/test/test_helper/bats-assert/src/assert.bash, line 114,
    in test file implementations/rust/ockam/ockam_command/tests/commands.bats, line 479)
     `assert_success' failed
   
   -- command failed --
   status : 2
   output (8 lines):
     error: Found argument '--enable-credential-checks' which wasn't expected, or isn't valid in this context
   
        If you tried to supply `--enable-credential-checks` as a value rather than a flag, use `-- --enable-credential-checks`
   
     USAGE:
         ockam node create <NODE_NAME>
   
     For more information try --help
   --
   

26 tests, 7 failures

Command I used: ORCHESTRATOR_TESTS=1 LONG_TESTS=1 bats implementations/rust/ockam/ockam_command/tests/commands.bats

[twittner]

@thaodt: It looks like the tests are running against an outdated ockam binary. Try compiling the current develop version, e.g.

> cargo build && cp target/debug/ockam /tmp
> env OCKAM=/tmp/ockam ORCHESTRATOR_TESTS=1 LONG_TESTS=1 bats implementations/rust/ockam/ockam_command/tests/commands.bats

[thaodt]

@thaodt: It looks like the tests are running against an outdated ockam binary. Try compiling the current develop version, e.g.

> cargo build && cp target/debug/ockam /tmp
> env OCKAM=/tmp/ockam ORCHESTRATOR_TESTS=1 LONG_TESTS=1 bats implementations/rust/ockam/ockam_command/tests/commands.bats

@twittner yeah, I already built before asking, its weird a bit, just clean and rebuilt, then it worked. But right now, there are 2 failed tests.

Details

 ✗ create space, create project, send message, delete project, delete space
   (from function `assert_success' in file /home/thaodt/projects/bats-setup/test/test_helper/bats-assert/src/assert.bash, line 114,
    in test file implementations/rust/ockam/ockam_command/tests/commands.bats, line 283)
     `assert_success' failed
   
   -- command failed --
   status : 70
   output : An error occurred while processing the request. Status code: 400 BadRequest. Message: limit reached
   --
   
 ✗ project requiring credentials
   (from function `assert_success' in file /home/thaodt/projects/bats-setup/test/test_helper/bats-assert/src/assert.bash, line 114,
    in test file implementations/rust/ockam/ockam_command/tests/commands.bats, line 474)
     `assert_success' failed
   
   -- command failed --
   status : 70
   output : An error occurred while processing the request. Status code: 400 BadRequest. Message: limit reached
   --
   

26 tests, 2 failures

I checked the code but nothing seems out of the ordinary.

[adrianbenavides]

Done