ONE icon indicating copy to clipboard operation
ONE copied to clipboard

[CI] lightweight one-cmds test

Open mhs4670go opened this issue 2 years ago • 3 comments

What

Let's introduce lightweight one-cmds test.

Why

Current one-cmds test takes sooo long time that we are not supporting one-cmds test on CI. But, one-cmds related errors happened not rarely. FYI, https://github.com/Samsung/ONE/issues/5514 addresses this issue. But before resolving that issue, we could have another option.

The option is to introduce lightweight one-cmds test. "lightweight" here means that the test can be done in a feasible time with current frequency of posting PR. one-cmds test has lots of duplicate flow in each test.

For example, onecc_001.test and onecc_002.test have the same flow that imports an inception model.

https://github.com/Samsung/ONE/blob/811989cc8f5aeecd7ebad42648d6733401f5e940/compiler/one-cmds/tests/onecc_001.cfg#L10-L16

https://github.com/Samsung/ONE/blob/811989cc8f5aeecd7ebad42648d6733401f5e940/compiler/one-cmds/tests/onecc_002.cfg#L10-L16

In lightweight test, we could just run the basic test that doesn't have duplicate logic.

Related: #9765

mhs4670go avatar Oct 12 '22 02:10 mhs4670go

Better to use a more specific title? For example, "Remove duplicate logics in one-cmds tests"

jinevening avatar Oct 12 '22 03:10 jinevening

@jinevening oh, I didn't mean to revisit current test. This test will be newly introduced for CI purpose.

mhs4670go avatar Oct 12 '22 04:10 mhs4670go

Ah, I get it. You're making a new test set (maybe sampled from existing test cases). Thanks for explanation.

jinevening avatar Oct 12 '22 04:10 jinevening

To run the test lightly, I'll choose a few tests among the package test. Tests will be chosen in the direction of maximizing the coverage.

In our CI, compiler build spends like 44 minutes and package test spends like 21 minutes. If I introduce a new job for the lightweight package test, it forces us to create the package first, which would need other 44 minutes again.

Instead, adding install command into existing release build job will do the trick.

mhs4670go avatar Dec 27 '22 00:12 mhs4670go

Now, current nncc-release job on CI runs following one-cmds tests.

## one-import-tf -> one-optimize -> one-quantize -> one-codegen
./nncc docker-run /bin/bash -c 'cd ${NNCC_INSTALL_PATH}/test;bash onecc_006.test'
## one-import-tflite
./nncc docker-run /bin/bash -c 'cd ${NNCC_INSTALL_PATH}/test;bash onecc_014.test'
## one-import-bcq
./nncc docker-run /bin/bash -c 'cd ${NNCC_INSTALL_PATH}/test;bash onecc_015.test'
## one-import-onnx
./nncc docker-run /bin/bash -c 'cd ${NNCC_INSTALL_PATH}/test;bash onecc_016.test'
## one-pack
./nncc docker-run /bin/bash -c 'cd ${NNCC_INSTALL_PATH}/test;bash onecc_019.test'
## one-profile
./nncc docker-run /bin/bash -c 'cd ${NNCC_INSTALL_PATH}/test;bash onecc_021.test'
## one-quantize
./nncc docker-run /bin/bash -c 'cd ${NNCC_INSTALL_PATH}/test;bash onecc_023.test'
./nncc docker-run /bin/bash -c 'cd ${NNCC_INSTALL_PATH}/test;bash onecc_026.test'
## one-infer
./nncc docker-run /bin/bash -c 'cd ${NNCC_INSTALL_PATH}/test;bash onecc_027.test'
## onecc workflows
./nncc docker-run /bin/bash -c 'cd ${NNCC_INSTALL_PATH}/test;bash onecc_032.test'
./nncc docker-run /bin/bash -c 'cd ${NNCC_INSTALL_PATH}/test;bash onecc_041.test'

If you want to add other tests from one-cmds, feel free to comment them at this issue.

mhs4670go avatar Jan 04 '23 06:01 mhs4670go