[Reduce CLI tests Execution time] : Move CLI tests dependencies to API (Entities way) as similar to UI tests

[vijay8451]

Hello @SatelliteQE/quality-engineers

Currently in UI tests we are creating/filling dependencies for tests by calling the APIs entities and then perform the actual execution steps and that helps to save a lot time there.

Another side in CLI to complete the dependency steps we are using helper/factory which are written on CLI(hammer) way and same is taking sufficient time to fulfill it.

Below are some examples(top 8) where they are calling by multiple times on CLI tests and when I tried to call same functionally in API it reduced the half (or more)of execution time:

Note: in CLI they all are calling from robottelo.cli.factory

Outcomes:

1. Hence as per above it would help us to save the time around 129 mins(approx 2 hours) of automation by using the API entities way to fill the dependencies.
2. It would not hamper the current functionality of CLI tests too.

How to achieve it?

1. By writing dependencies helpers under API utils and call under CLI tests.

What do you think on it ? Please share your thoughts and suggestion.

[omkarkhatavkar]

@vijay8451 I think it is a really good point you have added. If we are testing hammer options/command for all entities and avoid duplicate hammer call and replace with API calls reduces the execution time then it is a good thing.

Just to add more proof, the similar case we are doing in Robottelo with UI tests. We are using purely the API calls to create dependent entities and then verifying the UI. I'm not sure API utils be the correct place if not can create a new file to save all factory same as CLI

+1 @vijay8451 raising this.

[jyejare]

Just a suggestion:-

Can we write utils in Nailgun instead of robottelo so that projects like satllite6-upgrade can use them?

[vijay8451]

Just a suggestion:-

Can we write utils in Nailgun instead of robottelo so that projects like satllite6-upgrade can use them?

Great Idea @jyejare but I am not sure if that may create some noise at robottelo end :) ... But let's see what other's think on it as well ...

[JacobCallahan]

I am cautious about this. On one hand, it would be nice to increase performance, on the other we could be cutting out some cli coverage.

If we only use the api for test setups that are already covered in other tests, then I'm good with it. Additionally, api calls should never be used for smoke tests.

[vijay8451]

I am cautious about this. On one hand, it would be nice to increase performance, on the other we could be cutting out some cli coverage.

If we only use the api for test setups that are already covered in other tests, then I'm good with it. Additionally, api calls should never be used for smoke tests.

Thanks for your input , On UI we are already using same approach for setups/dependency fillup also it would not impact any CLI functionality.

[omkarkhatavkar]

I am cautious about this. On one hand, it would be nice to increase performance, on the other we could be cutting out some cli coverage.

I did not understand how CLI coverage will be cutting out? e.g. for in Test_Content_View Cli Test, we testing Content View it will require Organisation, Location and Repository. We have two option either I can create all dependant entities using CLI or API.

If all dependent entities are already covered as part own CLI tests e.g. Test_Orgnsation, Test_Location. then I think we are touching the development code over CLI. (I feel it will not be a coverage issue).

If still anyone feels that way, we can have only one end to end test for each entity should use all CLI entities call, other tests still can use API calls for dependent entities (Just a suggestion, I still feel it is not required)

[JacobCallahan]

@omkarkhatavkar I'm not saying we will lose coverage, I'm simply saying that it is possible if not handled correctly.

Perhaps a standard of what can and cannot be passed over to the API is needed.

[vijay8451]

Hello @SatelliteQE/quality-engineers any more thoughts on it ?

[jyejare]

@omkarkhatavkar @vijay8451 @JacobCallahan ,

Let's cover Tier1 tests completely (setup, assertions) from CLI and for rest Tiers, setup from API and assertions from CLI.

This should assure things are working from CLI plus the integration and system testing can be easily setup by API which will benefit in reducing the time.

Hope that solved all your queries.