Testing Farm on-demand customized execution

[obabec]

Description

Currently, Packit does not support adding tf_extra_params from the trigger command. This means that users cannot in any way be more specific about test execution.

Workaround using Packit does not exist. You would need to modify .packit.yaml and then change it back or use testing farm CLI directly.

Benefit

The benefit of this feature is definitely noticeable. Users will be able to specify extra environment variables to test environments.

For example, the project I am working on (Debezium - https://github.com/debezium/debezium/blob/main/.packit.yaml) is using Packit to trigger testing farm. We have quite a few jobs with 600+ tests running 1h+. But if I broke just one test with my PR and I want confirmation that I have fixed this issue I could save time and just add environment variable ORACLE_ARG=-Dit=io.debezium.connector.oracle.OracleConnectorIT (in our case) and our test script will run only one class.

That results as the test run is not 100 minutes but 15.

Importance

I don't think that it is a critical issue but it would be a very nice addition to a lot of projects.

Workaround

• [ ] There is an existing workaround that can be used until this feature is implemented.

Participation

• [X] I am willing to submit a pull request for this issue. (Packit team is happy to help!)

[mfocko]

We should probably consider getting argparse involved in the comment parsing, UX might benefit too

[lbarcziova]

This may be a little bit related: https://github.com/packit/packit-service/issues/1754

[lbarcziova]

Thanks for creating this issue, it is a nice idea. We will need to discuss the design of this feature (in relation to the issue linked above and the questions raised there) within the team before implementing it.

[obabec]

Ok, thanks a lot for looking at this issue. Whenever you decide about implementing it let me know, I will be more than happy to help with that :)

[lbarcziova]

We still haven't discussed this at our architecture meeting (mainly the reporting part), but for now, at least some pointers for the implementation:

• parsing of arguments from the /packit test comment is implemented here
• preparing the payload that we send to TF happens here
  • handling of tf_extra_params is here

[lbarcziova]

We have discussed this with the team, here are the notes:

• let's start with reusing the same status name for the test with modified parameters from comment
• we should consider argparse to ease the way of processing the comment arguments
• forwarding the env variables should be quite straightforward and be a good start

[lbarcziova]

@obabec would you be interested in the implementation of this? Could we somehow help?

[obabec]

@lbarcziova Sure I am interested, I will look into that next week hopefully.

[lbarcziova]

@obabec great, let me know if you will need some assistance! :)

[obabec]

@lbarcziova Just FYI, I have finally managed to get some time. I will start with implementation this :)

[lbarcziova]

@Frawless mentioned on Slack that he would like to implement this.