Summarize results using GPT

[sravantit25]

This PR is for a providing a new option to our framework - using OPENAI's GPT-4 for summarizing pytest results and providing recommendations to failures. This will be an optional feature. Users of the framework who wish to utilize this framework must provide their own OPENAI API key. The results will be displayed in the form of a simple HTML report present in the log folder.

This works for the following case: When tests are run using pytest

How to use: provide --summary y to have GPT summarize the results.

Note: export OPENAI_API_KEY before using the command

Eg usage: python -m pytest -k example_form --summary y

python -m pytest --summary y

How it works: The input for the GPT will be consolidated log files of all the tests that are run in a session. For this, I have created temporary log files (change in the Base_Logging.py) with a -temp extension.

In conftest.py, used pytest_sessionfinish() fixture to read the data of all the log files and consolidate to a single file. And then delete the individual files. Sometimes, when deleting the files, an error message might come up that the file cannot be deleted as it is in use with some other process. In order to handle this, I have used logger.remove(None) to detach all handlers from the logger in order to release the file handle and then can delete the file

As part of cleanup, I used the pytest_sessionstart() to delete the consolidated log file before new session starts. And also if any -temp log files remain from earlier session, deleting them too.

For performing, the summary using GPT, I created a utility gpt_summary_generator (placed under utils). It reads the consolidated log file and generates a summary. This summary will be presented in the form of a html report. I have placed the logic of defining the html report (i.e how the html should look like) in the same file. Since it is specific to this scenario and cannot be used anywhere else.

The prompt for the GPT is present in conf/gpt_summarization_prompt. I did this since the prompt is large and also to update or make any change it will be easier.

As always is the case with LLMs and AI, the responses provided by the GPT will vary though I have taken measures to keep this as low as possible.

If large number of tests are run, then the log file will be huge and if this is provided to GPT, chances are there it might not work as expected due to limit of number of tokens. That usecase is not handled in this PR.