Fix tests costing a small fortune

[ntindle]

Duplicates

• [X] I have searched the existing issues

Summary 💡

Add timeouts for tests in python so that we don't burn credits like its free money

Add check to what files have changed and only run tests if code has changed

Add timeouts for testing step in gh actions so even if we burning money, its limited and semi predictible

Examples 🌈

No response

Motivation 🔦

No response

[AnuragBalhra]

I can pick it up, if no one else is working on it yet.

[ntindle]

@merwanehamadi is working on part 1 I believe

[ntindle]

@AnuragBalhra csn you do part two and three?

[AnuragBalhra]

sure, I'll pick up part two and three.

[ashuk2000]

hey @AnuragBalhra I am kinda working on 2

[Pwuts]

~~Important note here: since we run our CI on pull_request events, the workflow only has access to our OPENAI_API_KEY if the source branch is in the main repo, meaning only PRs from maintainers will cost "us" money.~~

~~Outside contributors will have to set their own API key on their fork repo.~~

@merwanehamadi implemented the smoke test using a cassette that stores HTTP requests to the OpenAI API and uses those instead of actually making a request. This is probably a good solution for our current problem.

[ashuk2000]

@Pwuts do we need this , "Add check to what files have changed and only run tests if code has changed" if we have the casesste to fake the api calls

[Pwuts]

Fixed in #3359

[Pwuts]

@ashuk2000 not running CI when it doesn't add any value is generally a good idea. It isn't strictly necessary to be selective, but it can speed up some workflows and save runner minutes and with that also electrical energy.