Consider removing or replacing smoke test

[diox]

trafficstars

addons-frontend has a fairly complex smoke test setup. It does the following:

• Installs tox
• clone addons-server
• build addons-server image with custom group & user ids
• build addons-frontend image
• start the entire addons-server stack, updating deps
• trigger initial database & fixtures setup (installing additional python dependencies)
• execute the smoke test suite

All this to execute this single test: https://github.com/mozilla/addons-frontend/blob/2f77f6475835031c52044ccc9e257331f078e611/tests/smoke/test_homepage.py#L6-L9

The whole thing is done for every commit on master, and takes approximately 18 to 30 minutes. Thankfully the frontend image is pushed to dockerhub first, triggering a -dev deploy no matter what, but that seems like an incredibly wasteful and slow process, with very little benefit in the end.

Is it worth it ? Should we keep it ? Change it ?

[willdurand]

This is the reason why we have this smoke test in the first place: https://github.com/mozilla/addons-frontend/issues/4594, the fact that it runs after makes it less useful, though, but we had issues with broken docker images in the past. If -dev deployment succeeds but the app isn't responding, looking at the past circle builds might help (I am not sure).

Maybe we should have a simpler smoke test to validate the docker image? Maybe all we need is running the docker image and see if we can call, e.g., the /__frontend_version__ endpoint?

[diox]

Maybe we should have a simpler smoke test to validate the docker image? Maybe all we need is running the docker image and see if we can call, e.g., the /__frontend_version__ endpoint?

Yeah, that should probably be enough. Testing that the homepage works while only looking for a specific comment in the response isn't that much better anyway.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this bug should stay open, please comment on the issue with further details. Thank you for your contributions.