netlify-github: Not actually testing current deploy, but last deploy, and is fine if that is of same version
The logic in https://github.com/prisma/e2e-tests/tree/dev/platforms/netlify-ci effectively only works, if we deploy a new version. If we redeploy the same version, the logic is perfectly fine with a previous deployment of the same Prisma version (as a version comparison is the only thing used to confirm the deployment).
Solutions:
- Check actual deployment, not the main URL for when the deployment is put into "production"
- Add something unique to the deployment that is being checked for
Might also apply to other tests that deploy to an environment that could have earlier, working deployments.
Specialized solution for Netlify would be to use https://docs.netlify.com/configure-builds/environment-variables/#git-metadata - that is known to the script pushing to the Github repo. But a bit complicated, as those are not available in Functions env by default but would have to be injected into the function code on build time.
Of course only works for netlify-ci specifically. More generalized solution might be better.