CI leaks changes from master into other branches, breaking tests when they work locally

[mattiasgrenfeldt]

The CI is broken in a subtle way. Imagine the following timeline:

1. Create new branch 'A' based on 'master'.
2. Work on branch 'A'.
3. Submit PR from branch 'A', CI passes. (We have not merged into 'master'.)
4. Start work second branch 'B' based on 'master'.
5. Submit PR from branch 'B'.
6. Get PR from branch 'B' merged into 'master'.
// Now this shouldn't have affected branch/PR 'A', right?
7. Rerun CI for branch/PR 'A'. For example by submitting changes to code comments.
8. Now CI fails, complaining about things that could only have happened if the changes from 'B' were mixed in to this branch somehow.
// Apparently it did affect branch/PR 'A' :/

This exact timeline has happened twice.

The second time this happened was between #104 and #106. The CI logs can be seen here. It complains about that safehttptest.Recorder.Status has changed it's signature to return a safehttp.StatusCode instead of an int. This change was performed in #106 which is now merged to master. But #104 doesn't have these changes as can be seen here.

I suspect that this is caused by actions/checkout@v2 somehow. The action seems to not get a "fresh" copy of the branch each time but somehow mixes in old stuff? When you look at the logs of the action you see this:

> Run actions/checkout@v2
Syncing repository: google/go-safeweb
> Getting Git version info
Deleting the contents of '/home/runner/work/go-safeweb/go-safeweb'
> Initializing the repository
> Disabling automatic garbage collection
> Setting up auth
> Fetching the repository
> Determining the checkout info
> Checking out the ref
/usr/bin/git log -1
commit 4885fd6bb8b9c0aef9b3ae694217199b1f5cb82d
Author: mattiasgrenfeldt <[email protected]>
Date:   Fri Aug 28 08:23:52 2020 +0000

    Merge f29468b9756aed14f105369831ba4b9d1c1ffcee into ad27fca20cf199974d66ad9e97ebb486c8c8cb95

For some reason it is merging something? Why?

This needs more investigation to be solved.