dashboard: test reproducers on most recent commits before reporting bugs

[a-nogikh]

(As discussed with @melver)

There have been cases (1, 2, 3 during the last couple of days), when syzbot reported bugs that were already fixed by the time of reporting. There's some delay between that time and the actual bug detection time (due to moderation, etc.), so the developers indeed have a chance to fix the bug in the meanwhile.

It makes sense to implement some functionality that would prevent such situations from happening or at least make them rarer. The following signals might be important:

• The reproducer does no longer work on the latest commit from the original branch.
• The crash stopped appearing (might not be relevant for rare bugs).

Also, for bugs found on linux-next, we should also consider running reproducers on git tress of the corresponding subsystems.

[melver]

Looking at the links, and what Dmitry just pointed out [1], it appears that some uses of '#syz invalid' expect the fix to already be present on whatever tree is being tested and therefore not reappear.

So perhaps there is a simpler solution, the easiest of which would be to discourage use of '#syz invalid' and instead '#syz fix' like Dmitry points out. However, for -next that's tricky if e.g. a patch is dropped.

If that's the only problem, perhaps '#syz invalid' on certain trees could have a configurable time in which the bug is simply ignored by syzbot and doesn't reappear. E.g. for -next, it could be the time it usually takes for -next to pick up all changed trees (2-3 days?).

[1] https://lore.kernel.org/all/CACT4Y+a05_HXcUfooYP5Jp2V5QsxB6zoSZKM6g6P3DiVWUvcyg@mail.gmail.com/T/#u

[dvyukov]

If that's the only problem, perhaps '#syz invalid' on certain trees could have a configurable time in which the bug is simply ignored by syzbot and doesn't reappear. E.g. for -next, it could be the time it usually takes for -next to pick up all changed trees (2-3 days?).

I was thinking of something similar: waiting for each syzbot instance build to switch to a new kernel commit before closing bugs as invalid. This should ensure that a potential amended commit/dropped patch had a chance to propagate into tested trees. E.g. if Jens dropped a patch from the block tree by the time of issuing "syz invalid", it should reach the next linux-next rebuild. But this will be a bit harder to implement then just a timeout.