Fix the CI replay action

[jsign]

Currently, our GH Action "replay" is broken so we always get a :red_circle: in master merged PR commits which is a bit unfortunate.

We should fix it (i.e: fix the GH action or consider removing it).