join/part to invite-only channels are incorrectly debounced

[progval]

This is probably a duplicate of one of the existing desync issues, but I managed to reproduce it deterministically:

1. join an empty #testchan:libera.chat from Matrix (it's currently broken from matrix.org and some other homeservers because of #14596, though joining from e2e.zone seems to work)
2. make it invite-only by sending !cmd MODE #testchan +i to the appservice (or set it on the IRC side)
3. grant yourself an invite exemption; eg. !cmd MODE #testchan +I testnick!*@* if the nickname you plan to use in the next step is testnick
4. Connect to Libera.Chat using telnet irc.libera.chat 6667 and register by typing this:

NICK valtest
USER v v v v

5. then copy-paste this to the telnet session:

JOIN #testchan
PART #testchan
JOIN #testchan
PART #testchan

(make sure you select the newline at the end so the command is actually sent by telnet.)

On the IRC side, it shows: join, part, join, part.

On the Matrix side, it shows: invite, join, part, invite, join. And then the user remains in the room forever, or until they join again.