`PeerMap` invariants don't hold

[Frando]

This debug_assert in Endpoint::receive_ip triggers reliably for me on tests with many nodes:

 // the peer map maintains consistency
        let Some(state) = endpoint.direct_addr_state.get_mut(&ip_port) else {
            debug_assert!(false, "peer map inconsistency by_ip_port <-> direct addr");
            return None;
        };

https://github.com/n0-computer/iroh/blob/main/iroh-net/src/magicsock/endpoint.rs#L1417