"Invalid message: -40" and infinite loop

[cheesestraws]

When I connect using the realvnc viewer, I get this result. I'm less concerned about it not working and more about the infinite loop :-).

I think what's going on here (though this is only from a brief read-through of your code) is:

• -40 is, I think, "read beyond end of data" (though I may be misremembering here).
• Looking here: https://github.com/marciot/mac-minivnc/blob/main/mac-cpp-source/VNCServer.cpp#L382, this default does not set msgSize, so it stays as it was for the previous message, and breaks out of the switch, but not out of the loop.
• So it's going to keep trying to read from a closed TCP connection a message that'll never come.

[marciot]

@cheesestraws: The invalid message means it is seeing a VNC message ID of -40. This is not defined in https://datatracker.ietf.org/doc/html/rfc6143, so I don't know what may be causing it. You can maybe fix the infinite loop by replacing the break with a return on line 384 but it really should never reach that code anyway. The VNC protocol does not transmit the length with each message, so if an unrecognized message is received, the client can't skip it and resume anyway.

[cheesestraws]

I did take a wireshark trace and couldn't see anything that could be understood as a message of type -40. But don't fret: please feel free to close this issue and now that I've got stdout back working, I'll do my own debugging :-)