giant frame buffer on X10SLM+-LN4F

[Roguelazer]

So, I just got this built on OS X (by changing the makefile to reference stuff from Homebrew; why doesn't libjpeg have pkg-config, blargh) and once I try to connect to our SuperMicro +-LN4F, it attempts to resize the framebuffer to 41472x32930. The subsequent malloc for 5GB of RAM fails.

Connected
Completed handshake
Negotiated security type
Sent request for initial update
messagetype=57
messagetype=0
update[0]: (1024x768)+0+0 len=18176
framebuffer resizing!  640x480  -> 1024x768
framebuffer change: 0x1045a3000[640x480] -> 0x10463a000[1024x768]
messagetype=0
update[0]: (10378x128)+32930+2560 len=671754378
framebuffer resizing!  1024x768  -> 10378x128
update[1]: (41472x32930)+10250+40 len=2656930

Where should I start to investigate? Do you think the protocol's changed?

[thefloweringash]

The other project has a protocol decoder for the communication between the client and server. I used it when reverse engineering the protocol and then used that information to implement the clients. See if it can decode it. Given it seemed to authenticate and start a session, it's possible the changes required are minimal.

[mpalmer]

I'm trying out aten-proxy (great idea, BTW!), and I'm getting the exact same issue. I tried waving the protocol decoder over the pcap, but it is extremely unhappy. I switched the "Unknown {encoding,type,tag}" raises to instead print the errors, and got:

Unknown encoding '87'
Unknown type '4'
Unknown tag '1'
Unknown tag '166'
Unknown tag '138'
Unknown tag '162'
Unknown tag '27'
Unknown tag '15'
Unknown tag '40'
Unknown tag '10'
Unknown encoding '2720530472'
Unknown type '162'
Unknown encoding '[2720530472, 671744040]'

... and then got this exception:

./lens.rb:160:in `[]': no implicit conversion of String into Integer (TypeError)

I added some printf debugging at the top of get_relative, and this is what is causing things to choke:

get_relative("aten-unknown/data-len")
@current is {"id"=>0, "padding"=>"28  | (", "x"=>[32930, 41512], "y"=>[2560, 2560], "width"=>[10378, 35490], "height"=>[128, 128], "encoding-type"=>[2720530472, 671744040], "aten-unknown"=>[{"unknown1"=>"8a a2 80 a2  | ????", "data-len"=>671754378}, {"unknown1"=>"a2 00 80 a2  | ?\x00??", "data-len"=>167782538}], "frame-data"=>{"frame"=>{"type"=>162, "padding"=>"00  | \x00"}}}

I'm coming at this from a position of knowing absolutely nothing about the VNC protocol and such things, but it seems to me like there might be a whole bunch of extra protocol going on in here, so it might not be trivial to handle. However, if you're up for it, I've got a pcap here of a session I can send you, and I can test/hack along with you to get this working (we're in fairly compatible timezones; I'm in Australia). I can probably also give you live access to a KVM instance, if that'd be easier for you.

[thefloweringash]

Hi @mpalmer,

As you've probably noticed, this project isn't actively developed. It's mostly left published on GitHub for the case that someone finds it useful. As far as I'm aware, it still works with X9SCM-iiF.

From a quick google, the X10SLM+-LN4F uses an "ASPEED AST2400 BMC", which is also based on ATEN software. Since the vendor is the same, and security handshake succeeds, the changes required are probably small. From your logs above, the "Unknown type: '4'" message indicates there's a new pixel data format. Neither the decoder nor the proxy considered this case. I've pushed a minor change to the proxy that should at least prevent complete derail of the stream reader in the case of an unknown type. Making the same change to the decoder might let you dump the contents of these updates.

If there's no sensitive data in the pcap files, you could make them generally available in the case that I find some time to work on this.

[mpalmer]

The fix you pushed has definitely improved matters -- thanks for taking the time to poke at it (and to reply). The proxy doesn't freak out about ridiculously huge framebuffers now, and I can connect a VNC client to the proxy. However, all I get is a black window, sadly.

I did end up, after a ridiculous amount of faffing around, getting the JNLP applet thingy to run (with enough SSH forwarded ports, you can do anything!), so the need I had for a functional proxy has somewhat diminished. I'd still like to be able to nuke Java, so I might have a poke around at some point, but the motivation has somewhat diminished. Thankfully I very rarely need to get into the few remaining non-HTML5 KVM boxes...