NettyDecoder OOM risk

[single-wolf]

1. Not length check for byteBuf to be read

Class com.weibo.api.motan.transport.netty4.NettyDecoder

Problem : Not check the length to be read from the peer that will consume a lot of memory by a poisonous message

int metaSize = in.readInt();
size += 4;
if (metaSize > 0) {
    size += metaSize;
    // line 73, we should check metaSize before return
    if (in.readableBytes() < metaSize) {
        in.resetReaderIndex();
        return;
    }

int dataLength = in.readInt();
// line 112, we should check dataLength before return
if (in.readableBytes() < dataLength) {
    in.resetReaderIndex();
    return;
}

2. Not release byteBuf before close channel when meeting length check exception

Class com.weibo.api.motan.transport.netty4.NettyDecoder

Problem : when meeting a failed length check , the channel will be closed by NettyChannelHandler.exceptionCaught() that will invoke ByteToMessageDecoder.channelInactive() ,that cause NettyDecoder.decode() will be invoked again if we not release byteBuf before throw exception

private void checkMaxContext(...) {
    if (maxContentLength > 0 && dataLength > maxContentLength) {
        ....
        // line 127, we should release ByteBuf by invoke ByteBuf.skipBytes(ByteBuf.readableBytes())
        // before throw the Exception
        throw e;
    }
}