Handshake phase support

[okumin]

https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1

[okkez]

I'm implementing handshake phase support https://github.com/okkez/influent/tree/support-fluentd-handshake But it is work in progress. Do you like this? Feel free to comment, please.

[okumin]

I think that NioForwardConnection with handshake support is a finite state machine, so it looks nice that onReadable branches into switch (state). So your plan is good.

In detail,

• HELO is the first message of the protocol and it is sent by a server. So it should be fired in constructor
  • onReadable is invoked when the server is received some packets, but the client never send messages until HELO is received
  • Stores the HELO message in responses(the variable name became bad……) and invoke eventLoop.enableInterestSet(key, SelectionKey.OP_WRITE)
• Messages for handshake may be arrived partially, so it should be parsed incrementally
  • For instance, a ping message may be divided to two packets and onReadable is invoked twice
  • Probably, it is good that msgpack array is parsed by MsgpackStreamUnpacker at first
    • https://github.com/okkez/influent/blob/support-fluentd-handshake/influent-java/src/main/java/influent/forward/NioForwardConnection.java#L202-L212
    • not onReadable -> checkPing but onReadable -> unpacker.feed -> checkPing

[okumin]

[okumin]

There are three conversions.

• Fluentd domain object <-> An array of Msgpack
  • One message of Fluentd matches one array of msgpack
    • So both encoding and decoding are too simple
• An array of Msgpack <-> ByteBuffer
  • Encoding is easy since an array of msgpack has full information needed for binary representation from the beginning
  • Decoding is a bit difficult since TCP stack may send one array of msgpack as multiple packets
    • NioTcpChannel#read may receive only a part of binary of one array of msgpack
    • Explanation in Japanese
• java.util.ByteBuffer <-> socket buffer
  • Write/Read ByteBuffers to/from TCP stack
  • Influent is implemented by non-blocking API of NIO

Array of Msgpack <-> ByteBuffer is already provided by MessageBufferPacker and MsgpackStreamUnpacker. So we can support handshake phase by implementing handshake logic and Fluentd domain object <-> An array of Msgpack conversion.

[okkez]

We need IP address checking like following:

• https://github.com/fluent/fluentd/blob/v0.14.20/lib/fluent/plugin/in_forward.rb#L132-L141
• https://github.com/fluent/fluentd/blob/v0.14.20/lib/fluent/plugin/in_forward.rb#L415

But Java standard libraries don't have such utilities as far as I know. Do you have any idea for remote address checking?

[okumin]

I feel that it is simple to implement such a logic. But it is also OK to use third party library if it is light(not such Netty).

[okkez]

Thank you for comments! I could not find out tiny library. So I've implemented InetNetwork class with reference to Apache James.

[okumin]

I found Apache Commons Net.

https://commons.apache.org/proper/commons-net/apidocs/org/apache/commons/net/util/SubnetUtils.html

How about SubnetInfo#isInRange ?

https://commons.apache.org/proper/commons-net/apidocs/org/apache/commons/net/util/SubnetUtils.SubnetInfo.html#isInRange(java.lang.String)

[okkez]

I think commons-net is large, it is 300KB. https://mvnrepository.com/artifact/commons-net/commons-net/3.6

commons-net does not support IPv6, but I've implemented both IPv4 and IPv6.

[okkez]

Created #32