Simple use case to connect docker compose container to remote vpn kafka cluster over socks5a proxy via ssh

[warrenc5]

Hi,

Firstly thanks for the proxy app - it's great, however it's use in a standard use case scenario was non-obvious from the documentation examples (using localhost as a be-all-end-all) is confusing for new comers. I couldn't distinguish what was the reference point and what was the direction.

I'm familiar with docker networking and things like host.docker.internal (which didn't work), localhost in a container with networking is not the same as localhost on the host and bridge and host etc.

I'm familiar with kafka in that a broker can advertises the connection address of a node and also introduce other nodes in the cluster.

I tried every possible combination and permutation, until I could proxy through over socks to one of the nodes, however I kept on getting broker errors in my container logs about node2 and node3.

The entire thing wouldn't work until I added the --default-listener-ip. I had tried to "--disable-dynamic -listeners" but the consumer also complained about the bootstrap.

Note, I'm using compose as part of a larger project.

Scenario

Firstly set up the socks5a proxy. (I couldn't work out how to use the "tool" option of the kafka-proxy and I already have a socks5a proxy here. And eventually worked out I had to bind the socks5 proxy onto the docker interface.

ssh -N -v -D 172.17.0.1:1080 remote_vpn_host

kafka:
    image: grepplabs/kafka-proxy:0.3.6 
    command: 
        - server
        - --bootstrap-server-mapping 
        - "shared-kafka-01.mylab.xxx:9092,172.24.0.9:9092" 
        - "shared-kafka-02.mylab.xxx:9092,172.24.0.9:29092" 
        - "shared-kafka-03.mylab.xxx:9092,172.24.0.9:39092" 
        - --default-listener-ip 
        - 172.24.0.9
        - --forward-proxy
        - socks5://172.17.0.1:1080
        - --debug-enable
    networks:
      lnet1:
        ipv4_address: 172.24.0.9

netstat -latnp | grep 1080

 will not be shown, you would have to be root to see it all.)
tcp        0      0 172.17.0.1:1080         0.0.0.0:*               LISTEN      1439150/ssh

In the end the performance was poor. I could see a new connection opened for every message. I thought it would reuse reuse connections. I looked at the idle, reuse switches but couldn't see anything obvious. Perhaps this is what "tool" does.

I hope this example shows a config that worked for me and saves a user several hours tinkering around.

[richard-noble-cko]

Hi Warren

It's not documented (that I could find), but there is a third possible options to the bootstrap server mappings. try "shared-kafka-01.mylab.xxx:9092,172.24.0.9:9092,kafka:9092" instead. That should allow the mapping of the service to advertise itself.