docker-zeek

docker-zeek copied to clipboard

This is a very useful zeek container project, I want to apply it to my project. I have a self-built docker network (spark-net), which contains a Kafka container and a Spark container.I want to use zeek to ingest real-time traffic data for some analysis, but I don't know how to make zeek communicate with Kafka in spark-net while being able to listen to external traffic. Use docker network list to view, the DRIVER of the self-built network is bridge, and the SCOPE is local Use ifconfig -a to view the host network card as follows:

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:22ff:fe87:6f8a  prefixlen 64  scopeid 0x20<link>
        ether 02:42:22:87:6f:8a  txqueuelen 0  (Ethernet)
        RX packets 827091  bytes 4974746136 (4.9 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 963491  bytes 1691371345 (1.6 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.66.7  netmask 255.255.255.0  broadcast 192.168.66.255
        inet6 2408:8210:783e:e473:9b4:dee2:734c:5fb4  prefixlen 64  scopeid 0x0<global>

The configuration of local.zeek uses it I changed ["metadata.broker.list"] = "kafka:9092" to ["metadata.broker.list"] = "localhost:9092" Dockerfile uses it

I have tried the following: 1. Use docker-compose to integrate the zeek container into spark-net to start I refer to docker-compose.live.yml, part of the compose is:

kafka:
    container_name: kafka
    image: kafka:latest
    build: kafka/.
    volumes:
     - ./kafka/server.properties_template:/opt/kafka/config/server.properties_template
     - ./kafka/entrypoint.sh:/entrypoint/entrypoint.sh
    depends_on:
      - "zookeeper"
    restart: always
    ports:
      - "9092:9092"
    networks:
      - spark-net
        
zookeeper:
    container_name: zookeeper
    image: zookeeper:latest
    build: zookeeper/.
    depends_on:
      - "logstash"
    restart: always
    ports:
      - "2181:2181"
    networks:
      - spark-net
zeek:
    depends_on:
      - kafka
    build: ./zeek 
    image: blacktop/zeek:elastic
    volumes:
      - ./pcap:/pcap
    cap_add:
      - NET_RAW
    network_mode: "host"
    command: -i af_packet::eno1 local
    
networks:
  spark-net:

But after starting the network, the status of the zeek container is Restarting (0) About a minute ago, apparently it is not working properly

2. Start the zeek container by command without using docker-compose This method is divided into two cases： ①Start the zeek container in host mode (I refer to Capture Live Traffic)

docker run --rm \
        --cap-add=NET_RAW \
        --net host \
        -v `pwd`:/pcap:rw blacktop/zeek:kafka \
        -i af_packet::eno1 local\

I get the following error:

listening on eno1

WARNING: No Site::local_nets have been defined.  It's usually a good idea to define your local networks.
%3|1651201492.014|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Try again (after 5003ms in state CONNECT)
%3|1651201492.014|ERROR|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: 1/1 brokers are down
%3|1651201492.015|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Try again (after 5003ms in state CONNECT)
%3|1651201492.019|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Try again (after 5003ms in state CONNECT)
%3|1651201492.019|ERROR|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: 1/1 brokers are down
%3|1651201492.030|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Try again (after 5006ms in state CONNECT)
%3|1651201492.030|ERROR|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: 1/1 brokers are down
%3|1651201493.015|ERROR|rdkafka#producer-2| [thrd:app]: rdkafka#producer-2: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Try again (after 5003ms in state CONNECT)
%3|1651201493.015|ERROR|rdkafka#producer-3| [thrd:app]: rdkafka#producer-3: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Try again (after 5006ms in state CONNECT)
%3|1651201499.015|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Try again (after 5004ms in state CONNECT, 1 identical error(s) suppressed)
%3|1651201499.016|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Try again (after 5004ms in state CONNECT, 1 identical error(s) suppressed)
%3|1651201499.019|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Try again (after 5003ms in state CONNECT, 1 identical error(s) suppressed)
%3|1651201499.029|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Try again (after 5005ms in state CONNECT, 1 identical error(s) suppressed)
%3|1651201500.016|ERROR|rdkafka#producer-2| [thrd:app]: rdkafka#producer-2: kafka:9092/bootstrap: Failed to resolve 'kafka:9092': Try again (after 5003ms in state CONNECT, 1 identical error(s) suppressed)

It seems that zeek can listen to eno1 but cannot access Kafka inside spark-net

②Start after connecting the zeek container with spark-net via --link

docker run --rm \
        --cap-add=NET_RAW \
        --net elk_spark_elastinet \
        --link elksj-kafka:localhost\
        -v `pwd`:/pcap:rw blacktop/zeek:kafka \
        -i af_packet::eno1 local\

After running I get the error message:fatal error: problem with interface af_packet::eno1 (No such device) It seems that within spark-net, zeek cannot listen to the external network card So I tried listening to docker0 (and docker0's ip), but the result is the same What should I do to achieve real-time monitoring of the host (or any network card) in the container network? do you have any good advice?

Anyone have any idea can let me know, please😔