document examples for running kafka, etc

[heyitsanthony]

There are some procfiles that do this now, but they're not even very good for local development. Possible solution would be /hacks/ full of Dockerfile setups and some documentation around it.

[aland-zhang]

I run latest zetcd and kafka using the zetcd . ./bin/kafka-topics.sh -zookeeper 10.10.244.212:2181 -topic test -replication-factor 2 -partitions 3 --create show info :
[2018-03-16 07:11:43,370] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket) Created topic "test".

./bin/kafka-producer-perf-test.sh --topic test --throughput 1000000 --num-records 1000000 --producer-props bootstrap.servers=kafka-0.kafka:9092,kafka-1.kafka:9092,kafka-2.kafka:9092 acks=all --record-size 200 71690 records sent, 14338.0 records/sec (2.73 MB/sec), 559.5 ms avg latency, 1412.0 max latency. 113050 records sent, 22610.0 records/sec (4.31 MB/sec), 1092.0 ms avg latency, 3396.0 max latency. 109989 records sent, 21997.8 records/sec (4.20 MB/sec), 1714.5 ms avg latency, 4895.0 max latency. 134212 records sent, 26842.4 records/sec (5.12 MB/sec), 1774.9 ms avg latency, 5201.0 max latency. 131359 records sent, 26271.8 records/sec (5.01 MB/sec), 2766.5 ms avg latency, 7801.0 max latency.

restarted one of In the 3 node kafka cluster show errors：

[2018-03-16 07:26:12,326] WARN [ReplicaFetcherThread-1-3]: Error when sending leader epoch request for Map(test-0 -> 0) (kafka.server.ReplicaFetcherThread)

  | java.io.IOException: Connection to kafka-2.kafka.openshift.svc.cluster.local:9092 (id: 3 rack: null) failed.   | at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:68)   | at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:88)   | at kafka.server.ReplicaFetcherThread.fetchEpochsFromLeader(ReplicaFetcherThread.scala:303)   | at kafka.server.AbstractFetcherThread.maybeTruncate(AbstractFetcherThread.scala:128)   | at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:102)   | at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)   | [2018-03-16 07:26:12,327] INFO [ReplicaFetcherThread-1-3]: Retrying leaderEpoch request for partition test-0 as the leader reported an error: UNKNOWN (kafka.server.ReplicaFetcherThread)   | [2018-03-16 07:26:12,618] WARN [ReplicaFetcherThread-0-1]: Error when sending leader epoch request for Map(test-2 -> -1) (kafka.server.ReplicaFetcherThread)   | java.io.IOException: Connection to kafka-0.kafka.openshift.svc.cluster.local:9092 (id: 1 rack: null) failed.   | at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:68)   | at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:88)   | at kafka.server.ReplicaFetcherThread.fetchEpochsFromLeader(ReplicaFetcherThread.scala:303)   | at kafka.server.AbstractFetcherThread.maybeTruncate(AbstractFetcherThread.scala:128)   | at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:102)   | at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)   | [2018-03-16 07:26:12,669] INFO [ReplicaFetcherThread-0-1]: Retrying leaderEpoch request for partition test-2 as the leader reported an error: UNKNOWN (kafka.server.ReplicaFetcherThread)   | [2018-03-16 07:26:13,379] WARN [ReplicaFetcherThread-1-3]: Error when sending leader epoch request for Map(test-0 -> 0) (kafka.server.ReplicaFetcherThread)   | java.io.IOException: Connection to kafka-2.kafka.openshift.svc.cluster.local:9092 (id: 3 rack: null) failed.   | at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:68)   | at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:88)   | at kafka.server.ReplicaFetcherThread.fetchEpochsFromLeader(ReplicaFetcherThread.scala:303)   | at kafka.server.AbstractFetcherThread.maybeTruncate(AbstractFetcherThread.scala:128)   | at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:102)   | at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)   | [2018-03-16 07:26:13,380] INFO [ReplicaFetcherThread-1-3]: Retrying leaderEpoch request for partition test-0 as the leader reported an error: UNKNOWN (kafka.server.ReplicaFetcherThread)   | [2018-03-16 07:26:13,720] WARN [ReplicaFetcherThread-0-1]: Error when sending leader epoch request for Map(test-2 -> -1) (kafka.server.ReplicaFetcherThread)   | java.io.IOException: Connection to kafka-0.kafka.openshift.svc.cluster.local:9092 (id: 1 rack: null) failed.   | at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:68)   | at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:88)   | at kafka.server.ReplicaFetcherThread.fetchEpochsFromLeader(ReplicaFetcherThread.scala:303)   | at kafka.server.AbstractFetcherThread.maybeTruncate(AbstractFetcherThread.scala:128)   | at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:102)   | at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)   | [2018-03-16 07:26:13,720] INFO [ReplicaFetcherThread-0-1]: Retrying leaderEpoch request for partition test-2 as the leader reported an error: UNKNOWN (kafka.server.ReplicaFetcherThread)