kopf's port of elasticsearch 0.1 is not open

[mfilotto]

Hi there,

I try to replay your demo : http://rancher.com/container-logging-with-docker-and-elk-september-2015-recorded-online-meetup/ So I'm using the elasticsearch 0.1.0 docker-compose file : https://github.com/rancher/compose-templates/blob/master/elasticsearch/0.1.0/docker-compose.yml

But the port 80 of kopf is not open outside the container. Any clue ? Where should I start to investigate that problem ?

Thanks for your help

I like rancher by the way ;)

[mfilotto]

Here is an insight on my env, the stack state is stuck on activating, I don't know why

[cloudnautique]

Hmm.. that seems odd. What version of rancher server are you using? You can find out by clicking the cow in the upper left hand corner. Which version of rancher-compose (-v)? Is this a multinode setup? Datanodes can not be on the same host in this version because they use host volume mounts.

Other troubleshooting:

• You can view the logs of the containers in the ui(container hamburger menu) or on the command line(docker logs ). Check the Kopf container to see if Nginx had any issues starting up. If the service is flapping quickly, sometimes it is not apparent in the UI, but would prevent it from going active. It looks like, based on the docker ps output it isn't flapping though.
  • On the host with Kopf, you can see if the iptables rules are in place: sudo iptables -L -t nat you should see something forwarding port 80->80.
  • In the UI under the admin panel there is a process list for Rancher, there might be some long running processes in there that are hung up.

On a side note, this template should run perfectly fine, but we have been iterating on this a lot and have gotten upto version 0.3.1 of the stack.

Let us know if anything turns up.

[mfilotto]

Thanks for your help, I appreciate

Rancher server is v0.43.1 Rancher compose is v0.5.0

I work on a multi nodes. I have rancher server on node 1, and I added 2 custom hosts (CentOS 7)

What you call datanotes are the sidekicks ?

Strange there is no logs for kopf.

I should probably add a third host. I tried to register the host of the rancher server but with no success.

I tried the last version first, but it is not working, the sidekicks were not able to start

I found rancher-compose-executor errors in the processes :

io.cattle.platform.async.utils.TimeoutException
    at io.cattle.platform.async.retry.impl.RetryTimeoutServiceImpl$1.run(RetryTimeoutServiceImpl.java:51)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Here is my iptables

[cloudnautique]

The v0.3.1 elasticsearch template has sidekicks that start then immediately stop. They are the datavolume containers and do not need to be running. Is this what you saw?

It doesn't look like the ports have been exposed.

On the host where kopf is running, is there anything in the rancher-agent logs? Sometimes theres a message that repeats.

Adding the server node, requires you to set -e "CATTLE_AGENT_IP=<hostip>" on the agent run command.

[mfilotto]

Yes it's what I saw,, so it's the normal behavior

Yes the port is not expose

Here are the agent logs of koft's host.

time="2015-11-05T03:49:45Z" level="info" msg="Processing event: &docker.APIEvents{Status:\"start\", ID:\"82ae4fe9bb701dbb5daee29286e46b85701213f42f3a70abf40746187ff743c6\", From:\"rancher/kopf:v0.4.0\", Time:1446695385}"
time="2015-11-05T03:49:45Z" level="info" msg="Assigning IP [10.42.45.118/16], ContainerId [82ae4fe9bb701dbb5daee29286e46b85701213f42f3a70abf40746187ff743c6], Pid [25068]"
time="2015-11-05T03:49:45Z" level="info" msg="Processing event: &docker.APIEvents{Status:\"start\", ID:\"82ae4fe9bb701dbb5daee29286e46b85701213f42f3a70abf40746187ff743c6\", From:\"-simulated-\", Time:0}"
time="2015-11-05T03:49:45Z" level="info" msg="Container locked. Can't run StartHandler. ID: [82ae4fe9bb701dbb5daee29286e46b85701213f42f3a70abf40746187ff743c6]"

I have this message : Container locked. Can't run StartHandler

Then I have no route to host messages inside the rancher server. The VMs are behind a proxy, maybe I missed a config at some point I added the proxy to docker this way : https://docs.docker.com/articles/systemd/#http-proxy It's like my host has access but not the rancher server container

fatal: unable to access 'https://github.com/rancher/rancher-catalog.git/': Failed to connect to github.com port 443: No route to host
time="2015-11-05T03:56:01Z" level=fatal msg="Failed to clone the catalog from githubexit status 128"
time="2015-11-05T03:56:01Z" level=info msg="Setting log level" logLevel=info
time="2015-11-05T03:56:01Z" level=info msg="Starting go-machine-service..." gitcommit=3c74782
time="2015-11-05T03:56:02Z" level=error msg="Unable to create EventRouter" Err="Get http://192.168.90.111:8080/v1: dial tcp 192.168.90.111:8080: getsockopt: no route to host"
time="2015-11-05T03:56:02Z" level=info msg="Exiting go-machine-service..."
time="2015-11-05T03:56:03Z" level=info msg="Starting Rancher Catalog service"
time="2015-11-05T03:56:03Z" level=info msg="Cloning the catalog from github url https://github.com/rancher/rancher-catalog.git"

For the server node, I triied with the var env CATTLE_AGENT_IP but I'm stuck on message ERROR: http://192.168.90.111:8080/v1 is not accessible.

sudo docker run -d  -e CATTLE_AGENT_IP=192.168.90.111 -v /var/run/docker.sock:/var/run/docker.sock rancher/agent:v0.8.2 http://192.168.90.111:8080/v1/scripts/ACEEE1A6FBF4BCEA3ECA:1446685200000:QFeZOL88T9Lo9XFIffsijiZI

[mfilotto]

Ok i solved the no route to host problem by passing proxy env to the rancher server container. Now I can see the catalog.

The es stack state is now green and active. But the kopf's port 80 is still not open

I still can't add the server node, even with the proxy env, no more error message, but nothing happen.

[mfilotto]

Maybe the net configuration ?