k.m.a.KafkaManagerActor - Updating internal state

[calvinleungyk]

I am attempting to run MyDIG on Ubuntu 16.04 on AWS. After running docker-compose up, the terminal keeps printing:

kafka_manager_1 | [info] k.m.a.KafkaManagerActor - Updating internal state...

I was also unable to add new project, which I'm not sure if it's related to the same issue. Have you encountered this problem?

[GreatYYX]

It's not a problem, it's just Kafka manager's heart beat log.

Have you seen any error logs in terminal, could you please run docker logs dig_mydig_ws_1 and./diagnose.sh?

[calvinleungyk]

When I run docker-compose up on local (Mac OSX), the repeated message scenario did not happen.

When I run docker-compose up on Ubuntu 16.04, I couldn't create a new project, plus kafka_manager keeps printing "updating internal state".

This happens when I try to add a new project: kafka_manager_1 | [info] k.m.a.KafkaManagerActor - Updating internal state... kafka_manager_1 | [info] k.m.a.KafkaManagerActor - Updating internal state... mydig_ws_1 | 172.19.0.14 - - [21/Dec/2017 10:34:51] "GET / HTTP/1.0" 200 - mydig_ws_1 | 172.19.0.14 - - [21/Dec/2017 10:34:53] "GET /constants HTTP/1.0" 200 - kafka_manager_1 | [info] k.m.a.KafkaManagerActor - Updating internal state... kafka_manager_1 | [info] k.m.a.KafkaManagerActor - Updating internal state...

Attached logs from docker logs: /app/mydig-webservice/docker_run_mydig.sh: ./daemon.sh: /usr/bin/env: bad interpreter: Text file busy /app/mydig-webservice/docker_run_mydig.sh: ./backend.sh: /usr/bin/env: bad interpreter: Text file busy /app/mydig-webservice/docker_run_mydig.sh: ./frontend.sh: /usr/bin/env: bad interpreter: Text file busy serve: Running on port 9881 killing daemon process (if exists) starting daemon (ETK spaCy) done killing backend process (if exists) starting backend done killing frontend process (if exists) starting frontend done

• Running on http://0.0.0.0:9880/ (Press CTRL+C to quit) serve: Running on port 9881
• Running on http://localhost:12121/ (Press CTRL+C to quit)
• Running on http://0.0.0.0:9879/ (Press CTRL+C to quit) 172.19.0.14 - - [21/Dec/2017 07:07:39] "GET / HTTP/1.0" 200 - 172.19.0.14 - - [21/Dec/2017 07:07:39] "GET /constants HTTP/1.0" 200 - 172.19.0.14 - - [21/Dec/2017 08:01:30] "GET / HTTP/1.0" 200 - 172.19.0.14 - - [21/Dec/2017 08:01:30] "GET /constants HTTP/1.0" 200 - killing daemon process (if exists) starting daemon (ETK spaCy) done killing backend process (if exists) starting backend done killing frontend process (if exists) starting frontend done
• Running on http://0.0.0.0:9880/ (Press CTRL+C to quit) serve: Running on port 9881
• Running on http://localhost:12121/ (Press CTRL+C to quit)
• Running on http://0.0.0.0:9879/ (Press CTRL+C to quit) 172.19.0.11 - - [21/Dec/2017 10:07:36] "HEAD / HTTP/1.1" 200 - 172.19.0.11 - - [21/Dec/2017 10:07:36] "HEAD / HTTP/1.1" 200 - 127.0.0.1 - - [21/Dec/2017 10:07:36] "HEAD / HTTP/1.1" 200 -

[GreatYYX]

I haven't met this problem before, there's a possible solution from internet: https://www.cyberciti.biz/faq/binbash-bad-interpreter-text-file-busy/

Please try to kill other processes which are using this script and run docker-compose restart mydig_ws

[calvinleungyk]

You mean killing processes that use /app/mydig-webservice/docker_run_mydig.sh and then restart mydig_ws? There is only one process using it, I killed it and the same thing happens.

Does anyone who uses MyDIG host it on AWS Ubuntu 16.04?

[calvinleungyk]

Update: the GET HTTP 2.0 does not happen when I attempt to add a new project, but when I access the web interface. But I still couldn't add a new project. Spinning up another clean AWS Ubuntu 16.04 instance to see if the problem persists.

[GreatYYX]

Found a possible solution https://github.com/moby/moby/issues/9547, it could be a linux cache problem.

@calvinleungyk please help me test it by:

1. docker exec -it dig_mydig_ws_1 bash
2. cd /app/mydig-webservice
3. vi docker_run_mydig.sh (source code is here: https://github.com/usc-isi-i2/mydig-webservice/blob/master/docker_run_mydig.sh) and add sync between chmod and ./xxx.sh
4. ./docker_run_mydig.sh

[calvinleungyk]

It solves the problem on docker logs, yielding this:

killing daemon process (if exists) starting daemon (ETK spaCy) done killing backend process (if exists) starting backend done killing frontend process (if exists) starting frontend done

• Running on http://0.0.0.0:9880/ (Press CTRL+C to quit) serve: Running on port 9881
• Running on http://localhost:12121/ (Press CTRL+C to quit)

But kafka_manager still outputs Updating internal state.

Seems like the problem with not being able to add a project is not due to this.

[GreatYYX]

Thanks. I'll add sync into new release to avoid this cache problem.

Again, "kafka_manager still outputs Updating internal state" is not an error message, it's just the heart beat log.

From what you pasted above, it seems myDIG backend is not running, that's why I asked you to run diagnose.sh. Now, let's find out why backend is not running.

1. docker exec -it dig_mydig_ws_1 bash
2. cd /app/mydig-webservice/ws
3. tail -n 200 nohup.out