mistio
Elastic & Logstash containers have the "Exit 139" state
mist: v4.1.0 OS: Clear Linux (in VMware) RAM: 20GB
containers:
Name Command State Ports
---------------------------------------------------------------------------------------------------------------------------------------
mist_api_1 /mist.api/bin/docker-init ... Up 80/tcp
mist_apply-migrations_1 /mist.api/bin/docker-init ... Up
mist_beat_1 /mist.api/bin/docker-init ... Up
mist_celery-gevent_1 /mist.api/bin/docker-init ... Up
mist_celery-prefork_1 /mist.api/bin/docker-init ... Up
mist_cilia_1 /mist.api/bin/docker-init ... Up
mist_elasticsearch-manage_1 ./scripts/add_templates.py Up
mist_elasticsearch_1 /docker-entrypoint.sh elas ... Exit 139
mist_gocky_1 /go/src/github.com/mistio/ ... Up 0.0.0.0:9096->9096/tcp, 0.0.0.0:9097->9097/tcp
mist_hubshell_1 /mist.api/bin/docker-init ... Up
mist_influxdb_1 /entrypoint.sh influxd Up 0.0.0.0:8083->8083/tcp, 0.0.0.0:8086->8086/tcp
mist_kibana_1 /docker-entrypoint.sh kibana Up 5601/tcp
mist_landing_1 sh /entry.sh Up 80/tcp
mist_logstash_1 /docker-entrypoint.sh logs ... Exit 139
mist_mailmock_1 twistd -n localmail --smtp ... Up 8025/tcp
mist_memcached_1 docker-entrypoint.sh memcached Up 11211/tcp
mist_mongodb_1 docker-entrypoint.sh mongod Up 27017/tcp
mist_nginx_1 nginx -g daemon off; Up 0.0.0.0:80->80/tcp
mist_poller_1 /mist.api/bin/docker-init ... Up
mist_rabbitmq_1 docker-entrypoint.sh rabbi ... Up 15671/tcp, 15672/tcp, 25672/tcp, 4369/tcp, 5671/tcp, 5672/tcp
mist_scheduler_1 /mist.api/bin/docker-init ... Up
mist_socat_1 socat TCP4-LISTEN:2375,for ... Up 2375/tcp
mist_sockjs_1 /mist.api/bin/docker-init ... Up 8081/tcp
mist_swagger_1 sh /usr/share/nginx/docker ... Up 8080/tcp
mist_traefik_1 /traefik --accesslog --api ... Up 0.0.0.0:8040->80/tcp, 0.0.0.0:8041->8080/tcp
mist_ui_1 sh /entry.sh Up 80/tcp
elastic log:
elasticsearch_1 | # A fatal error has been detected by the Java Runtime Environment:
elasticsearch_1 | #
elasticsearch_1 | # SIGSEGV (0xb) at pc=0x00007fe79e132529, pid=1, tid=0x00007fe79eefe700
elasticsearch_1 | #
elasticsearch_1 | # JRE version: OpenJDK Runtime Environment (8.0_181-b13) (build 1.8.0_181-8u181-b13-1~deb9u1-b13)
elasticsearch_1 | # Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops)
elasticsearch_1 | # Problematic frame:
elasticsearch_1 | # C [libc.so.6+0x34529] abort+0x269
...
elasticsearch_1 | [Too many errors, abort]
elasticsearch_1 | [Too many errors, abort]
elasticsearch_1 | library initialization failed - unable to allocate file descriptor table - out of memorymist_elasticsearch_1 exited with code 139
logstash log:
logstash_1 | # A fatal error has been detected by the Java Runtime Environment:
logstash_1 | #
logstash_1 | # SIGSEGV (0xb) at pc=0x00007f8f34ece529, pid=1, tid=0x00007f8f336b1700
logstash_1 | #
logstash_1 | # JRE version: OpenJDK Runtime Environment (8.0_181-b13) (build 1.8.0_181-8u181-b13-1~deb9u1-b13)
logstash_1 | # Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops)
logstash_1 | # Problematic frame:
logstash_1 | # C [libc.so.6+0x34529] abort+0x269
...
logstash_1 | [Too many errors, abort]
logstash_1 | [Too many errors, abort]
logstash_1 | library initialization failed - unable to allocate file descriptor table - out of memorymist_logstash_1 exited with code 139
Both services seem to be run out of memory. Try restarting them and monitor their memory usage. Is there anything else on that box that could be using that much RAM? Is it a new installation or a long running one?
@d-mo new installation. I also tried version 4.0.3, there is the same problem there.
available RAM without docker ~16GB. In docker run only this containers.
the first of the two falling containers begins to reboot the logstash.
Max of RAM usage of Elastic (logged stats every 1s from the moment of launch until the fall of both containers with status 139):
2019-07-21 13:42:03 2.091GiB / 19.57GiB mist_elasticsearch_1 1c66ef8abeba
logstash ~25MiB
I'm unable to replicate the problem. In my case the elasticsearch container uses 2.2GB of RAM.
Could there be some other limit configured on your system that's preventing the elasticsearch container from claiming more than 2GB of RAM?
What happens when you run the same container image manually? eg docker run -it elasticsearch:5.6.10
I selectively checked newer container versions (5.6.16 - 7.2.0).
in one of the latest versions there were notifications that vm.max_map_count is not enough
after which I set
sudo sysctl -w vm.max_map_count = 262144
and in the latest versions this notification has been lost, but earlier versions (prior to 6.3.2, still crash with the error described earlier).
I try my set of programs with Linux distribution from Intel (Clear Linux), if it can somehow affect the result.