match icon indicating copy to clipboard operation
match copied to clipboard

Published 20 hours ago • verified publisher icon dsys

WORKER TIMEOUT on application start in AWS EC2 instance

Open dboterho opened this issue 6 years ago • 4 comments

Hello,

We have been trying to set up an instance of the application on an AWS EC2 instance. Everything seems to be setup correctly; EC2 instance is running, we can telnet to the ports, attached volume is OK, and AWS Elasticsearch instance appears accessible and running. When we hit the /ping URL it times out. And looking at the docker logs we can see the below messages:

[2018-12-04 16:50:33 +0000] [1946] [INFO] Booting worker with pid: 1946 [2018-12-04 16:50:33 +0000] [1947] [INFO] Booting worker with pid: 1947 [2018-12-04 16:50:33 +0000] [1950] [INFO] Booting worker with pid: 1950 [2018-12-04 16:50:33 +0000] [1954] [INFO] Booting worker with pid: 1954 [2018-12-04 16:51:33 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:1946) [2018-12-04 16:51:33 +0000] [1946] [INFO] Worker exiting (pid: 1946) [2018-12-04 16:51:33 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:1947) [2018-12-04 16:51:33 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:1950) [2018-12-04 16:51:33 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:1954) [2018-12-04 16:51:33 +0000] [1954] [INFO] Worker exiting (pid: 1954) [2018-12-04 16:51:33 +0000] [1947] [INFO] Worker exiting (pid: 1947) [2018-12-04 16:51:33 +0000] [1950] [INFO] Worker exiting (pid: 1950)

We have ruled out memory and cache issues and tried using different versions. The issue appears to originate from the dsys/match image. Does anybody know what our issue is?

AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20180912 (ami-07a3bd4944eb120a0) EC2 instance type: t2.medium Elastic Search version: 6.3

Thanks

dboterho avatar Dec 07 '18 00:12 dboterho

Same problem here. Did you get with a solution?

cucomans avatar Apr 25 '19 21:04 cucomans

@cucomans no, didn't find a workround and went with an alternate solution (which didn't include phash).

dboterho avatar Apr 26 '19 00:04 dboterho

It was working fine untill last monday and then boom this error started to occurr.

cucomans avatar Apr 26 '19 00:04 cucomans

I had the same problem, from the logs I was getting the following: Elasticsearch: Max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Following this stackoverflow i did the following: https://stackoverflow.com/questions/51445846/elasticsearch-max-virtual-memory-areas-vm-max-map-count-65530-is-too-low-inc

edit /etc/sysctl.conf and added vm.max_map_count = 262144 to the end of the file

ran sysctl -w vm.max_map_count=262144 then systemctl restart docker

lklic avatar Apr 16 '21 15:04 lklic