label-studio-ml-backend icon indicating copy to clipboard operation
label-studio-ml-backend copied to clipboard

Provided examples won't start

Open fralik opened this issue 2 years ago • 6 comments

I am using 7a4e3153d682c33e92740b6e0b24e1dc9e6bc6b6 and trying to spin up simple text classifier. The docker compose works OK, but I can't connect to http://localhost:9090/health.

>docker compose up
[+] Running 2/0
 - Container redis   Running 0.0s
 - Container server  Created 0.0s
 Attaching to redis, server
 server  | => LABEL STUDIO HOSTNAME =  http://localhost:8082

Nothing happens if I open http://localhost:9090/health or http://localhost:9090.

Version 1.0.3 and commit a1dc0bc runs (however gives internal server error).

fralik avatar Jun 01 '22 14:06 fralik

Could you please use the latest version from the master? (It's 1.0.7 as I remember)

makseq avatar Jun 01 '22 22:06 makseq

Hi! 7a4e315 is the latest one. I just used an older one for comparison.

fralik avatar Jun 02 '22 05:06 fralik

Hi @fralik Could you please find logs from your ML backend? It should be somewhere near with docker file in logs directory.

KonstantinKorotaev avatar Jun 02 '22 08:06 KonstantinKorotaev

There are unfortunately no logs. I see logs folder. Sometimes when I enter it, I see files popping up with name like uwsgi-2123a. However, the files appear and then they are deleted immediately after, so I have no chance to look into. This happens first time I enter logs folder after I bring up the container. After this initial log appearance/disappearance, the logs folder stays empty. In order to see these strange files, I need to restart the container.

It looks to me that there is some misconfiguration in how uswgi is used. However, I do not see any obvious problems.

After playing with CMD command in docker file I can say two thins:

  1. starting a development server works fine.
  2. Changing cmd command to smth like CMD ["gunicorn", "--bind", "0.0.0.0:9090", "--workers", "1", "--timeout", "0", "_wsgi:app"] resulted in more logs in the console. The gunicorn keeps restarting:
server    | [2022-06-04 08:15:33 +0000] [1] [INFO] Starting gunicorn 20.1.0
server    | [2022-06-04 08:15:33 +0000] [1] [INFO] Listening at: http://0.0.0.0:9090 (1)
server    | [2022-06-04 08:15:33 +0000] [1] [INFO] Using worker: sync
server    | [2022-06-04 08:15:33 +0000] [9] [INFO] Booting worker with pid: 9
server    | => LABEL STUDIO HOSTNAME =  http://localhost:8082
server    | After class
server    | [2022-06-04 08:15:34 +0000] [24] [INFO] Booting worker with pid: 24
server    | => LABEL STUDIO HOSTNAME =  http://localhost:8082
server    | After class
server    | [2022-06-04 08:15:36 +0000] [39] [INFO] Booting worker with pid: 39
server    | => LABEL STUDIO HOSTNAME =  http://localhost:8082
server    | After class
server    | [2022-06-04 08:15:37 +0000] [54] [INFO] Booting worker with pid: 54
server    | => LABEL STUDIO HOSTNAME =  http://localhost:8082
server    | After class
server    | [2022-06-04 08:15:38 +0000] [69] [INFO] Booting worker with pid: 69
and so on..

fralik avatar Jun 04 '22 08:06 fralik

@fralik Did you modify the example? How much memory is allocated to docker service?

KonstantinKorotaev avatar Jun 08 '22 15:06 KonstantinKorotaev

@fralik Do you run it on Windows? Could you please replace last command in docker file to: CMD exec python3 _wsgi.py and check one more time?

KonstantinKorotaev avatar Jun 13 '22 13:06 KonstantinKorotaev

Hello, I am also having this issue using Docker with WSL. Using CMD exec python3 _wsgi.py works.

This is really weird because there shouldn't be any difference between Windows and other platforms as gunicorn is running in a Docker container.

phungthomas avatar Nov 09 '22 15:11 phungthomas

Hi @phungthomas Could you please add --preload to gunicorn cmd and try to execute it?

Using CMD exec python3 _wsgi.py works.

You can leave as it is if you don't have requirement to use gunicorn

KonstantinKorotaev avatar Nov 10 '22 09:11 KonstantinKorotaev

Thank you for your response.

Adding "--preload" unfortunately does not work (it is already included with the generated Dockerfile). The server doesn't crash but it just doesn't respond.

Using something else like "waitress" does work, and I don't really need gunicorn so it's fine for me. It just seems weird that it doesn't work through Docker.

phungthomas avatar Nov 11 '22 09:11 phungthomas