[BUG] ServiceListenTimeout when using FiftyOne in Visual Studio Code

[radu-diaconescu13]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04.7 LTS):

• Linux Ubuntu 16.04
  

• FiftyOne installed from (pip or source):
• pip
• FiftyOne version (run fiftyone --version):
• 0.7.2
• Python version:
• 3.7.0

Commands to reproduce

As thoroughly as possible, please provide the Python and/or shell commands used to encounter the issue.

# commands here
python
import fiftyone

Describe the problem

Hello. Thank you for your product and your hard work. I have tried to install fiftyone on a remote server with Ubuntu 16.04 Below are the things I tried.

1.I tried running the following command before installing fiftyone, but I still have the below stacktrace: pip install --upgrade pip setuptools wheel

2.I also tried: pip install --index https://pypi.voxel51.com fiftyone-db-ubuntu1604 after installling fiftyone

3.Finally, I also installed mongoDB separately, but with no luck.

I have to mention that I also tried to install on a local Ubuntu 18.04.5 LTS system and this was ok. I was able to import the library in a python script.

If anything is unclear or you need more details, please let me know. Thank you

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

{"t":{"$date":"2021-02-04T16:19:47.231Z"},"s":"I", "c":"CONTROL", "id":20697, "ctx":"main","msg":"Renamed existing log file","attr":{"oldLogPath":"/home/mars/.fiftyone/var/lib/mongo/log/mongo.log","newLogPath":"/home/mars/.fiftyone/var/lib/mongo/log/mongo.log.2021-02-04T16-19-47"}} Subprocess ['/usr/bin/mongod', '--dbpath', '/home/mars/.fiftyone/var/lib/mongo', '--logpath', '/home/mars/.fiftyone/var/lib/mongo/log/mongo.log', '--port', '0', '--nounixsocket'] exited with error 100: Uncaught exception Traceback (most recent call last): File "", line 1, in File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/fiftyone/init.py", line 28, in from fiftyone.public import * File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/fiftyone/public.py", line 11, in _database_service = fos.DatabaseService() File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/fiftyone/core/service.py", line 199, in init super().init() File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/fiftyone/core/service.py", line 80, in init self.start() File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/fiftyone/core/service.py", line 287, in start food.set_default_port(self.port) File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/fiftyone/core/service.py", line 277, in port return self._wait_for_child_port() File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/fiftyone/core/service.py", line 178, in _wait_for_child_port return find_port() File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/retrying.py", line 212, in call raise attempt.get() File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/six.py", line 703, in reraise raise value File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/fiftyone/core/service.py", line 176, in find_port raise ServiceListenTimeout(etau.get_class_name(self), port) fiftyone.core.service.ServiceListenTimeout: fiftyone.core.service.DatabaseService failed to bind to port

[benjaminpkane]

Hi @radu-diaconescu13! Thanks for the issue. I'm testing on Ubuntu 16.04 now. We haven't been doing our due diligence with testing 16.04, sorry about that.

[benjaminpkane]

@radu-diaconescu13 I was able to resolve the issue I believe.

Be sure you have libcurl3 installed.

sudo apt install libcurl3

And we need to update the troubleshooting page, apologies. Please install the fiftyone-db-ubuntu1604 from the PyPI registry.

pip install fiftyone-db-ubuntu1604

Let us know if that doesn't work. Cheers!

[radu-diaconescu13]

Hello @benjaminpkane

Thank you for your quick answer.

The steps I have taken after reading your answer:

1. Completely uninstalled prior, existing mongo DB from the remote pc
2. Ran: sudo apt install libcurl3 but I was up to date with libcurl3 version, apparently.
3. Created new, clean conda env with python=3.7.0
4. Ran: pip install fiftyone and pip install fiftyone-db-ubuntu1604

Sadly, I get the same error. However, after doing these same steps on another remote pc with Ubuntu 16.04, the fiftyone library can be imported with no error.

Any help would be appreciated regarding the install on the first remote pc. Thank you

[benjaminpkane]

Right, let' see.

Does the following run successfully for you?

/home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/fiftyone/db/bin/mongod --version

And do you have localhost configured in your /etc/hosts file?

127.0.0.1       localhost

[radu-diaconescu13]

Thank you @benjaminpkane for your help so far!

So the output for: /home/mars/anaconda3/envs/fifty/lib/python3.7/site-packages/fiftyone/db/bin/mongod --version

is

db version v4.4.2 Build Info: { "version": "4.4.2", "gitVersion": "15e73dc5738d2278b688f8929aee605fe4279b0e", "openSSLVersion": "OpenSSL 1.0.2g 1 Mar 2016", "modules": [], "allocator": "tcmalloc", "environment": { "distmod": "ubuntu1604", "distarch": "x86_64", "target_arch": "x86_64" } }

Also, when I cat /etc/hosts I do have localhost configured, i.e. 127.0.0.1 localhost does appear

[radu-diaconescu13]

OK. I think I figured it out. It seems that when I try to run fiftyone in a python script in VSCode the first time it works. Afterwards, when the script finishes, the mongoDB connection hangs and this is why I get the

fiftyone.core.service.ServiceListenTimeout: fiftyone.core.service.DatabaseService failed to bind to port

message.

I have to kill the mongo process by hand each time I run the script in VSCode, i.e. run: sudo netstat -nlp and get the mongo pid.

However, if I run in the terminal it works OK. No hung mongo DB connections. So, I have kind of a work-around for my issue, but if you have any idea for a clean fix, then please do tell.

Thank you @benjaminpkane

[benjaminpkane]

This is helpful. I will try VSCode and get back to you. Thanks!

[benjaminpkane]

I was able to reproduce. Will resolve ASAP.

[fcakyon]

@benjaminpkane still facing this issue on vs code. It makes debugging on vs code impossible with fiftyone. Are you still working on a solution?

[benjaminpkane]

@fcakyon Can you share more details about your environment? Remote/local, fiftyone version. I'm now having trouble reproducing.

My concern is that this will require some core changes to our services, which might be tricky. We'll try to at least put a priority level on this this week, though.

[fcakyon]

Local: ubuntu 18.04, conda environment, python 3.8 fiftyone: 0.9.4 fiftyone-db: 0.2.1 vs-code: 1.57.1

@benjaminpkane would gladly provide more info if you need.

[bobokvsky]

Facing same issue there.

Desktop: Windows 10 + VS-code 1.58.0 Using Remote SSH to my machine, which is

Ubuntu 20.04.2 LTS Conda environment, python 3.8.8 fiftyone: 0.11.1 fiftyone-db: 0.2.1

The first launch of fiftyone works ok, the next steps after the restart stop working with the error fiftyone.core. service.ServiceListenTimeout: fiftyone.core.service.DatabaseService failed to bind to port. Running my script from VS Code's terminal.

[collinmccarthy]

Same issue here, if I can't debug fiftyone in VSCode then I can't use it..

[benjaminpkane]

I'll mark this as high priority and dig into things.

[collinmccarthy]

@benjaminpkane Awesome, thank you!

In the meantime, I found a workaround for VSCode for others, as long as they're not using mongodb for other things (I think). Worth a try at least.

Define a task to kill any mongodb processes before running debug:

In tasks.json:

{
    "tasks": [
        {
            "label": "Kill_Mongod",
            "type": "shell",
            "command": "pkill mongod"
        }
]}

And in launch.json:

{
    "configurations": [
        {
            "name": "Fiftyone_Download",
            "type": "python",
            "request": "launch",
            "preLaunchTask": "Kill_Mongod",
            "program": "~/fiftyone_download.py",
            "args": [
                "dataset_name=open-images-v6",
                "dataset_dir=/nvme/deep-learning/open-images-v6"
            ],
            "console": "integratedTerminal",
        }
]}

The key is the preLaunchTask above. Note if switching between debugging and the command line, you'll need to manually run pkill mongod.

[MassimilianoBiancucci]

I solved it with 2 lines in python before importing fiftyone,

import os
os.system('pkill mongod')

# then import fiftyone
import fiftyone

it's simple but do its job, and you don't need to lose time configuring any task in vscode

update: in this way, after several debug iterations the problem came back, you just need to run the code without the debugger active, and then you can run it again with the debugger.

[fcakyon]

I solved it with 2 lines in python before importing fiftyone,

import os
os.system('pkill mongod')

# then import fiftyone
import fiftyone

@MassimilianoBiancucci does it work on Windows?

Let me answer that: No, Windows equivalent is tskill.

[DanTaranis]

@benjaminpkane any updates on this?

[spokucinski]

@benjaminpkane Any news? It's been 'some' time and it still happened to me...

[benjaminpkane]

Configuring your own MongoDB connection is the recommended solution. A solution for the built-in database may be worked on in the future

[spokucinski]

Ok. In the meantime I tried to trace what is causing a problem in my configuration. It seems this little feature of VSCode:

My scenario is a .devcontainer configuration based on the voxel51/fiftyone docker image. After app launch there is a pop-up with info about port redirecting on 5151, but a smaller one, in the corner as well. If I follow the suggestion and open a in-VSCode browser - everything works correctly. It seems the VS Code is itself internally proposing to show us the redirected port and keeps an alive connection, preventing a gracefull shutdown.