Run Ai2thor in a docker container (ubuntu 20.04 system) And access GUI on windows 10 host

[xubo92]

Hi @ekolve @synapticarbors @Lucaweihs @dirkgr @schmmd

Thank you so much for creating this powerful simulator!

Since I have a Windows laptop, and it seems AI2THOR support Linux, so I create a docker container with a Ubuntu 20.04 system and install ai2thor using pip install. Then I try to convey the GUI of simulator from container to Windows X server.

I already configured the correct DISPLAY variable and launch a VcXsrv on Windows. I try first to run some X11-apps in container, such as xeyes, xclock, and I can successfully see the GUI of them from Windows. But when I try to run ai2thor example, it always said Platform Linux64 failed validation with the following errors: Invalid display, non-existent screen:192.168.1.67:0.0. But this `"192.168.1.67:0.0" is just what I use for xeyes or xclock and they all work.

The I trace back from the error and found that the ai2thor program tries to validate my screen using this function from ai2thor.platform.py @classmethod def _valid_x_displays(cls, width, height): \\ open_display_strs = [ int(os.path.basename(s)[1:]) for s in glob.glob("/tmp/.X11-unix/X[0-9]*") ] valid_displays = [] for display_str in open_display_strs: try: disp = Xlib.display.Display(":%s" % display_str) for screen in range(0, disp.screen_count()): disp_screen_str = ":%s.%s" % (display_str, screen) if cls._is_valid_screen(disp_screen_str, width, height): valid_displays.append(disp_screen_str) except Xlib.error.DisplayConnectionError as e: warnings.warn( "could not connect to X Display: %s, %s" % (display_str, e) ) return valid_displays

And I double-check the ubuntu container and found out I have no "/tmp/.X11-unix/X[0-9]*" exist. I guess this unix socket is probably the reason but I don't know how to fix. Since my X server is actually on Windows, not in the ubuntu container.

Could you please take a look and help me with it? Thanks a lot! (it took me two days)

p.s. I know there is a ai2thor-docker but on my Windows machine I don't have Nvidia-GPU and CUDA things, so I didn't try that.

Best Xubo

[mattdeitke]

Hi @xubo92,

We don't currently publicly support Windows, but it should be possible, and is something I've looked into.

AI2-THOR Colab

First, if you are just looking to experiment with AI2-THOR on a Windows device, and not training any models, I'd suggest using ai2thor-colab. It will handle all the dependency installations and setup, and provides a nice environment to play around with AI2-THOR. Granted, using Colab probably will not be fast enough to train any models.

Running Locally on Windows

This is a bit more manually involved. Hopefully at some point soon, we will officially support Windows, so one doesn't need to manually open up Unity. But, here are the steps:

1. Download the Unity Editor version 2019.4.20 LTS from the Unity Download Archive
2. Clone AI2-THOR locally (e.g., git clone https://github.com/allenai/ai2thor.git)
3. Open up the ai2thor/unity/ folder in Unity.
4. Go to File -> Build Settings. You should see a screen similar to what is shown below. Click Build and save the build to some path. This will take at least a couple of minutes to save.
5. Install ai2thor v3.1.0 (e.g., pip install ai2thor==3.1.0). Note: There have since been some updates that break this with the lastest Python version, hence why v3.1.0 of the Python API must be installed. Specifically, build.py not expecting a Windows OS.
6. Step 4 will save a file called AI2-THOR.exe into the provided directory. Now in Python, make sure ai2thor is installed (e.g., pip install ai2thor), and point the Controller to that AI2-THOR.exe path:

from ai2thor.controller import Controller
controller = Controller(local_executable_path="AI2-THOR.exe")

This creates a Controller and should have everything working with Windows.

Also Note: Windows tends to be slower than a Linux based system because one cannot utilize the FIFO Server speedups from AI2-THOR v2.7.

[xubo92]

Hi @mattdeitke

Thank you so much for providing this alternative! I've configured in windows following your steps and it works!

[mariiak2021]

Hi @mattdeitke

Thank you for the instructions! I managed to follow and install, but have some problems with simulator output. If I initialize the Controller, I'm getting the window with graphical output, but can't do anything with it - no rotations, no movements. The window is just appearing and that's it. Can you give any advice please?

[mattdeitke]

Hi, @mariiak2021. Yes, this is expected.

As pictured in the loop above, we use Python commands to control the agent.

What you might try is some movement actions, such as:

import time
import random

from ai2thor.controller import Controller
controller = Controller(local_executable_path="AI2-THOR.exe")

for _ in range(10):
    time.sleep(1)
    action = random.choice(["MoveAhead", "RotateRight"])
    event = controller.step(action)
    rgb_frame = event.frame

I'd recommend checking out this page of the documentation for more navigation actions :)

[mariiak2021]

Hi @mattdeitke, thank you a lot for the example! More clear for me now. :)