jpype

jpype copied to clipboard

Hi, I am having trouble running the GUIs using Jframe in macos. The code I am running is as follows:

import jpype
import jpype.imports

jpype.startJVM()
import java
import javax
from javax.swing import *

def createAndShowGUI():
    frame = JFrame("HelloWorldSwing")
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
    label = JLabel("Hello World")
    frame.getContentPane().add(label)
    frame.pack()
    frame.setVisible(True)

# Start an event loop thread to handling gui events
@jpype.JImplements(java.lang.Runnable)
class Launch:
    @jpype.JOverride
    def run(self):
        createAndShowGUI()
javax.swing.SwingUtilities.invokeLater(Launch())

And after running the code above, there is a python icon showing up in the task bar, but not really showing anything when I clicked on it as shown in the screenshot.

I can verify this is not an issue on the linux side.

If you guys have issues accessing a mac for debugging, I will be happy to provide a macos on EC2 for the purpose of debugging this issue. Feel free to contact me personally at costa.huang at outlook dot com to get the macos VNC credentials.

This is also a blocker for https://github.com/vwxyzjn/gym-microrts/issues/3

I believe there are some macros in jpype.gui for this (using AppHelper) though I have never used them. They were never documented and I am not sure if they ever worked.

Unfortunately I don't have access to a mac. It runs fine on windows and linux for me. I should note that your application is falling through to the exit routine so Python may not be in a health state. I would try to add something to make sure that you are not about to exit. Just add a long sleep for now. This would make sure that we don't have problems because the main thread has died and you are running code from a spawned thread. (Which could be a race condition.)

Also try adding some print statement to make sure that it is actually hitting the createAndShowGUI. It is possible that it failed due to a race with main.

I remember that I tried one of these gui examples (Swing) with success a long time ago. @vwxyzjn Are you sure, you have set up your ssh connection to forward the window to your local X11 (e.g. use ssh -X, no idea about OSX X11, if installed/enabled by default).

That was done using VNC, and everything seems set up correctly. What Mac OS version did you run it on?

I don't use Apple products. Could it be, that you're using a headless version of the JRE/JDK?

@marscher Thanks for the reply. On Windows and Linux it seems fine, it’s just on Mac this strange issue appears. I am pretty sure that I used a non headless version.

OK, it then would be very helpful, if you could annotate the script with some print statements to see where the code gets stuck.

Looks like the script got stuck at import with https://adoptopenjdk.net/releases.html?variant=openjdk8&jvmVariant=hotspot

Apologies for the screenshots, hard to modify the script over vnc. The following are results for the official java here https://www.oracle.com/java/technologies/javase-jdk15-downloads.html

This is also using Python 3.9, which I don't know if it makes a difference.

Sadly I don't know how to begin debugging something like this. It looks like it fails on the first call to create a screen resource.

What happens if you try to create a screen resource in the main thread. Does that work?

Thanks fro the reply. What does that mean in the main thread? Do you have a code sample I could try it out by any chance?

I think this "invokeLater" method does create a new thread, if you directly invoke your JFrame creating function it should run in the main thread.

I mean you should not wrap it in the Runnable (Thread) interface

import jpype
import jpype.imports

jpype.startJVM()
print("jvm started")
import java
import javax
from javax.swing import *
print("java swing imported")

def createAndShowGUI():
    print("l1")
    frame = JFrame("HelloWorldSwing")
    print("l2")
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
    print("l3")
    label = JLabel("Hello World")
    print("l4")
    frame.getContentPane().add(label)
    print("l5")
    frame.pack()
    print("l6")
    frame.setVisible(True)
    print("l7")


createAndShowGUI()

So definitely have some interesting results :) The code is able to finish but still not having the window show up.

If by chance you are interested in using this mac instance through VNC, feel free to download this private key (two one-time link)

https://file.io/o5eQB6ShSj9x https://file.io/plInRCcXekvZ

and run

ssh -L 5900:localhost:5900 -i m.pem [email protected]
# now open a vnc to connect `vnc://ec2-user@localhost`

I unfortunately have work for now so I can't help with a debugging session. I did try the code you sent and it runs and shows the window as expected. So this is still looking a lot like X session or headless issue. So the next step would be to make sure this isn't a race condition with termination. As your main body just has Python terminated after calling the create... so lets make it

createAndShowGUI()

# Make sure we are not getting into a race
print("post")
java.lang.Thread.sleep(10000)
print("done")

Or we can make sure that X11 is working at all.

import matplotlib.pyplot as plt
plt.plot([0,1])

createAndShowGUI()

# Make sure that X11 is reachable
plt.show()

second snippet seems very interesting

So the hello world window only shows when I do the following

import matplotlib.pyplot as plt
plt.plot([0,1])

createAndShowGUI()

# Make sure that X11 is reachable
plt.show()

And the following does not show the hello world window

import matplotlib.pyplot as plt

createAndShowGUI()

# Make sure that X11 is reachable
plt.show()

And it kind of works with gym-microrts now!! :D

import gym
import gym_microrts
import time
import numpy as np
from gym.wrappers import Monitor
from gym_microrts import microrts_ai
import matplotlib.pyplot as plt

env = gym.make(
    "MicrortsDefeatCoacAIShaped-v3",
    render_theme=2, # optional customization
    frame_skip=0, # optional customization
    ai2=microrts_ai.coacAI, # optional customization
    map_path="maps/16x16/basesWorkers16x16.xml", # optional customization
    reward_weight=np.array([10.0, 1.0, 1.0, 0.2, 1.0, 4.0, 0.0]) # optional customization
    # the above `reward_weight` (in order) means +10 for wining,
    # +1 for each resource gathered or returned, +1 for each worker produced
    # +0.2 for each building produced, +1 for each attack action issued
    # +4 for each combat units produced, +0.0 for getting `closer` to enemy base
)
# env = Monitor(env, f'videos', force=True)
env.action_space.seed(0)
env.reset()
for i in range(100):
    plt.plot([0,1])
    env.render()
    plt.show()
    time.sleep(0.001)
    action = env.action_space.sample()

    # optional: selecting only valid units.
    if len((env.unit_location_mask==1).nonzero()[0]) != 0:
        action[0] = (env.unit_location_mask==1).nonzero()[0][0]

    next_obs, reward, done, info = env.step(action)
    if done:
        env.reset()
env.close()
print("done")

However I have to close the plotted window for the game to continue

So that seems to indicate that something has not opened the X11 connection prior to call to JFrame. When we call matplotlib it initializes the X11 connection, and then once initialized Java can use the open connection. But it we don't have an open connection then X11 fails. What happens if you simply clear the figure without asking it to be shown?

So the question we need to resolve is how to get the X11 connection open without depending on matplotlib being called first.

Thanks so much for helping to debug this. What do you mean by "clearing the figure". Do you mean plt.clf()? So the following does not work

plt.plot([0,1])
createAndShowGUI()

plt.plot([1,2])
createAndShowGUI()
plt.clf()

So the show is required. What about?

# Create a figure and proceed
plt.show(block=false)

# Close the figure
plt.close()
``

import matplotlib.pyplot as plt
plt.plot([1,2])

createAndShowGUI()
# Make sure that X11 is reachable
# Create a figure and proceed
plt.show(block=False)

# Close the figure
plt.close()

does not work

Interesting. Well we have some clue of what is going on, but not the root source of the problem. I am sure that it is some osx specific command that is required to start the application loop. When you are using matplotlib on osx it creates the required resource, but when you use Java it is getting missed. Unfortunately beyond pointing you to the jpype/_gui hooks and isolating it to not being a connection problem but rather something missing (because matplotlib succeeds where Java does not), I am at my limit of osx knowledge.

I hope this will point in the direction of useful web searching, though I feel that it is not specific enough yet.

Have you tried the simple experiment of writing the application in pure Java and testing using the same java that is being launched from JPype. Perhaps we can isolate this further?

Have you tried the simple experiment of writing the application in pure Java and testing using the same java that is being launched from JPype. Perhaps we can isolate this further?

The short answer is yes. The renderer for gym-microrts is written entirely on the java side: https://github.com/vwxyzjn/microrts/blob/3461c3ecf6f20344c89d36b0bf48da18a1843df7/src/tests/JNIClient.java#L105-L120

This issue is extremely interesting to one of our projects: paquo

Essentially we've been facing similar problems when trying to run the QuPath GUI with full control from Python on OSX. I'm not sure if it's helpful, but here are some related links:

See: https://forum.image.sc/t/paquo-read-write-qupath-projects-from-python/41892/14?u=poehlmann And the referenced thread from above: https://forum.image.sc/t/displaying-imagej-and-napari-ui-simultaneously/32187

If I understand correctly, the main issue for us is that (and I quote @sdvillal):

"macOS constrains the event loop of GUI applications to run in the main thread of the process, which means the python process itself gets blocked".

I have a few experiments with jpype._gui where I can get some functionality to work, but the Python interpreter falls through and is in a non healthy state - kept alive before exit. I'm relatively busy for the next few days, but might be able to post an example the day after tomorrow.

Cheers, Andreas 😃

@ap-- Thanks so much for chiming in. By using %gui osx in jupyter, I can indeed get it work with the hello world example

However, still causing issue for gym-microrts

@ap-- Can you make the Python main call some kind of handle even loop and wait for event loop to complete before proceeding? I am not sure what call that is in Java side by it would transfer control back to Java which frees the Python interpreter to run code in other threads and would prevent falling into the exit state until after the GUI is complete, but I am sure there should be something.

Perhaps it can be something as simple as adding join statement.

http://www.javased.com/?post=1341699

I know there is a call to check which thread is the event dispatch, but I don't know how to get the thread through swing.

https://docs.oracle.com/javase/8/docs/api/javax/swing/SwingUtilities.html#isEventDispatchThread--

@vwxyzjn what happens if instead you use the "%gui qt" magic? https://forum.image.sc/t/paquo-read-write-qupath-projects-from-python/41892/17

@sdvillal does not seem to work