scijava
feat: add scyjava-stubgen cli command, and `scyjava.types` namespace, which provide type-safe imports with lazy init
many more details and tests to follow... but just wanted to open this as a WIP. edit: see https://github.com/scijava/scyjava/pull/82#issuecomment-3207325549 for details
Basic idea, after checking out this branch and running pip install -e . again:
- create stubs with a cli commend, e.g.
scyjava-stubgen org.scijava:parsington:3.1.0 - Import names provided by that endpoint:
python -c "from scyjava.types.org.scijava.parsington import Function; print(Function(1))". Only at the moment of class instantiation will the jvm be started.
Codecov Report
Attention: Patch coverage is 77.46479% with 48 lines in your changes missing coverage. Please review.
Project coverage is 75.98%. Comparing base (
34bfae2) to head (0d231cc). Report is 8 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@
## main #82 +/- ##
===========================================
+ Coverage 52.72% 75.98% +23.25%
===========================================
Files 12 20 +8
Lines 1303 1653 +350
===========================================
+ Hits 687 1256 +569
+ Misses 616 397 -219
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
hey @ctrueden, I'm struggling to run the jep tests locally (on my mac). I have openjdk version "11.0.27" 2025-04-15 but still get:
--> tests/it/java_heap.py [OK]
The operation couldn’t be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.
Traceback (most recent call last):
File "/Users/talley/dev/self/scyjava/tests/it/jvm_version.py", line 11, in <module>
before_version = scyjava.jvm_version()
File "/Users/talley/dev/self/scyjava/src/scyjava/_jvm.py", line 70, in jvm_version
default_jvm_path = jpype.getDefaultJVMPath()
File "/Users/talley/dev/self/scyjava/.venv/lib/python3.13/site-packages/jpype/_jvmfinder.py", line 70, in getDefaultJVMPath
return finder.get_jvm_path()
~~~~~~~~~~~~~~~~~~~^^
File "/Users/talley/dev/self/scyjava/.venv/lib/python3.13/site-packages/jpype/_jvmfinder.py", line 184, in get_jvm_path
jvm = method()
File "/Users/talley/dev/self/scyjava/.venv/lib/python3.13/site-packages/jpype/_jvmfinder.py", line 311, in _javahome_binary
return subprocess.check_output(
~~~~~~~~~~~~~~~~~~~~~~~^
['/usr/libexec/java_home']).strip()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/talley/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/subprocess.py", line 472, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**kwargs).stdout
^^^^^^^^^
File "/Users/talley/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/subprocess.py", line 577, in run
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/libexec/java_home']' returned non-zero exit status 1.
it's a bit unclear to me whether I should be trying to get these to work at all with jep, or just add a skip to the test
As a workaround, you could try setting JAVA_HOME? Then maybe jpype wouldn't try to invoke the java_home command. Out of curiosity: what does /usr/libexec/java_home -V say when you run it from the CLI?
got that working... and commented out the "don't run on macos cause it's flaky" bit, and now get this:
-------------------------------------------
| Testing Jep mode (Python inside Java) |
-------------------------------------------
DEBUG 2025-05-01 08:20:23,154: Using settings: {'m2repo': '/Users/talley/.m2/repository', 'cachedir': '/Users/talley/.jgo', 'links': 'auto'}
DEBUG 2025-05-01 08:20:23,155: Using repositories: {'scijava.public': 'https://maven.scijava.org/content/groups/public'}
DEBUG 2025-05-01 08:20:23,155: Using shortcuts: {}
DEBUG 2025-05-01 08:20:23,155: Returning expanded coordinate black.ninia:jep:jep.Run.
DEBUG 2025-05-01 08:20:23,155: Returning expanded coordinate org.scijava:scijava-table.
INFO 2025-05-01 08:20:23,155: First time start-up may be slow. Downloaded dependencies will be cached for shorter start-up times in subsequent executions.
DEBUG 2025-05-01 08:20:23,155: Executing: ('/Users/talley/Library/Caches/cjdk/v0/misc-dirs/98cdba9371f93e1b5b8b95941b562d1647aecc21/apache-maven-3.9.9/bin/mvn', '-B', '-f', '/Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/pom.xml', 'dependency:resolve', '-X')
DEBUG 2025-05-01 08:20:31,600: Relevant mvn output: [INFO] black.ninia:jep:jar:4.2.2:compile -- module jep (auto)
DEBUG 2025-05-01 08:20:31,600: Linking source /Users/talley/.m2/repository/black/ninia/jep/4.2.2/jep-4.2.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/jep-4.2.2.jar with link_type auto
DEBUG 2025-05-01 08:20:31,600: Linking source /Users/talley/.m2/repository/black/ninia/jep/4.2.2/jep-4.2.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/jep-4.2.2.jar with link_type hard
DEBUG 2025-05-01 08:20:31,601: Relevant mvn output: [INFO] org.scijava:scijava-table:jar:1.0.2:compile -- module org.scijava.table [auto]
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-table/1.0.2/scijava-table-1.0.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-table-1.0.2.jar with link_type auto
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-table/1.0.2/scijava-table-1.0.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-table-1.0.2.jar with link_type hard
DEBUG 2025-05-01 08:20:31,601: Relevant mvn output: [INFO] org.scijava:scijava-common:jar:2.89.0:compile -- module org.scijava [auto]
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-common/2.89.0/scijava-common-2.89.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-common-2.89.0.jar with link_type auto
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-common/2.89.0/scijava-common-2.89.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-common-2.89.0.jar with link_type hard
DEBUG 2025-05-01 08:20:31,601: Relevant mvn output: [INFO] org.scijava:parsington:jar:3.0.0:compile -- module org.scijava.parsington [auto]
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/parsington/3.0.0/parsington-3.0.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/parsington-3.0.0.jar with link_type auto
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/parsington/3.0.0/parsington-3.0.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/parsington-3.0.0.jar with link_type hard
DEBUG 2025-05-01 08:20:31,601: Relevant mvn output: [INFO] org.bushe:eventbus:jar:1.4:compile -- module eventbus (auto)
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/bushe/eventbus/1.4/eventbus-1.4.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/eventbus-1.4.jar with link_type auto
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/bushe/eventbus/1.4/eventbus-1.4.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/eventbus-1.4.jar with link_type hard
DEBUG 2025-05-01 08:20:31,602: Relevant mvn output: [INFO] org.scijava:scijava-optional:jar:1.0.1:compile -- module org.scijava.optional [auto]
DEBUG 2025-05-01 08:20:31,602: Linking source /Users/talley/.m2/repository/org/scijava/scijava-optional/1.0.1/scijava-optional-1.0.1.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-optional-1.0.1.jar with link_type auto
DEBUG 2025-05-01 08:20:31,602: Linking source /Users/talley/.m2/repository/org/scijava/scijava-optional/1.0.1/scijava-optional-1.0.1.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-optional-1.0.1.jar with link_type hard
DEBUG 2025-05-01 08:20:31,602: class path: /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/*
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00000001716a3000 (most recent call first):
<no Python frame>
that's probably not the flaky bit right? looks like a poor setup of the python environment
i think i see the issue, i'll work on the jgo ... jep_test.py command directly and see if I can figure out what assumptions it's making about my environment setup
after digging a bit deeper into how jgo and jep itself is working, I'm skipping stubgen tests on jep for now. it seems extremely dependent on some careful manual setup of the python environment. I was able to get it to find my standard library, but then it lost the pure python parts of jep. After fixing that, it was unable to find my (editable) install of scyjava because it's not following .pth files in a standard way. I don't understand all the variables well enough yet (I don't understand who's doing the magic, whether it's jep.Run itself or the jgo command line), so don't know where to attack it
code-wise, this is ready to go... however, I don't expect it to be "human-interpretable" yet. There are many ways we could choose to document this and encourage/discourage its usage in various scenarios. Might be best to have a zoom about it so we can tinker with it together and discuss
This pull request has been mentioned on Image.sc Forum. There might be relevant details there:
https://forum.image.sc/t/fiji-friends-weekly-dev-update-thread/103718/94
This pull request has been mentioned on Image.sc Forum. There might be relevant details there:
https://forum.image.sc/t/poll-which-priorities-are-most-urgent-for-fijis-python-support/113172/1
Just a quick update: I removed the testing of jep mode awhile back, so that should no longer gum up the works here. @tlambert03 I'm hoping we can make a little time to revisit this PR some time next week.
yes that would be great! i do think a little chat would be super useful here
Writing some notes to self here (after having not looked at it for a while) that might help others understand what's going on here. This PR...
-
adds a new CLI command called
scyjava-stubgen. It's worth mentioning this is the first CLI command offered by scyjava (i.e. this PR addsproject.scriptstopyproject.tomlfor the first time). This command can be used to generate type stubs for any maven endpoint, for example:scyjava-stubgen org.scijava:parsington:3.1.0. -
That CLI command just parses the arguments and calls the function
scyjava._stubs.generate_stubs. (generate_stubsis a good candidate for a public function but I haven't touched the scyjava public API yet). Importantly, when called via the CLI it writes stubs by default toscyjava.typesnamespace itself. In other words it permantently writes stub files inside of thescyjavapackage (this could have permissions issues on some computers). For example, thatparsingtoncall above would createscyjava.types.org.scijava.parsington... -
After running that command (and having generated stubs) you can now import those stubs directly from the namespace:
from scyjava.types.org.scijava.parsington import Functionwith full static type support in IDEs. Which is great! that's the main goal here. 🎉
-
The important question to ask here, of course, is "when is the JVM started". That import alone does not start the JVM, instead, it returns a thin Proxy object that will start the JVM when an instance of that class is created (i.e. in the
__new__method). The logic for that is defined inscyjava._stubs.setup_java_imports. (Thatsetup_java_importsfunction is used heavily by the generated type stubs, it generates the module level__getattr__function that returns the Proxy objects imported fromscyjava.types...) -
an important detail here (as usual) is that all things would need to be imported before any of them are instantiated
open questions:
to me, the biggest open question here is "who is responsible for generating stubs, and where do they go?".
possibilities include:
- after the environment and scyjava is setup. the user could call that command line argument. That's fine and good, but not "portable" since it requires action after pip install to get the type stubs.
- Which brings the second part of this PR (which should probably be broken into a new PR), which are hatch and setuptools plugins to allow developers to build and include stubs in their own packages. More on that elsewhere
after talking with @ctrueden ... something I will try to implement:
- implement a
sys.meta_pathimporter that sees imports fromscyjava.typesand dynamically generates types - implement a entry-point specification that allows packages to declare which endpoints they need (in order to generate dynamically imported types)
for this PR I will pull out everything by the stub generating mechanism, and do the above in another PR
ok @ctrueden, I think this is ready for final review. I've removed the hatch/setuptools plugins, and double checked all the documentation. will follow up on usage patterns in another PR. Let me know if you have any questions!
@ctrueden, any thoughts on this step? I think it stands alone as useful, without imposing/deciding too much on who is going to use it
@tlambert03 Yes, sorry! I started working on a revamp of jgo a few days ago, and it's been eating my entire brain. I will try to review and merge this PR this week!
Just a quick followup to say that I'm looking at this today. I am also considering whether this feature would be best in scyjava or jgo. It will definitely land somewhere, hopefully today.
Yeah I think it could still easily live anywhere... the only thing that might eventually pin it somewhere is a decision to establish a shared public namespace like "scyjava.types". Which is still a big question mark anyway (ie, it make stay forever as a local dev tool)
OK, after reflecting on it, this work is much more suitable here in scyjava than it would be in jgo. The jgo project does not depend on jpype, whereas scyjava is, in a nutshell, unioning the benefits of jgo and jpype. (And now with the stub generation: +stubgenj as well) The only other places I could imagine this work to go would be one of jpype or stubgenj—but we can always explore pushing it upstream later after we run it through its paces.
So, in testing this work, I ran into problems making the type completions work in VSCode and PyCharm. The issue was a mismatch between the prefixed classes e.g. scyjava.types.org.scijava.Context that get stubified with stubgenj's assumption that the Python class names will match the Java class names in the toplevel Python namespace.
Or to put it another way: the stub file at e.g. src/scyjava/types/org/scijava/__init__.pyi had imports beginning with import org.scijava., rather than import scyjava.types.org.scijava.. So the type reasoning logic would have problems, like hitting dead ends for every method return type. So you couldn't type e.g.:
from scyjava.types.org.scijava import Context
from scyjava.types.org.scijava.ui import UIService
ctx = Context()
ui = ctx.service(UIService).show<ctrl+space>
and see the UIService's show methods.
With 4f546117c409456eff221ead4188b606a07c14ad, this now works, because all the type references in the stubs now get rewritten to have the proper Python package prefix preceding the Java code's own package prefix.
@tlambert03 I'm not actually sure though: is the above code how you intended this feature to be used? Or did you want to write:
from org.scijava import Context
from org.scijava.ui import UIService
ctx = Context()
ui = ctx.service(UIService).show<ctrl+space>
without the scyjava.types Python package prefixes? If the latter, we would need to generate the stubs at the top level src, not at src/scyjava/types, right?
Secondly: there remains an issue with core Java classes: this commit does not rewrite imports like import java.lang to import scyjava.types.java.lang. I think maybe it should, but wanted your opinion.
Thirdly: due to my lack of knowledge in this area, I leaned rather heavily on Claude to write this patch, and it may have errors or stupidities. What do you think? Are we on the right track here?