ClassNotFoundException MemoryAddress with jdk-19

[metasoarous]

I'm trying out libpython-clj 2.018 on JDK 19, and I'm getting the following error, both the first time I try to initialize, and when attempting to require libpython-clj2.require:

java.lang.ClassNotFoundException: jdk.incubator.foreign.MemoryAddress

I saw issue #180, which suggested adding :jvm-opts ["--add-modules" "jdk.incubator.foreign" "--enable-native-access=ALL-UNNAMED"] to my deps.edn for JVM versions 17 and up. I may tinker around with the version a bit, but it will require rebuilding docker images, so it would be helpful to know if anyone has any experience using libpython-clj 2 with JDK 19.

Thanks

[jjtolton]

Pinging @cnuernber on this one.

[cnuernber]

So JDK-18, 19 aren't at this time supported as they both have incompatible changes to the pathway from an integer to a direct buffer.

[cnuernber]

Specifically 16, 17, 18, 19 all contain incompatible changes there so I have been avoiding upgrading dtype-next to those versions. At this time only JDK-17 is supported beyond JDK-11.

[metasoarous]

Got it :+1: Thanks @cnuernber!

I don't have a strong reason for using any particular JDK version, so happy to stick with JDK 17 for now. It's too bad that we're having to fuss over platform "changes" though. In theory, if someone were motivated, could newer versions be supported?

Thanks!

[cnuernber]

Of course. Its just work - scanning the api changes and fixing dtype-next.

[metasoarous]

I switched to JDK 17, and am seeing the same behavior as before.

For context, I'm pretty sure libpythonclj is finding the correct python installation. And functions like py/->python works. But others do not (such as py/as-python and py/->jvm).

I have a pretty minimal reproduction of this in a Docker container at https://github.com/metasoarous/libpython-require-error. Please let me know if you see anything obviously wrong there, or have hints about what I should be looking for.

Thanks again

[jjtolton]

@metasoarous I see your issue -- and raise you an issue! https://github.com/metasoarous/libpython-require-error/issues/1

[metasoarous]

Thanks @jjtolton; That should be resolved now. Please let me know if it gets you to up the issue stack :-)

[jjtolton]

Much appreciated. I'll dive in shortly!

[cnuernber]

One thing I want to note is that to my knowledge :jvm-opts only works in an alias. A top-level :jvm-opts is silently ignored. So I think the example deps.edn needs an alias with it in it and the dockerfile's cmd needs to be updated to include that alias.

As noted in the other issue if you don't see exactly this warning:

WARNING: Using incubator modules: jdk.incubator.foreign

Then you have not loaded the module.

[jjtolton]

@metasoarous , I reconfigured your deps.edn per @cnuernber 's instructions.

Current file:

  {:paths ["src"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}
        techascent/tech.ml.dataset {:mvn/version "6.093"}
        ;techascent/tech.ml.dataset {:mvn/version "6.086"
                                    ;:exclusions [org.slf4j/slf4j-api]}
        org.bytedeco/mkl-platform-redist {:mvn/version "2020.1-1.5.3"}
        clj-python/libpython-clj {:mvn/version "2.018"}
        cider/cider-nrepl {:mvn/version "0.28.5"}
        com.taoensso/timbre {:mvn/version "5.2.1"}
        ch.qos.logback/logback-classic {:mvn/version "1.1.3"}}
 :aliases {:opts {:jvm-opts ["--add-modules" "jdk.incubator.foreign"
                       "--enable-native-access=ALL-UNNAMED"]}}}

I now get a different error, but still not working. Note:

root@802025a3b951:/app# java -version
openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment (build 17.0.2+8-86)
OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)

New error:

root@802025a3b951:/app# clj -A:opts -m libpython-test.test
WARNING: Implicit use of clojure.main with options is deprecated, use -M
WARNING: Using incubator modules: jdk.incubator.foreign
WARNING: abs already refers to: #'clojure.core/abs in namespace: tech.v3.datatype.functional-api, being replaced by: #'tech.v3.datatype.functional-api/abs
WARNING: infinite? already refers to: #'clojure.core/infinite? in namespace: tech.v3.datatype.functional-api, being replaced by: #'tech.v3.datatype.functional-api/infinite?
WARNING: abs already refers to: #'clojure.core/abs in namespace: tech.v3.datatype.functional, being replaced by: #'tech.v3.datatype.functional/abs
WARNING: infinite? already refers to: #'clojure.core/infinite? in namespace: tech.v3.datatype.functional, being replaced by: #'tech.v3.datatype.functional/infinite?
20:15:39.461 [main] INFO  libpython-clj2.python.info - Detecting startup info
20:15:39.502 [main] INFO  libpython-clj2.python - Startup info {:lib-version "3.9", :java-library-path-addendum "/usr/lib", :exec-prefix "/usr", :executable "/usr/bin/python3", :libnames ("python3.9m" "python3.9"), :prefix "/usr", :base-prefix "/usr", :libname "python3.9m", :base-exec-prefix "/usr", :python-home "/usr", :version [3 9 2], :platform "linux"}
20:15:39.502 [main] INFO  libpython-clj2.python - Prefixing java library path: /usr/lib
20:15:39.577 [main] INFO  libpython-clj2.python - Loading python library: python3.9
20:15:40.095 [main] DEBUG libpython-clj2.python.ffi - Initializing Python C Layer
20:15:40.098 [tech.resource.gc ref thread] INFO  tech.v3.resource.gc - Reference thread starting
20:15:40.100 [main] DEBUG libpython-clj2.python.ffi - Python Home: /usr
20:15:40.124 [main] INFO  tech.v3.datatype.nio-buffer - Unable to find direct buffer constructor -
falling back to jdk16 memory model.
:OK
Execution error (NullPointerException) at clojure.main/main (main.java:40).
Cannot invoke "clojure.lang.IFn.applyTo(clojure.lang.ISeq)" because "f" is null

Full report at:
/tmp/clojure-7296416279702110182.edn

[jjtolton]

Hopefully this is a more tractable error?

[jjtolton]

I think to rule out installation issues, I will try to merge the example Dockerfile with the libpython-clj example Dockerfile to try to rule out installation/version issues.

[jjtolton]

@metasoarous not sure if this solves the initial problem or not, but I put in a PR to your repo so that at least you have a minimal working setup: https://github.com/metasoarous/libpython-require-error/pull/2

[metasoarous]

@cnuernber D'oh! I did not in fact realize that this is a constraint with :jvm-opts. Not clear to me why that's not possible but :shrug: Thanks for pointing out.

@jjtolton Thank you for digging in! As mentioned in #213, I was able to get baseline functionality working by starting FROM clojure:openjdk-8-tools-deps, which I think is the same version used in the libpython-clj example Dockerfile. I may be fine continuing to use that, which would at least remove any immediate pressure for me, but it would still be nice to sort out connecting to jdk-17 if possible. (PS I had jdk-11 working at one point, but it stopped building some of the python pieces, presumably because the dockerfiles on dockerhub were updated with some weird environmental artifact :shrug:)

Thanks again!

[jjtolton]

No problem, @metasoarous. Will tinker around later and see if I can a Dockerfile for JDK-17 going. Hopefully this gives you enough for now.

[metasoarous]

First off, just wanted to confirm @cnuernber's suspicion that I wasn't using :jdk-opts properly. Now that I've done that, at least jdk-17 works. I'm happy to stick with that for a good while.

It's up to you whether you want to leave this issue open to track jdk-19 support, in case someone is inspired to work on it. I think it would be great for beginner friendliness to the ecosystem if folks aren't having to worry about JDK version so much. Trying to at least support the LTS versions is probably a pretty tenable path forward if you want to keep things simple.

Regardless, I think the README could be made a bit clearer regarding which versions are supported. I'm happy to submit an issue &/or PR for that if you agree.

Thanks again

[jjtolton]

Great point. Documentation is key for this experience! If it is fresh in your mind about where you would expect to see that documented, I would encourage you to submit a PR for it!

On Sun, Sep 4, 2022 at 1:19 AM Christopher Small @.***> wrote:

First off, just wanted to confirm @cnuernber https://github.com/cnuernber's suspicion that I wasn't using :jdk-opts properly. Now that I've done that, at least jdk-17 works. I'm happy to stick with that for a good while.

It's up to you whether you want to leave this issue open to track jdk-19 support, in case someone is inspired to work on it. I think it would be great for beginner friendliness to the ecosystem if folks aren't having to worry about JDK version so much. Trying to at least support the LTS versions is probably a pretty tenable path forward if you want to keep things simple.

Regardless, I think the README could be made a bit clearer regarding which versions are supported. I'm happy to submit an issue &/or PR for that if you agree.

Thanks again

— Reply to this email directly, view it on GitHub https://github.com/clj-python/libpython-clj/issues/212#issuecomment-1236262207, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPJX47YCP67467YQ7BPXFTV4QWMLANCNFSM6AAAAAAQCVTTMY . You are receiving this because you were mentioned.Message ID: @.***>

[jjtolton]

@metasoarous set a PR for a working reference copy with JDK-17

[cnuernber]

2.023 and dtype-next version 10.000-beta-18 FIX THIS ISSUE :-)

[metasoarous]

@cnuernber AMAZING! Thank you kindly for all your work!

[AlexAti]

Hello, just a note for people reaching for a problem similar to mine and end up here, in case it saves someone's time: Java 20 no longer has idk.incubator.foreign as a module, so your programs might have broken if you autoupdated your openjdk version (brew did this to me automatically luckily there's still openjdk@17).