libpython-clj icon indicating copy to clipboard operation
libpython-clj copied to clipboard

Published 20 hours ago • verified publisher icon clj-python

Python Interop on a (32-bit, ARM) Raspberry Pi Zero?

Open ajpierce opened this issue 5 years ago • 14 comments

Thanks for this project! I enjoyed the talk from this year's Conj and it made me want to dive right in.

My use case is somewhat unique, I think: I'm trying to use Python to drive some hardware I have soldered to the GPIO pins on a Raspberry Pi Zero. The Pi is a 32-bit ARM processor, and Python works great on it... but Python interop has been giving me a bit of trouble.

Here's what we're working with:

 $ ldconfig -p | grep python
        libpython3.7m.so.1.0 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libpython3.7m.so.1.0
        libpython3.7m.so (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libpython3.7m.so
        libpython2.7.so.1.0 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libpython2.7.so.1.0
        libpython2.7.so (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libpython2.7.so

and Java:

$ java -version
openjdk version "1.8.0_152"
OpenJDK Runtime Environment (Zulu Embedded 8.25.0.76-linux-aarch32hf) (build 1.8.0_152-b76)
OpenJDK Client VM (Zulu Embedded 8.25.0.76-linux-aarch32hf) (build 25.152-b76, mixed mode, Evaluation)

I'm using Python 3.7 to drive the hardware, and Python itself runs fine. Likewise, I can run Clojure (both JARs and Clojure in the REPL) without any issue. But when I try to run Python from within Clojure via the REPL, here's what happens:

=> (require '[libpython-clj.python :as py])
nil
=> (py/initialize!)
Jan 09, 2020 3:05:16 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Executing python initialize!
Jan 09, 2020 3:05:17 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Startup info detected: {:python-home "/usr", :lib-version "3.7", :libname "python3.7m", :java-library-path-addendum "/usr/lib"}
Jan 09, 2020 3:05:17 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Reference thread starting
Jan 09, 2020 3:05:18 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Library python3.7m found at [:system "python3.7m"]
Execution error (IllegalArgumentException) at libpython-clj.jna.base/eval22533$fn$G (base.clj:20).
No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil

The error repeats itself if you try any other commands:

=> (py/run-simple-string "print('Hello')")
Execution error (IllegalArgumentException) at libpython-clj.jna.base/eval22533$fn$G (base.clj:20).
No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil

I started the repl using lein repl in a new project root. The project was scaffolded with the lein new app command.

Do you see anything off the top of your head that may be the root cause of these issues?

My gut tells me it's one of three things:

  1. The 32-bit ARM processor, and everything that goes along with that
  2. The Zulu Embedded Java 8 JDK
  3. Something dumb that I haven't noticed yet.

Any thoughts as to why this is happening, or is there any more information I could provide?

Really appreciate your time. Thank you!

ajpierce avatar Jan 09 '20 03:01 ajpierce

Hey, thanks, an embedded use case is an interesting pathway for sure. The entire stack trace would be helpful. That method gets called from a lot of potential places.

On line 48 there is ensure-pyobj - I think something is nil in there and thus you are getting the exception but we need to figure out what part of the initialization process is failing. So a full stack trace of the error would be helpful.

It looks as if the first half of initialize happens correctly. The second half is when it is overriding stdout/stderr. One thing you may try is to request the system not attempt io redirection as this simplifies the initialization process (but you won't get stderr/stdout reflected into out and err).

https://github.com/cnuernber/libpython-clj/blob/master/src/libpython_clj/python.clj#L229

Aside from that, we need to get full stack traces :-).

cnuernber avatar Jan 09 '20 18:01 cnuernber

Hi, Chris! Thanks for the prompt reply.

I don't yet have the pleasure of being able to use Clojure at my day job, so I was pleasantly surprised to learn about the *e variable tonight after seeing your message! I had no idea that we could extract more from the REPL when things go awry, so thank you for that 😁

=>  (require '[libpython-clj.python :as py])
nil

=> (py/initialize!)
Jan 10, 2020 2:49:19 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Executing python initialize!
Jan 10, 2020 2:49:20 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Startup info detected: {:python-home "/usr", :lib-version "3.7", :libname "python3.7m", :java-library-path-addendum "/usr/lib"}
Jan 10, 2020 2:49:20 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Reference thread starting
Jan 10, 2020 2:49:21 AM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Library python3.7m found at [:system "python3.7m"]
Execution error (IllegalArgumentException) at libpython-clj.jna.base/eval22533$fn$G (base.clj:20).
No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil

=> *e
#error {
 :cause "No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil"
 :via
 [{:type java.lang.IllegalArgumentException
   :message "No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil"
   :at [clojure.core$_cache_protocol_fn invokeStatic "core_deftype.clj" 583]}]
 :trace
 [[clojure.core$_cache_protocol_fn invokeStatic "core_deftype.clj" 583]
  [clojure.core$_cache_protocol_fn invoke "core_deftype.clj" 575]
  [libpython_clj.jna.base$eval22533$fn__22534$G__22524__22539 invoke "base.clj" 20]
  [libpython_clj.jna.base$ensure_pyobj invokeStatic "base.clj" 50]
  [libpython_clj.jna.base$ensure_pyobj invoke "base.clj" 48]
  [libpython_clj.jna.concrete.unicode$PyUnicode_AsUTF8 invokeStatic "unicode.clj" 79]
  [libpython_clj.jna.concrete.unicode$PyUnicode_AsUTF8 invoke "unicode.clj" 79]
  [libpython_clj.jna.interpreter$get_type_name invokeStatic "interpreter.clj" 219]
  [libpython_clj.jna.interpreter$get_type_name invoke "interpreter.clj" 216]
  [clojure.lang.Var invoke "Var.java" 384]
  [libpython_clj.python.interpreter$py_type_keyword$fn__24162 invoke "interpreter.clj" 340]
  [clojure.lang.Atom swap "Atom.java" 37]
  [clojure.core$swap_BANG_ invokeStatic "core.clj" 2352]
  [clojure.core$swap_BANG_ invoke "core.clj" 2345]
  [libpython_clj.python.interpreter$py_type_keyword invokeStatic "interpreter.clj" 333]
  [libpython_clj.python.interpreter$py_type_keyword invoke "interpreter.clj" 327]
  [libpython_clj.python.object$eval26422$fn__26423$fn__26424 invoke "object.clj" 231]
  [libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 377]
  [libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
  [libpython_clj.python.object$eval26422$fn__26423 invoke "object.clj" 230]
  [libpython_clj.python.protocols$eval24268$fn__24269$G__24259__24274 invoke "protocols.clj" 21]
  [libpython_clj.python.protocols$python_type invokeStatic "protocols.clj" 30]
  [libpython_clj.python.protocols$python_type invoke "protocols.clj" 26]
  [libpython_clj.python.object$wrap_pyobject invokeStatic "object.clj" 136]
  [libpython_clj.python.object$wrap_pyobject doInvoke "object.clj" 122]
  [clojure.lang.RestFn invoke "RestFn.java" 410]
  [libpython_clj.python.object$incref_wrap_pyobject$fn__26409 invoke "object.clj" 189]
  [libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 377]
  [libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
  [libpython_clj.python.object$incref_wrap_pyobject invokeStatic "object.clj" 186]
  [libpython_clj.python.object$incref_wrap_pyobject invoke "object.clj" 182]
  [libpython_clj.python.interop$add_module$fn__26904 invoke "interop.clj" 87]
  [libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 377]
  [libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
  [libpython_clj.python.interop$add_module invokeStatic "interop.clj" 85]
  [libpython_clj.python.interop$add_module invoke "interop.clj" 83]
  [libpython_clj.python.interop$register_bridge_type_BANG_$fn__26947 invoke "interop.clj" 281]
  [libpython_clj.python.interpreter$with_gil_fn$fn__24167$fn__24168 invoke "interpreter.clj" 360]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 665]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1973]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1973]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [libpython_clj.python.interpreter$with_gil_fn$fn__24167 invoke "interpreter.clj" 358]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 665]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1973]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1973]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 356]
  [libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
  [libpython_clj.python.interop$register_bridge_type_BANG_ invokeStatic "interop.clj" 280]
  [libpython_clj.python.interop$register_bridge_type_BANG_ doInvoke "interop.clj" 276]
  [clojure.lang.RestFn invoke "RestFn.java" 397]
  [libpython_clj.python$initialize_BANG_ invokeStatic "python.clj" 244]
  [libpython_clj.python$initialize_BANG_ doInvoke "python.clj" 229]
  [clojure.lang.RestFn invoke "RestFn.java" 397]
  [squishy_kitty.core$eval28099 invokeStatic "form-init4165223302715120006.clj" 1]
  [squishy_kitty.core$eval28099 invoke "form-init4165223302715120006.clj" 1]
  [clojure.lang.Compiler eval "Compiler.java" 7176]
  [clojure.lang.Compiler eval "Compiler.java" 7131]
  [clojure.core$eval invokeStatic "core.clj" 3214]
  [clojure.core$eval invoke "core.clj" 3210]
  [clojure.main$repl$read_eval_print__9068$fn__9071 invoke "main.clj" 414]
  [clojure.main$repl$read_eval_print__9068 invoke "main.clj" 414]
  [clojure.main$repl$fn__9077 invoke "main.clj" 435]
  [clojure.main$repl invokeStatic "main.clj" 435]
  [clojure.main$repl doInvoke "main.clj" 345]

Please let me know if there are any more gaps I can help fill in!

ajpierce avatar Jan 10 '20 02:01 ajpierce

To follow up on :no-io-redirect?, we get the same behavior as before:

squishy-kitty.core=>  (require '[libpython-clj.python :as py])
nil

squishy-kitty.core=> (py/initialize! :no-io-redirect? true)
Jan 10, 2020 12:54:30 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Executing python initialize!
Jan 10, 2020 12:54:31 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Startup info detected: {:python-home "/usr", :lib-version "3.7", :libname "python3.7m", :java-library-path-addendum "/usr/lib"}
Jan 10, 2020 12:54:31 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Reference thread starting
Jan 10, 2020 12:54:32 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Library python3.7m found at [:system "python3.7m"]
Execution error (IllegalArgumentException) at libpython-clj.jna.base/eval22533$fn$G (base.clj:20).
No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil

squishy-kitty.core=> *e
#error {
 :cause "No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil"
 :via
 [{:type java.lang.IllegalArgumentException
   :message "No implementation of method: :->py-object-ptr of protocol: #'libpython-clj.jna.base/PToPyObjectPtr found for class: nil"
   :at [clojure.core$_cache_protocol_fn invokeStatic "core_deftype.clj" 583]}]
 :trace
 [[clojure.core$_cache_protocol_fn invokeStatic "core_deftype.clj" 583]
  [clojure.core$_cache_protocol_fn invoke "core_deftype.clj" 575]
  [libpython_clj.jna.base$eval22533$fn__22534$G__22524__22539 invoke "base.clj" 20]
  [libpython_clj.jna.base$ensure_pyobj invokeStatic "base.clj" 50]
  [libpython_clj.jna.base$ensure_pyobj invoke "base.clj" 48]
  [libpython_clj.jna.concrete.unicode$PyUnicode_AsUTF8 invokeStatic "unicode.clj" 79]
  [libpython_clj.jna.concrete.unicode$PyUnicode_AsUTF8 invoke "unicode.clj" 79]
  [libpython_clj.jna.interpreter$get_type_name invokeStatic "interpreter.clj" 219]
  [libpython_clj.jna.interpreter$get_type_name invoke "interpreter.clj" 216]
  [clojure.lang.Var invoke "Var.java" 384]
  [libpython_clj.python.interpreter$py_type_keyword$fn__24162 invoke "interpreter.clj" 340]
  [clojure.lang.Atom swap "Atom.java" 37]
  [clojure.core$swap_BANG_ invokeStatic "core.clj" 2352]
  [clojure.core$swap_BANG_ invoke "core.clj" 2345]
  [libpython_clj.python.interpreter$py_type_keyword invokeStatic "interpreter.clj" 333]
  [libpython_clj.python.interpreter$py_type_keyword invoke "interpreter.clj" 327]
  [libpython_clj.python.object$eval26422$fn__26423$fn__26424 invoke "object.clj" 231]
  [libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 377]
  [libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
  [libpython_clj.python.object$eval26422$fn__26423 invoke "object.clj" 230]
  [libpython_clj.python.protocols$eval24268$fn__24269$G__24259__24274 invoke "protocols.clj" 21]
  [libpython_clj.python.protocols$python_type invokeStatic "protocols.clj" 30]
  [libpython_clj.python.protocols$python_type invoke "protocols.clj" 26]
  [libpython_clj.python.object$wrap_pyobject invokeStatic "object.clj" 136]
  [libpython_clj.python.object$wrap_pyobject doInvoke "object.clj" 122]
  [clojure.lang.RestFn invoke "RestFn.java" 410]
  [libpython_clj.python.object$incref_wrap_pyobject$fn__26409 invoke "object.clj" 189]
  [libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 377]
  [libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
  [libpython_clj.python.object$incref_wrap_pyobject invokeStatic "object.clj" 186]
  [libpython_clj.python.object$incref_wrap_pyobject invoke "object.clj" 182]
  [libpython_clj.python.interop$add_module$fn__26904 invoke "interop.clj" 87]
  [libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 377]
  [libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
  [libpython_clj.python.interop$add_module invokeStatic "interop.clj" 85]
  [libpython_clj.python.interop$add_module invoke "interop.clj" 83]
  [libpython_clj.python.interop$register_bridge_type_BANG_$fn__26947 invoke "interop.clj" 281]
  [libpython_clj.python.interpreter$with_gil_fn$fn__24167$fn__24168 invoke "interpreter.clj" 360]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 665]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1973]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1973]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [libpython_clj.python.interpreter$with_gil_fn$fn__24167 invoke "interpreter.clj" 358]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 665]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1973]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1973]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [libpython_clj.python.interpreter$with_gil_fn invokeStatic "interpreter.clj" 356]
  [libpython_clj.python.interpreter$with_gil_fn invoke "interpreter.clj" 346]
  [libpython_clj.python.interop$register_bridge_type_BANG_ invokeStatic "interop.clj" 280]
  [libpython_clj.python.interop$register_bridge_type_BANG_ doInvoke "interop.clj" 276]
  [clojure.lang.RestFn invoke "RestFn.java" 397]
  [libpython_clj.python$initialize_BANG_ invokeStatic "python.clj" 244]
  [libpython_clj.python$initialize_BANG_ doInvoke "python.clj" 229]
  [clojure.lang.RestFn invoke "RestFn.java" 421]
  [squishy_kitty.core$eval28099 invokeStatic "form-init7511438152933693024.clj" 1]
  [squishy_kitty.core$eval28099 invoke "form-init7511438152933693024.clj" 1]
  [clojure.lang.Compiler eval "Compiler.java" 7176]
  [clojure.lang.Compiler eval "Compiler.java" 7131]
  [clojure.core$eval invokeStatic "core.clj" 3214]
  [clojure.core$eval invoke "core.clj" 3210]
  [clojure.main$repl$read_eval_print__9068$fn__9071 invoke "main.clj" 414]
  [clojure.main$repl$read_eval_print__9068 invoke "main.clj" 414]
  [clojure.main$repl$fn__9077 invoke "main.clj" 435]
  [clojure.main$repl invokeStatic "main.clj" 435]
  [clojure.main$repl doInvoke "main.clj" 345]
  [clojure.lang.RestFn invoke "RestFn.java" 1523]
  [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 79]
  [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 55]
  [nrepl.middleware.interruptible_eval$interruptible_eval$fn__939$fn__943 invoke "interruptible_eval.clj" 142]
  [clojure.lang.AFn run "AFn.java" 22]
  [nrepl.middleware.session$session_exec$main_loop__1040$fn__1044 invoke "session.clj" 171]
  [nrepl.middleware.session$session_exec$main_loop__1040 invoke "session.clj" 170]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.lang.Thread run "Thread.java" 748]]}

ajpierce avatar Jan 10 '20 12:01 ajpierce

That stack trace did help but I am stumped a bit. The module object created by calling 'add-module' has no __name__ attribute which seems to be a failure in the API.

There are a few questions I have at this point. What I am trying to do there is create a new module that holds the special python type we use to bridge things. Without that working almost none of the system will work.

It is odd that an added module wouldn't have a name attribute which I believe is the source of the issue.

I would like to play with this setup. Is there a way to get all this running in a VM or something like that so I can try it on my computer? Or, conversly, my buddy just got an embedded device and when I get back to Boulder I can grab it from him and try things out. For sure, there is a way to get this to work. It feels to me like there are just a few things that need to be tweaked for all of this to fly.

One way forward would be to hack libpython-clj to survive that possibility. If this code try/catches the error and just returns :unnamed-thing or something like that we could see what the next issue is. My guess is there will be 3 or for things like this before this setup works. If you are feeling bold, you can modify that file and locally install the library or modify it from within your repl and see if you can at least get the bridging type to register.

Great bug, this is interesting :-).

cnuernber avatar Jan 10 '20 19:01 cnuernber

Hi Chris, thanks again for the info!

The first thing I'm going to do is try to bypass the __name__ check. Then, probably in parallel, I'm going to see if I can get a VM to replicate what's happening on my hardware. If I can, I'll upload it to S3 and send you a link for downloading.

If I can't virtually reproduce what's happening, I'm going to go back to Microcenter and plunk down the ten whole dollars it'll cost me to buy another Pi Zero and see if I can replicate the exact environment I'm using (which, actually, the steps have been documented in this blog post and are continued in the next post.

If I'm successful physically but not virtually replicating the goings-on, I will reach out to you again for some shipping information and send you the hardware so you can play with it yourself :)

I'll report back later with developments!

owap avatar Jan 10 '20 20:01 owap

That is great. One thing I noticed is that you are running in a 32 bit arm environment. While I tried to make everything just work on 32 bit environments, I haven't tested this and so almost certainly there are things hardcoded to live in the 64 bit world. I can work through at least getting things to work in a 32 bit vm and making sure all of that is sane which I think is a good parallel pathway to what you are up to.

cnuernber avatar Jan 11 '20 19:01 cnuernber

I broke things!

First, let me say that my attempts to get a VM of Raspbian up and running have all resulted in failure. Part of me wonders if the issue lies in my host machine, but I haven't tried a base case of virtualizing something like Ubuntu or Debian yet. I may revisit that later.

On the "modify libpython-clj" front, I made two modifications locally:

diff --git a/src/libpython_clj/jna/interpreter.clj b/src/libpython_clj/jna/interpreter.clj
index 59dcabe..9270478 100644
--- a/src/libpython_clj/jna/interpreter.clj
+++ b/src/libpython_clj/jna/interpreter.clj
@@ -215,11 +215,14 @@
 
 (defn get-type-name
   [type-pyobj]
-  (-> (pyobj/PyObject_GetAttrString type-pyobj "__name__")
-      (pyuni/PyUnicode_AsUTF8)
-      (jna/variable-byte-ptr->string)
-      ->kebab-case
-      keyword))
+  (println "@@@@ GETTING THE TYPE NAME @@@@")
+  (try
+    (-> (pyobj/PyObject_GetAttrString type-pyobj "__name__")
+        (pyuni/PyUnicode_AsUTF8)
+        (jna/variable-byte-ptr->string)
+        ->kebab-case
+        keyword)
+    (catch Throwable e :unnamed-thing)))
 
 
 (defn lookup-type-symbols
diff --git a/src/libpython_clj/python/interop.clj b/src/libpython_clj/python/interop.clj
index 55713a3..7c8d8df 100644
--- a/src/libpython_clj/python/interop.clj
+++ b/src/libpython_clj/python/interop.clj
@@ -82,6 +82,7 @@
 
 (defn add-module
   [modname]
+  (println "IN MY SUPER MODIFIED add-module")
   (with-gil
     (-> (libpy/PyImport_AddModule modname)
         (incref-wrap-pyobject))))

When I attempted to start it up again, it actually segfaulted and crashed the REPL!

squishy-kitty.core=>  (require '[libpython-clj.python :as py])                                [968/1916]nil
squishy-kitty.core=> (py/initialize!)
Jan 12, 2020 1:00:16 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Executing python initialize!
Jan 12, 2020 1:00:18 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Startup info detected: {:python-home "/usr", :lib-version "3.7", :libname "python3.7m", :java-library-path-addendum "/usr/lib"}
Jan 12, 2020 1:00:18 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Reference thread starting
Jan 12, 2020 1:00:18 PM clojure.tools.logging$eval1906$fn__1909 invoke
INFO: Library python3.7m found at [:system "python3.7m"]
@@@@ GETTING THE TYPE NAME @@@@
@@@@ GETTING THE TYPE NAME @@@@
@@@@ GETTING THE TYPE NAME @@@@
@@@@ GETTING THE TYPE NAME @@@@
[...repeat ~40x]
@@@@ GETTING THE TYPE NAME @@@@
@@@@ GETTING THE TYPE NAME @@@@
IN MY SUPER MODIFIED add-module
@@@@ GETTING THE TYPE NAME @@@@
@@@@ GETTING THE TYPE NAME @@@@
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xa3f1b11c, pid=9994, tid=0xa86fe460
#
# JRE version: OpenJDK Runtime Environment (8.0_152-b76) (build 1.8.0_152-b76)
# Java VM: OpenJDK Client VM (25.152-b76 mixed mode, Evaluation linux-aarch32 )
# Problematic frame:
# C  [libpython3.7m.so+0x1ff11c]  PyObject_GetAttrString+0x0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unli$ited" before starting Java again
#
# An error report file with more information is saved as:
# /home/pi/squishy-kitty/hs_err_pid9994.log
#
# If you would like to submit a bug report, please visit:
#   http://www.azulsystems.com/support/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
SocketException The transport's socket appears to have lost its connection to the nREPL server
        nrepl.transport/bencode/fn--9182/fn--9183 (transport.clj:108)
Subprocess failed
        nrepl.transport/bencode/fn--9182 (transport.clj:108)
        nrepl.transport/fn-transport/fn--9150 (transport.clj:55)
        clojure.core/binding-conveyor-fn/fn--5739 (core.clj:2030)
        java.util.concurrent.FutureTask.run (FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
        java.lang.Thread.run (Thread.java:748)
Bye for now!

Here's the additional error information it spit out.

Now, it's very possible that I incorrectly caught the exception, or returned the wrong thing, but I am at least pleased that I got the whole checkout local library modification business sorted out 😄

Happy to do some asynchronous programming with you if there are changes you want me to stick into my local code and run. Also happy to hop on a hangout/zoom/screenshare and do some synchronous collaboration as well.

I'm going to head to Microcenter and pick up some more hardware to replicate it. Let me know which of the following you have laying around so I know what else to pick up:

  • MicroUSB cable w/ 1Amp (at least) charger
  • Mini HDMI to HDMI cable
  • USB OTG (on to go) cable -- looks like micro USB to female USB-A

I can set up SSH for you, but you'll still need keyboard/monitor access when you get the hardware to connect it to your wifi before you can SSH to it.

Thanks again for the help! Looking forward to working with you!

ajpierce avatar Jan 12 '20 13:01 ajpierce

I have none of those things lying around :-). Well, probably a micro-usb cable here or there. As I said, I think the first move is to get the entire tech stack passing unit tests (tech.jna, tech.datatype) on 32 bit platforms; I am not sure it will just yet.

In any case, this will be a push of mine coming up because I think it is interesting/useful. Looking forward to this pathway.

cnuernber avatar Jan 12 '20 20:01 cnuernber

I'm having a similar issue with my raspberrypi. I ran the tests for tech.jna & tech.datatype, they do work on the pi.

pi@raspberrypi:~/dev/tech.jna $ lein test
...
Ran 2 tests containing 2 assertions.
0 failures, 0 errors.

pi@raspberrypi:~/dev/tech.datatype $ lein test
...
Ran 133 tests containing 2341 assertions.
0 failures, 0 errors.

(Although for tech.datatype i had to bring down the number of elements in test/tech/v2/datatype/simple_ops_perf_test.clj to avoid an out of memory exception).

When running python/initialize! it crashes the JVM The issue seems to be in jna.concrete.module:

C  [libpython3.7m.so+0x2115cc]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  com.sun.jna.Native.invokeInt(Lcom/sun/jna/Function;JI[Ljava/lang/Object;)I+0
j  com.sun.jna.Function.invoke([Ljava/lang/Object;Ljava/lang/Class;ZI)Ljava/lang/Object;+211
j  com.sun.jna.Function.invoke(Ljava/lang/reflect/Method;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object;+271
j  com.sun.jna.Function.invoke(Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object;+37
j  com.sun.jna.Function.invoke(Ljava/lang/Class;[Ljava/lang/Object;)Ljava/lang/Object;+7
j  libpython_clj.jna.concrete.module$PyModule_AddObject.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+157
j  libpython_clj.jna.concrete.module$PyModule_AddObject.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+9
j  libpython_clj.python.interop$register_type_BANG_$fn__32211.invoke()Ljava/lang/Object;+712
J 597 c1 clojure.lang.AFn.applyToHelper(Lclojure/lang/IFn;Lclojure/lang/ISeq;)Ljava/lang/Object; (3238 bytes) @ 0x6bdc0850 [0x6bdbcae0+0x00003d70]
J 599 c1 clojure.lang.AFn.applyTo(Lclojure/lang/ISeq;)Ljava/lang/Object; (12 bytes) @ 0x6bdc723c [0x6bdc7200+0x0000003c]
J 993 c1 clojure.core$apply.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (21 bytes) @ 0x6be8ce4c [0x6be8cd90+0x000000bc]
j  clojure.core$with_bindings_STAR_.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Lclojure/lang/ISeq;)Ljava/lang/Object;+13
j  clojure.core$with_bindings_STAR_.doInvoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+12
J 2411 c1 clojure.lang.RestFn.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (119 bytes) @ 0x6c09e008 [0x6c09df50+0x000000b8]
j  libpython_clj.python.interop$register_type_BANG_.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+301
j  libpython_clj.python.interop$register_type_BANG_.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+6
j  libpython_clj.python.interop$register_bridge_type_BANG_.invokeStatic(Lclojure/lang/ISeq;)Ljava/lang/Object;+524
j  libpython_clj.python.interop$register_bridge_type_BANG_.doInvoke(Ljava/lang/Object;)Ljava/lang/Object;+6
j  clojure.lang.RestFn.invoke()Ljava/lang/Object;+26
j  libpython_clj.python$initialize_BANG_.invokeStatic(Lclojure/lang/ISeq;)Ljava/lang/Object;+205
j  libpython_clj.python$initialize_BANG_.doInvoke(Ljava/lang/Object;)Ljava/lang/Object;+6

Where do you suggest to look at?

obohrer avatar Jul 22 '20 01:07 obohrer

I'm interested in a 32 bit compilation, too (armhf). I run Clojure on my Android smartphone via termux (spacemacs and lein-clojupyter are running fine).

ronnac avatar Dec 01 '20 14:12 ronnac

OK, if we want to go here I think it will be instructive for all of us.

The python bindings assume 64 bit architectures in a few places. The first place we will have to get exactly right is the struct definitions, things like:

I should mention here the way I got these things exactly right the first time was by using c++, compiling little programs and outputting offsetof calculations for various members.

Then it is going through the JNA bindings and very carefully making sure that things that are 64 bits on our desktop machines are 32 bits in the bindings. There is a tech.v3.jna/size-t* set of definitions for doing something like this.

This is all theoretically possible of course but I think it is probably like 2-4 weeks of work of someone who cares if we want to go there we should branch the library and come up with a documented development flow that everyone can take part in.

At the same time we can probably simplify libpython-clj a bit; for example most of the stubbed out JNA interfaces aren't used. This will probably mean there is a 32bit release of the library and 32-bit specific java/clojure sections.

A similar translation would be needed to use libpython with a python library built with reference tracing as that changes the member variables on each python object.

cnuernber avatar Dec 01 '20 15:12 cnuernber

I was probably too quick. Graal is not available for ARM32. Is that a dependency?

ronnac avatar Dec 02 '20 05:12 ronnac

clj-python/libpython-clj-2.00-beta-1 now supports 32 bit architectures via a ton of work to the ffi interface in dtype-next. If you have a moment, could you please test this out?

cnuernber avatar Mar 02 '21 15:03 cnuernber

Hi Chris,That is very good news!  I will try it out. Please give me some time, I have a lot of work these days.I try do it in the coming week.

ronnac avatar Mar 03 '21 05:03 ronnac