`measure` doesn't work under openjdk 16

[Bost]

It looks like the measure function doesn't work under openjdk 16:

$ java -version
openjdk version "16" 2021-03-16
OpenJDK Runtime Environment (build 16+36-Ubuntu-320.10)
OpenJDK 64-Bit Server VM (build 16+36-Ubuntu-320.10, mixed mode, sharing)
$ rlwrap /usr/local/bin/clojure -J-Djdk.attach.allowAttachSelf
Clojure 1.10.3
user=> (require '[clj-memory-meter.core :as mm])
nil
user=> (mm/measure "Hello, world!")
Execution error (InaccessibleObjectException) at java.lang.reflect.AccessibleObject/checkCanSetAccessible (AccessibleObject.java:357).
Unable to make field private final byte[] java.lang.String.value accessible: module java.base does not "opens java.lang" to unnamed module @3aa275a8
user=> 

Compare with openjdk 15

$ java -version
openjdk version "15" 2020-09-15
OpenJDK Runtime Environment (build 15+36-Ubuntu-1)
OpenJDK 64-Bit Server VM (build 15+36-Ubuntu-1, mixed mode, sharing)
$ rlwrap /usr/local/bin/clojure -J-Djdk.attach.allowAttachSelf
Clojure 1.10.3
user=> (require '[clj-memory-meter.core :as mm])
nil
user=> (mm/measure "Hello, world!")
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.github.jamm.MemoryMeter (file:/tmp/jamm15103091203300272530.jar) to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of org.github.jamm.MemoryMeter
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
"56 B"
user=> 

[kanwei]

Can confirm this also happens on OpenJDK15 on Mac:

Execution error (InaccessibleObjectException) at java.lang.reflect.AccessibleObject/checkCanSetAccessible (AccessibleObject.java:361).
Unable to make field jdk.internal.ref.PhantomCleanable jdk.internal.ref.PhantomCleanable.prev accessible: module java.base does not "opens jdk.internal.ref" to unnamed module @772e02bf

Only seems to happen on larger more complicated objects though.

[Bost]

The --illegal-access=... may be used as a workaround. See man java

$ java -version
openjdk version "16.0.1" 2021-04-20
OpenJDK Runtime Environment (build 16.0.1+9-Ubuntu-1)
OpenJDK 64-Bit Server VM (build 16.0.1+9-Ubuntu-1, mixed mode, sharing)
[13:30:58] bost@tuxedo /home/bost/dec/corona_cases (0) 
> rlwrap /usr/local/bin/clojure -J-Djdk.attach.allowAttachSelf -J--illegal-access=warn
OpenJDK 64-Bit Server VM warning: Option --illegal-access is deprecated and will be removed in a future release.
Clojure 1.10.3
(require '[clj-memory-meter.core :as mm])
nil
(mm/measure "Hello, world!")
WARNING: Illegal reflective access by org.github.jamm.MemoryMeter (file:/tmp/jamm1640437876966844469.jar) to field java.lang.String.value
"56 B"
user=> 

Anyway it looks like it's an upstream problem. See https://github.com/jbellis/jamm/issues

Edit: I'm experiencing huge performance degradation with this workaround, hmm :-( Edit2: Looks like the --illegal-access=permit does not degrades performance, only the --illegal-access=warn does it.

[boxxxie]

I'm using java-16-adoptopenjdk and --illegal-access=permit work around is not working for me

FIXED: i didn't notice the required -Djdk.attach.allowAttachSelf params needed.

working on adoptopenjdk 16 for me with workaround

[Cyrik]

doesn't work anymore on jdk 17 since --illegal-access=permit was removed. but for some reason it didn't work for me on jdk 11 either

[jumarko]

@Cyrik clj-memory-meter still works for me with JDK 17. I've been using -Djdk.attach.allowAttachSelf=true and --illegal-access=permit. See here for my deps.edn config: https://github.com/jumarko/clojure-experiments/blob/master/deps.edn#L101

Alternatively, you can try using JOL - see examples here: https://github.com/jumarko/clojure-experiments/blob/master/src/clojure_experiments/performance/memory.clj#L73

[Cyrik]

@jumarko thanks for the help. I just tried in your repo with jdk17, it works for seqs and maps, but breaks on most other things, like:

(mm/measure '(1))
(mm/measure :a)
(mm/measure [])

I've switched to adoptjdk 16 hs and it all works on for basic stuff, but brakes on omni-trace traces with:

; Execution error (InaccessibleObjectException) at java.lang.reflect.AccessibleObject/checkCanSetAccessible (AccessibleObject.java:357).
; Unable to make field public final jdk.javadoc.internal.tool.ElementsTable jdk.javadoc.internal.tool.DocEnvImpl.etable accessible: module jdk.javadoc does not "exports jdk.javadoc.internal.tool" to unnamed module @39a51841

I'll try JOL and play around more to see if i can find out what exactly it doesn't like about my traces, but I'm guessing its the metadata.

ok so because I couldn't let go: don't pass clojue namspaces into mm, it breaks. It can handle all other metadata though, including unpacked stacktraces. Also some laze-seqs break, couldn't pin down which ones exactly though, but here's an example that breaks without the (into []) :

  (mm/measure  (->> @i/workspace
                    :log
                    (map #(update-in (second %) [:meta :ns] (constantly nil)))
                    (into [])))
;; => "54.7 KiB"

[alexander-yakushev]

@jumarko Hello Juraj! Can you please confirm that it is indeed JDK17 that you are using? If so, which distribution? I've just tried installing Temurin (prev. AdoptOpenJDK) 17, and it says to me:

OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=permit; support was removed in 17.0

Therefore, I get the same errors as @Cyrik when trying to measure pretty much anything.

[jumarko]

@alexander-yakushev This is my JDK version:

java -version
openjdk version "17" 2021-09-14
OpenJDK Runtime Environment Homebrew (build 17+0)
OpenJDK 64-Bit Server VM Homebrew (build 17+0, mixed mode, sharing)

I've just quickly read @Cyrik 's post and it seems that it might work partially - at least it shouldn't completely break. I haven't done more extensive testing. Note that Cyrik said that it didn't work for him on JDK 11 before so that was main motivation for my post (because it suggested that they didn't get JVM opts right).

[KaliszAd]

The --illegal-access=permit doesn't work for me with Leiningen and OpenJDK 17 on Linux. It even doesn't show up in jps -lvm.

In the REPL, I get:

(mm/measure "BIxdrcz39IcK_zQZmj6p0o")
Execution error (InaccessibleObjectException) at java.lang.reflect.AccessibleObject/checkCanSetAccessible (AccessibleObject.java:354).
Unable to make field private final byte[] java.lang.String.value accessible: module java.base does not "opens java.lang" to unnamed module @202e401

-Djdk.attach.allowAttachSelf in jvm-opts shows up in jps.

[jumarko]

@alexander-yakushev @Cyrik I can confirm that it breaks for me too - when you try to measure ArrayList like described here: https://www.reddit.com/r/Clojure/comments/vmul4a/help_with_jvm_memory_optimization/ it fails with

1. Unhandled java.lang.reflect.InaccessibleObjectException
   Unable to make field transient java.lang.Object[] java.util.ArrayList.elementData accessible:
   module java.base does not "opens java.util" to unnamed module @1e4cdce5

     AccessibleObject.java:  387  java.lang.reflect.AccessibleObject/throwInaccessibleObjectException
     AccessibleObject.java:  363  java.lang.reflect.AccessibleObject/checkCanSetAccessible
     AccessibleObject.java:  311  java.lang.reflect.AccessibleObject/checkCanSetAccessible
                Field.java:  180  java.lang.reflect.Field/checkCanSetAccessible
                Field.java:  174  java.lang.reflect.Field/setAccessible
          MemoryMeter.java:  330  org.github.jamm.MemoryMeter/addFieldChildren
          MemoryMeter.java:  269  org.github.jamm.MemoryMeter/measureDeep
DirectMethodHandleAccessor.java:  104  jdk.internal.reflect.DirectMethodHandleAccessor/invoke
               Method.java:  578  java.lang.reflect.Method/invoke
            Reflector.java:  167  clojure.lang.Reflector/invokeMatchingMethod
            Reflector.java:  102  clojure.lang.Reflector/invokeInstanceMethod
                  core.clj:  144  clj-memory-meter.core/measure

[KaliszAd]

My colleague @pavel-klavik figured out what to add as JVM options to make clj-memory-meter run on the JVM from OpenJDK/ JRE 17.

;; needed since JVM 17 for clj-memory-meter
;; https://docs.oracle.com/javase/9/migrate/toc.htm#JSMIG-GUID-12F945EB-71D6-46AF-8C3D-D354FD0B1781
;; https://stackoverflow.com/questions/41265266/how-to-solve-inaccessibleobjectexception-unable-to-make-member-accessible-m/41265267#41265267
"--add-opens" "java.base/java.io=ALL-UNNAMED"
"--add-opens" "java.base/java.math=ALL-UNNAMED"
"--add-opens" "java.base/java.nio=ALL-UNNAMED"
"--add-opens" "java.base/java.nio.file.attribute=ALL-UNNAMED"
"--add-opens" "java.base/java.nio.charset=ALL-UNNAMED"
"--add-opens" "java.base/java.nio.channels=ALL-UNNAMED"
"--add-opens" "java.base/java.nio.channels.spi=ALL-UNNAMED"
"--add-opens" "java.base/java.lang=ALL-UNNAMED"
"--add-opens" "java.base/java.lang.invoke=ALL-UNNAMED"
"--add-opens" "java.base/java.lang.module=ALL-UNNAMED"
"--add-opens" "java.base/java.lang.ref=ALL-UNNAMED"
"--add-opens" "java.base/java.lang.reflect=ALL-UNNAMED"
"--add-opens" "java.base/java.util=ALL-UNNAMED"
"--add-opens" "java.base/java.util.jar=ALL-UNNAMED"
"--add-opens" "java.base/java.util.zip=ALL-UNNAMED"
"--add-opens" "java.base/java.util.regex=ALL-UNNAMED"
"--add-opens" "java.base/java.util.concurrent=ALL-UNNAMED"
"--add-opens" "java.base/java.util.concurrent.atomic=ALL-UNNAMED"
"--add-opens" "java.base/java.util.concurrent.locks=ALL-UNNAMED"
"--add-opens" "java.base/java.security=ALL-UNNAMED"
"--add-opens" "java.base/java.security.cert=ALL-UNNAMED"
"--add-opens" "java.base/java.security.spec=ALL-UNNAMED"
"--add-opens" "java.base/java.net=ALL-UNNAMED"
"--add-opens" "java.base/java.text=ALL-UNNAMED"
"--add-opens" "java.base/java.time=ALL-UNNAMED"
"--add-opens" "java.base/java.time.format=ALL-UNNAMED"
"--add-opens" "java.base/java.time.zone=ALL-UNNAMED"
"--add-opens" "java.base/java.time.temporal=ALL-UNNAMED"
"--add-opens" "java.base/javax.security.auth.x500=ALL-UNNAMED"
"--add-opens" "java.base/javax.net.ssl=ALL-UNNAMED"
"--add-opens" "java.base/javax.crypto.spec=ALL-UNNAMED"
"--add-opens" "java.base/jdk.internal.module=ALL-UNNAMED"
"--add-opens" "java.base/jdk.internal.loader=ALL-UNNAMED"
"--add-opens" "java.base/jdk.internal.perf=ALL-UNNAMED"
"--add-opens" "java.base/jdk.internal.ref=ALL-UNNAMED"
"--add-opens" "java.base/jdk.internal.util.jar=ALL-UNNAMED"
"--add-opens" "java.base/sun.invoke.util=ALL-UNNAMED"
"--add-opens" "java.base/sun.nio.ch=ALL-UNNAMED"
"--add-opens" "java.base/sun.nio.cs=ALL-UNNAMED"
"--add-opens" "java.base/sun.nio.fs=ALL-UNNAMED"
"--add-opens" "java.base/sun.net.www.protocol.jar=ALL-UNNAMED"
"--add-opens" "java.base/sun.security.util=ALL-UNNAMED"
"--add-opens" "java.base/sun.security.provider=ALL-UNNAMED"
"--add-opens" "java.base/sun.security.provider.certpath=ALL-UNNAMED"
"--add-opens" "java.base/sun.security.pkcs=ALL-UNNAMED"
"--add-opens" "java.base/sun.security.rsa=ALL-UNNAMED"
"--add-opens" "java.base/sun.security.ssl=ALL-UNNAMED"
"--add-opens" "java.base/sun.security.validator=ALL-UNNAMED"
"--add-opens" "java.base/sun.security.x509=ALL-UNNAMED"
"--add-opens" "java.base/sun.reflect.generics.factory=ALL-UNNAMED"
"--add-opens" "java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED"
"--add-opens" "java.base/sun.reflect.generics.repository=ALL-UNNAMED"
"--add-opens" "java.base/sun.reflect.generics.tree=ALL-UNNAMED"
"--add-opens" "java.base/sun.reflect.generics.scope=ALL-UNNAMED"
"--add-opens" "java.base/sun.util.calendar=ALL-UNNAMED"
"--add-opens" "java.base/sun.util.locale=ALL-UNNAMED"
"--add-opens" "java.logging/java.util.logging=ALL-UNNAMED"
"--add-opens" "jdk.crypto.ec/sun.security.ec=ALL-UNNAMED"

[alexander-yakushev]

@Bost @jumarko @Cyrik @KaliszAd Hey guys, can you try 0.2.0 with Java 17/18 (and without -add-opens)? I think I bought us some time with Unsafe, this won't last forever but should do for now.

[jumarko]

@Bost @jumarko @Cyrik @KaliszAd Hey guys, can you try 0.2.0 with Java 17/18 (and without -add-opens)? I think I bought us some time with Unsafe, this won't last forever but should do for now.

@alexander-yakushev Unfortunately, it's still broken for me: https://github.com/jumarko/clojure-experiments/blob/develop/src/clojure_experiments/performance/memory.clj#L330-L357

[jumarko]

Here's some vmInfo from me:

# JRE version: OpenJDK Runtime Environment (19.0+22) (build 19-ea+22-1598)
# Java VM: OpenJDK 64-Bit Server VM (19-ea+22-1598, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-amd64)

---------------  S U M M A R Y ------------

Command Line: -Xmx1g --enable-preview -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -Djol.tryWithSudo=true -Djdk.attach.allowAttachSelf=true --illegal-access=permit --add-modules=jdk.hotspot.agent --add-exports=jdk.hotspot.agent/sun.jvm.hotspot=ALL-UNNAMED --add-exports=jdk.hotspot.agent/sun.jvm.hotspot.oops=ALL-UNNAMED --add-exports=jdk.hotspot.agent/sun.jvm.hotspot.tools=ALL-UNNAMED --add-exports=jdk.hotspot.agent/sun.jvm.hotspot.runtime=ALL-UNNAMED --add-exports=jdk.hotspot.agent/sun.jvm.hotspot.classfile=ALL-UNNAMED -XX:+ShowCodeDetailsInExceptionMessages -Dclojure.basis=.cpcache/2037762702.basis clojure.main -m nrepl.cmdline --middleware [refactor-nrepl.middleware/wrap-refactor,cider.nrepl/cider-middleware]

Host: "MacBookPro15,1" x86_64 2600 MHz, 12 cores, 32G, Darwin 21.5.0, macOS 12.4 (21F79)
Time: Wed Jul 27 11:19:34 2022 CEST elapsed time: 24.031694 seconds (0d 0h 0m 24s)

[alexander-yakushev]

@jumarko EDIT: sorry, clicked the link now 😄

[alexander-yakushev]

Could you try 0.2.1-SNAPSHOT? I made JDK version detection hopefully more robust. If that doesn't work either, I have another idea.

[jumarko]

Could you try 0.2.1-SNAPSHOT? I made JDK version detection hopefully more robust. If that doesn't work either, I have another idea.

@alexander-yakushev it works now, many thanks!

And sorry for being confusing before - 0.2.0 worked with 17, just not with 19-ea.

[alexander-yakushev]

Cool, thank you! Pushed as 0.2.1. I'll wait for one more confirmation and then can finally close it.

[Bost]

It works on:

$ java -version
openjdk version "18" 2022-03-22
OpenJDK Runtime Environment (build 18+0-adhoc..source)
OpenJDK 64-Bit Server VM (build 18+0-adhoc..source, mixed mode, sharing)

(openjdk 17 not tested) Thanx for the fix!

[alexander-yakushev]

Thanks for confirming!