measure doesn't work under openjdk 17, despite -J-Djdk.attach.allowAttachSelf

[pmonks]

Possibly also related to Java 8 Lambda classes?

Steps to reproduce:

$ java -version
openjdk version "17.0.5" 2022-10-18
OpenJDK Runtime Environment Temurin-17.0.5+8 (build 17.0.5+8)
OpenJDK 64-Bit Server VM Temurin-17.0.5+8 (build 17.0.5+8, mixed mode, sharing)
$ clj -version
Clojure CLI version 1.11.1.1200
$ clj -J-Djdk.attach.allowAttachSelf -Sdeps '{:deps {org.spdx/java-spdx-library {:mvn/version "1.1.2"} com.clojure-goes-fast/clj-memory-meter {:mvn/version "0.2.1"}}}'
user=> (require '[clj-memory-meter.core :as mm])
nil
user=> (def apache-2 (org.spdx.library.model.license.LicenseInfoFactory/getListedLicenseById "Apache-2.0"))
#'user/apache-2
user=> (mm/measure apache-2)
Execution error (UnsupportedOperationException) at sun.misc.Unsafe/objectFieldOffset (Unsafe.java:645).
can't get field offset on a hidden class: private final org.spdx.library.model.ModelCollection org.spdx.library.model.ModelCollection$$Lambda$133/0x00000008010cf5b8.arg$1

Full Stack Trace:

user=> *e
#error {
 :cause "can't get field offset on a hidden class: private final org.spdx.library.model.ModelCollection org.spdx.library.model.ModelCollection$$Lambda$133/0x00000008010cf5b8.arg$1"
 :via
 [{:type java.lang.RuntimeException
   :message "java.lang.RuntimeException: java.lang.UnsupportedOperationException: can't get field offset on a hidden class: private final org.spdx.library.model.ModelCollection org.spdx.library.model.ModelCollection$$Lambda$133/0x00000008010cf5b8.arg$1"
   :at [org.github.jamm.MemoryMeterBase measureDeep "MemoryMeterBase.java" 160]}
  {:type java.lang.RuntimeException
   :message "java.lang.UnsupportedOperationException: can't get field offset on a hidden class: private final org.spdx.library.model.ModelCollection org.spdx.library.model.ModelCollection$$Lambda$133/0x00000008010cf5b8.arg$1"
   :at [org.github.jamm.MemoryMeterBase declaredClassFieldOffsets0 "MemoryMeterBase.java" 314]}
  {:type java.lang.UnsupportedOperationException
   :message "can't get field offset on a hidden class: private final org.spdx.library.model.ModelCollection org.spdx.library.model.ModelCollection$$Lambda$133/0x00000008010cf5b8.arg$1"
   :at [sun.misc.Unsafe objectFieldOffset "Unsafe.java" 645]}]
 :trace
 [[sun.misc.Unsafe objectFieldOffset "Unsafe.java" 645]
  [org.github.jamm.MemoryMeterBase declaredClassFieldOffsets0 "MemoryMeterBase.java" 310]
  [org.github.jamm.MemoryMeterBase access$100 "MemoryMeterBase.java" 16]
  [org.github.jamm.MemoryMeterBase$2 computeValue "MemoryMeterBase.java" 34]
  [org.github.jamm.MemoryMeterBase$2 computeValue "MemoryMeterBase.java" 30]
  [java.lang.ClassValue getFromHashMap "ClassValue.java" 228]
  [java.lang.ClassValue getFromBackup "ClassValue.java" 210]
  [java.lang.ClassValue get "ClassValue.java" 116]
  [org.github.jamm.MemoryMeterBase declaredClassFieldOffsets "MemoryMeterBase.java" 255]
  [org.github.jamm.MemoryMeterBase measureDeep "MemoryMeterBase.java" 141]
  [jdk.internal.reflect.NativeMethodAccessorImpl invoke0 "NativeMethodAccessorImpl.java" -2]
  [jdk.internal.reflect.NativeMethodAccessorImpl invoke "NativeMethodAccessorImpl.java" 77]
  [jdk.internal.reflect.DelegatingMethodAccessorImpl invoke "DelegatingMethodAccessorImpl.java" 43]
  [java.lang.reflect.Method invoke "Method.java" 568]
  [clojure.lang.Reflector invokeMatchingMethod "Reflector.java" 167]
  [clojure.lang.Reflector invokeInstanceMethod "Reflector.java" 102]
  [clj_memory_meter.core$measure invokeStatic "core.clj" 166]
  [clj_memory_meter.core$measure doInvoke "core.clj" 150]
  [clojure.lang.RestFn invoke "RestFn.java" 410]
  [user$eval202 invokeStatic "NO_SOURCE_FILE" 1]
  [user$eval202 invoke "NO_SOURCE_FILE" 1]
  [clojure.lang.Compiler eval "Compiler.java" 7181]
  [clojure.lang.Compiler eval "Compiler.java" 7136]
  [clojure.core$eval invokeStatic "core.clj" 3202]
  [clojure.core$eval invoke "core.clj" 3198]
  [clojure.main$repl$read_eval_print__9110$fn__9113 invoke "main.clj" 437]
  [clojure.main$repl$read_eval_print__9110 invoke "main.clj" 437]
  [clojure.main$repl$fn__9119 invoke "main.clj" 458]
  [clojure.main$repl invokeStatic "main.clj" 458]
  [clojure.main$repl_opt invokeStatic "main.clj" 522]
  [clojure.main$main invokeStatic "main.clj" 667]
  [clojure.main$main doInvoke "main.clj" 616]
  [clojure.lang.RestFn invoke "RestFn.java" 397]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.RestFn applyTo "RestFn.java" 132]
  [clojure.lang.Var applyTo "Var.java" 705]
  [clojure.main main "main.java" 40]]}

Environment: macOS Ventura (13.0.1) 2019 Intel MacBookPro

[alexander-yakushev]

Hi, Peter. Indeed, the implementation of clj-memory-meter on JDK17+ uses Unsafe, and that approach is quite brittle and will continue to show more and more shortcomings in the future. Unfortunately, we can't do much with that up until JDK-8249196 ships someday.

For now, I have pushed 0.2.2-SNAPSHOT that presses on the calculation skipping the errors of failed Unsafe access. I'm not sure if it's much of a relief, e.g. in this example of yours, it manages to count 136Kb while the real size is 216Kb 🤷.

[pmonks]

Thank you for the fast response @alexander-yakushev! I wondered if this was a JAMM / JVM issue but thought I'd track it here anyway, just in case.

[jsa-aerial]

As an FYI, I just hit this as well with Oracle JDK-17. Different error but likely same basic problem.

[alexander-yakushev]

Is it an actual exception thrown or just the warnings being printed while some result is still returned? Are you using 0.2.3?

[jsa-aerial]

it is an actual exception - basically a NPE. I will post the actual result when I get a chance tomorrow. Yes, it is 0.2.3 and it actually occurs during the require of the core namespace.

[jsa-aerial]

(deps '[[com.clojure-goes-fast/clj-memory-meter "0.2.3"]])
=> :success
(require '[clj-memory-meter.core :as mm])
=> "ERROR : Compiler$CompilerException, Cannot open <nil> as an InputStream."

[alexander-yakushev]

Could you please tell which OS you try it on and the output of java -version? Also, is there a stacktrace anywhere, or just this string? Does (pst) in the REPL print anything?

What does the function deps do exactly, is this something custom from your user.clj?

[jsa-aerial]

OS: ubuntu 22.04 $ uname -a Linux aerobio1 5.19.0-1022-gcp #24~22.04.1-Ubuntu SMP Sun Apr 23 09:51:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux $ ldd --version ldd (Ubuntu GLIBC 2.35-0ubuntu3.1) 2.35

java: java version "17.0.6" 2023-01-17 LTS

In this use case, I am using aerosaite and I don't return full stack traces. Hmmm, OTOH, I could connect via repl port and try via cider to get a full trace.

deps is a dynamic dependency loader. It's been used against all manner libs many with very complex dependencies - if it succeeds it has always provided the correct dependencies. I suppose memory-meter could be the exception, but seems very unlikely.

Re: pst and 'repl'. Hang on - does this thing require a "standard" clojure repl? If so, that is probably the problem here - Saite does not use that. I will plug in to the server via repl port to see if works from standard repl.

[alexander-yakushev]

The full stacktrace would really help if you can somehow extract that.

Also, meanwhile, could you please do the following:

1. Check the value of this in you REPL or what you have instead: @#'clj-memory-meter.core/extracted-jamm-jar. The value should be a path to temp file, e.g., /var/folders/zy/3gfxx9h1355dgvtkcyz2stc40000gn/T/jamm8743997171192180037.jar.
2. Navigate to that file in your terminal and verify that it exists.

[alexander-yakushev]

java version "17.0.6" 2023-01-17 LTS

Can you please confirm that you have a complete JDK installed and not just JRE?

[jsa-aerial]

since the core require blows up immediately, none of its resources appear to be loaded - actually I don't even see the namespace.

Yes, it is full JDK

I think this is probably due to not using typical repl middleware, but will verify by trying it via typical cider repl

[alexander-yakushev]

Please, do try with the regular REPL. However, clj-memory-meter does not rely on anything REPL-related being present (I think). I guess, getting the stacktrace is the only way to decipher this.