pulsar
pulsar copied to clipboard
Published
20 hours ago •
apache
apache
[Bug] Pulsar RocksDB metadata driver causes JVM to crash in unit tests
Search before asking
- [X] I searched in the issues and found nothing similar.
Version
https://github.com/apache/pulsar/commit/334c3a541f00c08aa2f378a6cc45f37cbeb54a5c
Minimal reproduce step
none
What did you expect to see?
no JVM crashes
What did you see instead?
There's a crash in calling org.rocksdb.RocksDB.get
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f2848a874dc, pid=4771, tid=12583
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.5+8 (17.0.5+8) (build 17.0.5+8)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.5+8 (17.0.5+8, mixed mode, sharing, tiered, compressed class ptrs, z gc, linux-amd64)
# Problematic frame:
# C [librocksdbjni15077744768266540673.so+0x2944dc] rocksdb_get_helper(JNIEnv_*, rocksdb::DB*, rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, _jbyteArray*, int, int)+0x1ac'.
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/runner/work/pulsar/pulsar/pulsar-metadata/core.4771)
#
# An error report file with more information is saved as:
# /home/runner/work/pulsar/pulsar/pulsar-metadata/hs_err_pid4771.log
#
# If you would like to submit a bug report, please visit:'.
# https://github.com/adoptium/adoptium-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Anything else?
It happened in org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClientTest.
https://ge.apache.org/s/x2u5wjemaiim6/failure?focused-exception-line=0-9#1
Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Here's another crash: https://gist.github.com/lhotari/00101de825a7858db17dc3f1d5d88291 .
This happened in https://github.com/apache/pulsar/actions/runs/4074068864/jobs/7018986238
Current thread (0x00007ff85924d1d0): JavaThread "-54-1" [_thread_in_native, id=8952, stack(0x00007ff6081ec000,0x00007ff6082ed000)]
Stack: [0x00007ff6081ec000,0x00007ff6082ed000], sp=0x00007ff6082ea6a8, free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C 0x00007ff8591df560
j org.rocksdb.RocksDB.get(JJ[BII)[B+0
j org.rocksdb.RocksDB.get(Lorg/rocksdb/ReadOptions;[B)[B+13
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.storeGet(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.get(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+97
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.readValueFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.lambda$refresh$13(Ljava/lang/String;Ljava/lang/String;Ljava/util/concurrent/CompletableFuture;)Ljava/util/concurrent/CompletableFuture;+10
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl$$Lambda$396+0x0000000800fe6f40.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+16
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.lambda$computeIfPresent$2([Ljava/util/concurrent/CompletableFuture;Ljava/util/function/BiFunction;Ljava/lang/Object;Ljava/util/concurrent/CompletableFuture;)Ljava/util/concurrent/CompletableFuture;+13
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView$$Lambda$397+0x0000000800fe75d0.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+13
j com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$remap$16(Z[Ljava/lang/Object;Ljava/util/function/BiFunction;Ljava/lang/Object;[J[ILjava/lang/Object;[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node;+254
j com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$400+0x0000000800fe5800.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+53
j java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object;+297 [email protected]
j com.github.benmanes.caffeine.cache.BoundedLocalCache.remap(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/BiFunction;[JZ)Ljava/lang/Object;+65
j com.github.benmanes.caffeine.cache.BoundedLocalCache.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;ZZZ)Ljava/lang/Object;+64
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.computeIfPresent(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/util/concurrent/CompletableFuture;+51
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.computeIfPresent(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object;+3
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.refresh(Ljava/lang/String;)V+24
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.accept(Lorg/apache/pulsar/metadata/api/Notification;)V+66
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$accept$7(Lorg/apache/pulsar/metadata/api/Notification;Lorg/apache/pulsar/metadata/cache/impl/MetadataCacheImpl;)V+9
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$399+0x0000000800fe7a40.accept(Ljava/lang/Object;)V+8
j java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V+35 [email protected]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.accept(Lorg/apache/pulsar/metadata/api/Notification;)V+229
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.accept(Ljava/lang/Object;)V+12
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$5(Lorg/apache/pulsar/metadata/api/Notification;Ljava/util/function/Consumer;)V+9
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$392+0x0000000800fe0d00.accept(Ljava/lang/Object;)V+8
j java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V+35 [email protected]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$6(Lorg/apache/pulsar/metadata/api/Notification;)Ljava/lang/Void;+17
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$391+0x0000000800fe0ad8.get()Ljava/lang/Object;+8
j java.util.concurrent.CompletableFuture$AsyncSupply.run()V+37 [email protected]
J 4672 c1 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007ff841db14ec [0x00007ff841db13c0+0x000000000000012c]
j java.util.concurrent.FutureTask.run()V+39 [email protected]
J 4671 c1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007ff841db0e24 [0x00007ff841db0b40+0x00000000000002e4]
J 5984 c1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x00007ff841fccbf4 [0x00007ff841fcc2a0+0x0000000000000954]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 [email protected]
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
j java.lang.Thread.run()V+11 [email protected]
v ~StubRoutines::call_stub
V [libjvm.so+0x822715] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x315
V [libjvm.so+0x823f0b] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x1cb
V [libjvm.so+0x8eda53] thread_entry(JavaThread*, JavaThread*)+0xa3
V [libjvm.so+0xe5e974] JavaThread::thread_main_inner()+0x184
V [libjvm.so+0xe62020] Thread::call_run()+0xc0
V [libjvm.so+0xc187e1] thread_native_entry(Thread*)+0xe1
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.rocksdb.RocksDB.get(JJ[BII)[B+0
j org.rocksdb.RocksDB.get(Lorg/rocksdb/ReadOptions;[B)[B+13
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.storeGet(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.get(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+97
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.readValueFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.lambda$refresh$13(Ljava/lang/String;Ljava/lang/String;Ljava/util/concurrent/CompletableFuture;)Ljava/util/concurrent/CompletableFuture;+10
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl$$Lambda$396+0x0000000800fe6f40.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+16
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.lambda$computeIfPresent$2([Ljava/util/concurrent/CompletableFuture;Ljava/util/function/BiFunction;Ljava/lang/Object;Ljava/util/concurrent/CompletableFuture;)Ljava/util/concurrent/CompletableFuture;+13
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView$$Lambda$397+0x0000000800fe75d0.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+13
j com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$remap$16(Z[Ljava/lang/Object;Ljava/util/function/BiFunction;Ljava/lang/Object;[J[ILjava/lang/Object;[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node;+254
j com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$400+0x0000000800fe5800.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+53
j java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object;+297 [email protected]
j com.github.benmanes.caffeine.cache.BoundedLocalCache.remap(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/BiFunction;[JZ)Ljava/lang/Object;+65
j com.github.benmanes.caffeine.cache.BoundedLocalCache.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;ZZZ)Ljava/lang/Object;+64
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.computeIfPresent(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/util/concurrent/CompletableFuture;+51
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.computeIfPresent(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object;+3
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.refresh(Ljava/lang/String;)V+24
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.accept(Lorg/apache/pulsar/metadata/api/Notification;)V+66
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$accept$7(Lorg/apache/pulsar/metadata/api/Notification;Lorg/apache/pulsar/metadata/cache/impl/MetadataCacheImpl;)V+9
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$399+0x0000000800fe7a40.accept(Ljava/lang/Object;)V+8
j java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V+35 [email protected]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.accept(Lorg/apache/pulsar/metadata/api/Notification;)V+229
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.accept(Ljava/lang/Object;)V+12
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$5(Lorg/apache/pulsar/metadata/api/Notification;Ljava/util/function/Consumer;)V+9
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$392+0x0000000800fe0d00.accept(Ljava/lang/Object;)V+8
j java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V+35 [email protected]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$6(Lorg/apache/pulsar/metadata/api/Notification;)Ljava/lang/Void;+17
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$391+0x0000000800fe0ad8.get()Ljava/lang/Object;+8
j java.util.concurrent.CompletableFuture$AsyncSupply.run()V+37 [email protected]
J 4672 c1 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007ff841db14ec [0x00007ff841db13c0+0x000000000000012c]
j java.util.concurrent.FutureTask.run()V+39 [email protected]
J 4671 c1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007ff841db0e24 [0x00007ff841db0b40+0x00000000000002e4]
J 5984 c1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x00007ff841fccbf4 [0x00007ff841fcc2a0+0x0000000000000954]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 [email protected]
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
j java.lang.Thread.run()V+11 [email protected]
v ~StubRoutines::call_stub
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Another one:
Current thread (0x00007f15cc118cc0): JavaThread "metadata-store-62-1" [_thread_in_native, id=3700, stack(0x00007f15d800a000,0x00007f15d810b000)]
Stack: [0x00007f15d800a000,0x00007f15d810b000], sp=0x00007f15d8108728, free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C 0x00007f15cc1805e0
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+11
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/Object;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+12
j com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache$$Lambda$362+0x0000000800fdefc8.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+9
j com.github.benmanes.caffeine.cache.LocalAsyncCache.lambda$get$2([Ljava/util/concurrent/CompletableFuture;Ljava/util/function/BiFunction;Ljava/lang/Object;Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture;+15
j com.github.benmanes.caffeine.cache.LocalAsyncCache$$Lambda$363+0x0000000800fdf658.apply(Ljava/lang/Object;)Ljava/lang/Object;+17
j com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14([Ljava/lang/Object;Ljava/util/function/Function;Ljava/lang/Object;[J[I[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node;+9
j com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$364+0x0000000800fdbc48.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+45
J 9622 c1 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007f15d9772594 [0x00007f15d9771f60+0x0000000000000634]
j com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/Function;[JZ)Ljava/lang/Object;+62
j com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(Ljava/lang/Object;Ljava/util/function/Function;ZZ)Ljava/lang/Object;+157
j com.github.benmanes.caffeine.cache.LocalAsyncCache.get(Ljava/lang/Object;Ljava/util/function/BiFunction;Z)Ljava/util/concurrent/CompletableFuture;+43
j com.github.benmanes.caffeine.cache.LocalAsyncCache.get(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/util/concurrent/CompletableFuture;+4
j com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.get(Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture;+16
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+69
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getWritableBookies()Ljava/util/concurrent/CompletableFuture;+12
J 8547 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.updatedBookies(Lorg/apache/pulsar/metadata/api/Notification;)V (243 bytes) @ 0x00007f15d96d9d0c [0x00007f15d96d9860+0x00000000000004ac]
J 8546 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$383+0x0000000800ff8c00.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007f15d9b9e474 [0x00007f15d9b9e3c0+0x00000000000000b4]
J 8556 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$5(Lorg/apache/pulsar/metadata/api/Notification;Ljava/util/function/Consumer;)V (55 bytes) @ 0x00007f15d997e7e4 [0x00007f15d997e6e0+0x0000000000000104]
J 8520 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$373+0x0000000800fe3a70.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007f15d97a4ee4 [0x00007f15d97a4d40+0x00000000000001a4]
J 8140 c1 java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V [email protected] (47 bytes) @ 0x00007f15da12092c [0x00007f15da120860+0x00000000000000cc]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$6(Lorg/apache/pulsar/metadata/api/Notification;)Ljava/lang/Void;+17
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$367+0x0000000800fe60e0.get()Ljava/lang/Object;+8
j java.util.concurrent.CompletableFuture$AsyncSupply.run()V+37 [email protected]
J 7012 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007f15e1250084 [0x00007f15e1250020+0x0000000000000064]
J 6984 c1 java.util.concurrent.FutureTask.run()V [email protected] (123 bytes) @ 0x00007f15d9edb1d4 [0x00007f15d9edaa20+0x00000000000007b4]
J 7031 c1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007f15d9ef74a4 [0x00007f15d9ef71c0+0x00000000000002e4]
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 [email protected]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 [email protected]
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
j java.lang.Thread.run()V+11 [email protected]
v ~StubRoutines::call_stub
V [libjvm.so+0x822715] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x315
V [libjvm.so+0x823f0b] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x1cb
V [libjvm.so+0x8eda53] thread_entry(JavaThread*, JavaThread*)+0xa3
V [libjvm.so+0xe5e974] JavaThread::thread_main_inner()+0x184
V [libjvm.so+0xe62020] Thread::call_run()+0xc0
V [libjvm.so+0xc187e1] thread_native_entry(Thread*)+0xe1
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+11
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/Object;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+12
j com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache$$Lambda$362+0x0000000800fdefc8.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+9
j com.github.benmanes.caffeine.cache.LocalAsyncCache.lambda$get$2([Ljava/util/concurrent/CompletableFuture;Ljava/util/function/BiFunction;Ljava/lang/Object;Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture;+15
j com.github.benmanes.caffeine.cache.LocalAsyncCache$$Lambda$363+0x0000000800fdf658.apply(Ljava/lang/Object;)Ljava/lang/Object;+17
j com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14([Ljava/lang/Object;Ljava/util/function/Function;Ljava/lang/Object;[J[I[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node;+9
j com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$364+0x0000000800fdbc48.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+45
J 9622 c1 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007f15d9772594 [0x00007f15d9771f60+0x0000000000000634]
j com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/Function;[JZ)Ljava/lang/Object;+62
j com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(Ljava/lang/Object;Ljava/util/function/Function;ZZ)Ljava/lang/Object;+157
j com.github.benmanes.caffeine.cache.LocalAsyncCache.get(Ljava/lang/Object;Ljava/util/function/BiFunction;Z)Ljava/util/concurrent/CompletableFuture;+43
j com.github.benmanes.caffeine.cache.LocalAsyncCache.get(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/util/concurrent/CompletableFuture;+4
j com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.get(Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture;+16
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+69
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getWritableBookies()Ljava/util/concurrent/CompletableFuture;+12
J 8547 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.updatedBookies(Lorg/apache/pulsar/metadata/api/Notification;)V (243 bytes) @ 0x00007f15d96d9d0c [0x00007f15d96d9860+0x00000000000004ac]
J 8546 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$383+0x0000000800ff8c00.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007f15d9b9e474 [0x00007f15d9b9e3c0+0x00000000000000b4]
J 8556 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$5(Lorg/apache/pulsar/metadata/api/Notification;Ljava/util/function/Consumer;)V (55 bytes) @ 0x00007f15d997e7e4 [0x00007f15d997e6e0+0x0000000000000104]
J 8520 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$373+0x0000000800fe3a70.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007f15d97a4ee4 [0x00007f15d97a4d40+0x00000000000001a4]
J 8140 c1 java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V [email protected] (47 bytes) @ 0x00007f15da12092c [0x00007f15da120860+0x00000000000000cc]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$6(Lorg/apache/pulsar/metadata/api/Notification;)Ljava/lang/Void;+17
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$367+0x0000000800fe60e0.get()Ljava/lang/Object;+8
j java.util.concurrent.CompletableFuture$AsyncSupply.run()V+37 [email protected]
J 7012 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007f15e1250084 [0x00007f15e1250020+0x0000000000000064]
J 6984 c1 java.util.concurrent.FutureTask.run()V [email protected] (123 bytes) @ 0x00007f15d9edb1d4 [0x00007f15d9edaa20+0x00000000000007b4]
J 7031 c1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007f15d9ef74a4 [0x00007f15d9ef71c0+0x00000000000002e4]
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 [email protected]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 [email protected]
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
j java.lang.Thread.run()V+11 [email protected]
v ~StubRoutines::call_stub
in https://github.com/apache/pulsar/actions/runs/4604751972/jobs/8136228746?pr=19945#step:11:1553 complete file https://gist.github.com/lhotari/b70ad90c74e3af27e44fc630ed4d6608
Another one:
Current thread (0x00007f47382f4210): JavaThread "-225-1" [_thread_in_native, id=12582, stack(0x00007f470f4f4000,0x00007f470f5f5000)]
Stack: [0x00007f470f4f4000,0x00007f470f5f5000], sp=0x00007f470f5f25d8, free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni948785792524234568.so+0x2b8844] rocksdb::StackableDB::DefaultColumnFamily() const+0x4
C [librocksdbjni948785792524234568.so+0x2aece0] Java_org_rocksdb_RocksDB_iterator__JJ+0x30
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+184
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+11
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/Object;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+12
j com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache$$Lambda$439+0x00000008013f64e8.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+9
j com.github.benmanes.caffeine.cache.LocalAsyncCache.lambda$get$2([Ljava/util/concurrent/CompletableFuture;Ljava/util/function/BiFunction;Ljava/lang/Object;Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture;+15
j com.github.benmanes.caffeine.cache.LocalAsyncCache$$Lambda$440+0x00000008013f6b78.apply(Ljava/lang/Object;)Ljava/lang/Object;+17
j com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14([Ljava/lang/Object;Ljava/util/function/Function;Ljava/lang/Object;[J[I[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node;+9
j com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$441+0x00000008013f77f0.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+45
J 14898 c1 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007f494aaf0474 [0x00007f494aaf01e0+0x0000000000000294]
j com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/Function;[JZ)Ljava/lang/Object;+62
j com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(Ljava/lang/Object;Ljava/util/function/Function;ZZ)Ljava/lang/Object;+157
j com.github.benmanes.caffeine.cache.LocalAsyncCache.get(Ljava/lang/Object;Ljava/util/function/BiFunction;Z)Ljava/util/concurrent/CompletableFuture;+43
J 13721 c1 com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.get(Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture; (20 bytes) @ 0x00007f494a8cdec4 [0x00007f494a8cdde0+0x00000000000000e4]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+177
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getWritableBookies()Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.updatedBookies(Lorg/apache/pulsar/metadata/api/Notification;)V+119
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$932+0x00000008017cd0d8.accept(Ljava/lang/Object;)V+8
J 13724 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$5(Lorg/apache/pulsar/metadata/api/Notification;Ljava/util/function/Consumer;)V (55 bytes) @ 0x00007f494a8d2c24 [0x00007f494a8d2bc0+0x0000000000000064]
J 13723 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$838+0x000000080173a478.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007f494a8d280c [0x00007f494a8d2740+0x00000000000000cc]
J 13722 c1 java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V [email protected] (47 bytes) @ 0x00007f494a8cd42c [0x00007f494a8cd360+0x00000000000000cc]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$6(Lorg/apache/pulsar/metadata/api/Notification;)Ljava/lang/Void;+17
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$836+0x000000080173a018.get()Ljava/lang/Object;+8
J 13764 c1 java.util.concurrent.CompletableFuture$AsyncSupply.run()V [email protected] (61 bytes) @ 0x00007f494a8dc86c [0x00007f494a8dc660+0x000000000000020c]
J 8546 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007f4951310804 [0x00007f49513107a0+0x0000000000000064]
J 11474 c1 java.util.concurrent.FutureTask.run()V [email protected] (123 bytes) @ 0x00007f494a581054 [0x00007f494a580ea0+0x00000000000001b4]
J 10010 c1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007f494a38a8f4 [0x00007f494a38a640+0x00000000000002b4]
J 11448 c1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x00007f494a56d3dc [0x00007f494a56d160+0x000000000000027c]
J 14601 c1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V [email protected] (9 bytes) @ 0x00007f494aa757dc [0x00007f494aa75740+0x000000000000009c]
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
J 14165 c1 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x00007f494a9a82a4 [0x00007f494a9a8140+0x0000000000000164]
v ~StubRoutines::call_stub
V [libjvm.so+0x825045] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x315
V [libjvm.so+0x82683b] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x1cb
V [libjvm.so+0x8f14f3] thread_entry(JavaThread*, JavaThread*)+0xa3
V [libjvm.so+0xe62564] JavaThread::thread_main_inner()+0x184
V [libjvm.so+0xe65c10] Thread::call_run()+0xc0
V [libjvm.so+0xc1aa21] thread_native_entry(Thread*)+0xe1
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+184
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+11
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/Object;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+12
j com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache$$Lambda$439+0x00000008013f64e8.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+9
j com.github.benmanes.caffeine.cache.LocalAsyncCache.lambda$get$2([Ljava/util/concurrent/CompletableFuture;Ljava/util/function/BiFunction;Ljava/lang/Object;Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture;+15
j com.github.benmanes.caffeine.cache.LocalAsyncCache$$Lambda$440+0x00000008013f6b78.apply(Ljava/lang/Object;)Ljava/lang/Object;+17
j com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14([Ljava/lang/Object;Ljava/util/function/Function;Ljava/lang/Object;[J[I[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node;+9
j com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$441+0x00000008013f77f0.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+45
J 14898 c1 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007f494aaf0474 [0x00007f494aaf01e0+0x0000000000000294]
j com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/Function;[JZ)Ljava/lang/Object;+62
j com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(Ljava/lang/Object;Ljava/util/function/Function;ZZ)Ljava/lang/Object;+157
j com.github.benmanes.caffeine.cache.LocalAsyncCache.get(Ljava/lang/Object;Ljava/util/function/BiFunction;Z)Ljava/util/concurrent/CompletableFuture;+43
J 13721 c1 com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.get(Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture; (20 bytes) @ 0x00007f494a8cdec4 [0x00007f494a8cdde0+0x00000000000000e4]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+177
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getWritableBookies()Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.updatedBookies(Lorg/apache/pulsar/metadata/api/Notification;)V+119
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$932+0x00000008017cd0d8.accept(Ljava/lang/Object;)V+8
J 13724 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$5(Lorg/apache/pulsar/metadata/api/Notification;Ljava/util/function/Consumer;)V (55 bytes) @ 0x00007f494a8d2c24 [0x00007f494a8d2bc0+0x0000000000000064]
J 13723 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$838+0x000000080173a478.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007f494a8d280c [0x00007f494a8d2740+0x00000000000000cc]
J 13722 c1 java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V [email protected] (47 bytes) @ 0x00007f494a8cd42c [0x00007f494a8cd360+0x00000000000000cc]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$6(Lorg/apache/pulsar/metadata/api/Notification;)Ljava/lang/Void;+17
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$836+0x000000080173a018.get()Ljava/lang/Object;+8
J 13764 c1 java.util.concurrent.CompletableFuture$AsyncSupply.run()V [email protected] (61 bytes) @ 0x00007f494a8dc86c [0x00007f494a8dc660+0x000000000000020c]
J 8546 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007f4951310804 [0x00007f49513107a0+0x0000000000000064]
J 11474 c1 java.util.concurrent.FutureTask.run()V [email protected] (123 bytes) @ 0x00007f494a581054 [0x00007f494a580ea0+0x00000000000001b4]
J 10010 c1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007f494a38a8f4 [0x00007f494a38a640+0x00000000000002b4]
J 11448 c1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x00007f494a56d3dc [0x00007f494a56d160+0x000000000000027c]
J 14601 c1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V [email protected] (9 bytes) @ 0x00007f494aa757dc [0x00007f494aa75740+0x000000000000009c]
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
J 14165 c1 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x00007f494a9a82a4 [0x00007f494a9a8140+0x0000000000000164]
v ~StubRoutines::call_stub
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000
full file: https://gist.github.com/lhotari/6f33d841b7aa41f0d602d6494ed585b6 in https://github.com/apache/pulsar/actions/runs/5113478831/jobs/9193024272#step:10:1941
Recent crash: https://gist.github.com/lhotari/6c66d0dc55a5104d4364cdbafe26d2e9
Stack: [0x00007fe85af00000,0x00007fe85b000000], sp=0x00007fe85affd4b0, free space=1013k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni6450051497838230364.so+0x2aece0] Java_org_rocksdb_RocksDB_iterator__JJ+0x30
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/Object;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+13
j com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache$$Lambda$373+0x00007fe874410fa0.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+9
j com.github.benmanes.caffeine.cache.LocalAsyncCache.lambda$get$2([Ljava/util/concurrent/CompletableFuture;Ljava/util/function/BiFunction;Ljava/lang/Object;Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture;+15
j com.github.benmanes.caffeine.cache.LocalAsyncCache$$Lambda$374+0x00007fe874411630.apply(Ljava/lang/Object;)Ljava/lang/Object;+17
j com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14([Ljava/lang/Object;Ljava/util/function/Function;Ljava/lang/Object;[J[I[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node;+9
j com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$375+0x00007fe8744122a8.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+45
J 9835 c1 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007feada176494 [0x00007feada175e60+0x0000000000000634]
j com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/Function;[JZ)Ljava/lang/Object;+62
j com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(Ljava/lang/Object;Ljava/util/function/Function;ZZ)Ljava/lang/Object;+157
j com.github.benmanes.caffeine.cache.LocalAsyncCache.get(Ljava/lang/Object;Ljava/util/function/BiFunction;Z)Ljava/util/concurrent/CompletableFuture;+43
J 10126 c1 com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.get(Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture; (20 bytes) @ 0x00007feada20d98c [0x00007feada20d340+0x000000000000064c]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+69
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getBookiesThenFreshCache(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+54
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getWritableBookies()Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.lambda$updatedBookies$26(Lorg/apache/pulsar/metadata/api/Notification;Lorg/apache/bookkeeper/net/BookieId;Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+355
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$445+0x00007fe8745008d0.get()Ljava/lang/Object;+16
j org.apache.pulsar.common.util.FutureUtil$Sequencer.sequential(Ljava/util/function/Supplier;)Ljava/util/concurrent/CompletableFuture;+72
J 9837 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.updatedBookies(Lorg/apache/pulsar/metadata/api/Notification;)V (146 bytes) @ 0x00007feada1815e4 [0x00007feada1812c0+0x0000000000000324]
J 8862 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$395+0x00007fe87442c230.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007fead96805f4 [0x00007fead9680540+0x00000000000000b4]
J 8829 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$5(Lorg/apache/pulsar/metadata/api/Notification;Ljava/util/function/Consumer;)V (57 bytes) @ 0x00007fead99e29e4 [0x00007fead99e28e0+0x0000000000000104]
J 8828 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$381+0x00007fe874419180.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007fead9a202e4 [0x00007fead9a20140+0x00000000000001a4]
J 8617 c1 java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V [email protected] (47 bytes) @ 0x00007fead98d62ac [0x00007fead98d61e0+0x00000000000000cc]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$6(Lorg/apache/pulsar/metadata/api/Notification;)Ljava/lang/Void;+17
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$378+0x00007fe8744181f8.get()Ljava/lang/Object;+8
j java.util.concurrent.CompletableFuture$AsyncSupply.run()V+37 [email protected]
J 6908 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007feae1156104 [0x00007feae11560a0+0x0000000000000064]
J 7466 c1 java.util.concurrent.FutureTask.run()V [email protected] (123 bytes) @ 0x00007fead9f7da2c [0x00007fead9f7d220+0x000000000000080c]
J 6998 c1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007fead9e8850c [0x00007fead9e88140+0x00000000000003cc]
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 [email protected]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 [email protected]
J 10156 c1 io.netty.util.concurrent.FastThreadLocalRunnable.run()V (22 bytes) @ 0x00007feada21c90c [0x00007feada21c7e0+0x000000000000012c]
J 9973 c1 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x00007feada1b1624 [0x00007feada1b14c0+0x0000000000000164]
v ~StubRoutines::call_stub
V [libjvm.so+0x827355] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x315
V [libjvm.so+0x828b4b] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x1cb
V [libjvm.so+0x8f3ae3] thread_entry(JavaThread*, JavaThread*)+0xa3
V [libjvm.so+0xe65584] JavaThread::thread_main_inner()+0x184
V [libjvm.so+0xe68d40] Thread::call_run()+0xc0
V [libjvm.so+0xc1e681] thread_native_entry(Thread*)+0xe1
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/Object;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+13
j com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache$$Lambda$373+0x00007fe874410fa0.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+9
j com.github.benmanes.caffeine.cache.LocalAsyncCache.lambda$get$2([Ljava/util/concurrent/CompletableFuture;Ljava/util/function/BiFunction;Ljava/lang/Object;Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture;+15
j com.github.benmanes.caffeine.cache.LocalAsyncCache$$Lambda$374+0x00007fe874411630.apply(Ljava/lang/Object;)Ljava/lang/Object;+17
j com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14([Ljava/lang/Object;Ljava/util/function/Function;Ljava/lang/Object;[J[I[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node;+9
j com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$375+0x00007fe8744122a8.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+45
J 9835 c1 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007feada176494 [0x00007feada175e60+0x0000000000000634]
j com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/Function;[JZ)Ljava/lang/Object;+62
j com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(Ljava/lang/Object;Ljava/util/function/Function;ZZ)Ljava/lang/Object;+157
j com.github.benmanes.caffeine.cache.LocalAsyncCache.get(Ljava/lang/Object;Ljava/util/function/BiFunction;Z)Ljava/util/concurrent/CompletableFuture;+43
J 10126 c1 com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.get(Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture; (20 bytes) @ 0x00007feada20d98c [0x00007feada20d340+0x000000000000064c]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+69
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getBookiesThenFreshCache(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+54
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getWritableBookies()Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.lambda$updatedBookies$26(Lorg/apache/pulsar/metadata/api/Notification;Lorg/apache/bookkeeper/net/BookieId;Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+355
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$445+0x00007fe8745008d0.get()Ljava/lang/Object;+16
j org.apache.pulsar.common.util.FutureUtil$Sequencer.sequential(Ljava/util/function/Supplier;)Ljava/util/concurrent/CompletableFuture;+72
J 9837 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.updatedBookies(Lorg/apache/pulsar/metadata/api/Notification;)V (146 bytes) @ 0x00007feada1815e4 [0x00007feada1812c0+0x0000000000000324]
J 8862 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$395+0x00007fe87442c230.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007fead96805f4 [0x00007fead9680540+0x00000000000000b4]
J 8829 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$5(Lorg/apache/pulsar/metadata/api/Notification;Ljava/util/function/Consumer;)V (57 bytes) @ 0x00007fead99e29e4 [0x00007fead99e28e0+0x0000000000000104]
J 8828 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$381+0x00007fe874419180.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007fead9a202e4 [0x00007fead9a20140+0x00000000000001a4]
J 8617 c1 java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V [email protected] (47 bytes) @ 0x00007fead98d62ac [0x00007fead98d61e0+0x00000000000000cc]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$6(Lorg/apache/pulsar/metadata/api/Notification;)Ljava/lang/Void;+17
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$378+0x00007fe8744181f8.get()Ljava/lang/Object;+8
j java.util.concurrent.CompletableFuture$AsyncSupply.run()V+37 [email protected]
J 6908 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007feae1156104 [0x00007feae11560a0+0x0000000000000064]
J 7466 c1 java.util.concurrent.FutureTask.run()V [email protected] (123 bytes) @ 0x00007fead9f7da2c [0x00007fead9f7d220+0x000000000000080c]
J 6998 c1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007fead9e8850c [0x00007fead9e88140+0x00000000000003cc]
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 [email protected]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 [email protected]
J 10156 c1 io.netty.util.concurrent.FastThreadLocalRunnable.run()V (22 bytes) @ 0x00007feada21c90c [0x00007feada21c7e0+0x000000000000012c]
J 9973 c1 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x00007feada1b1624 [0x00007feada1b14c0+0x0000000000000164]
v ~StubRoutines::call_stub
Another recent crash: https://gist.github.com/lhotari/9dc43fae65b12e9b812cdefc845ee96d
Stack: [0x00007f0b84d00000,0x00007f0b84e00000], sp=0x00007f0b84dfd3b8, free space=1012k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni10071231183631223410.so+0x2adff0] rocksdb_iterator_helper(rocksdb::DB*, rocksdb::ReadOptions, rocksdb::ColumnFamilyHandle*)+0x60
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+12
J 16830 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/Object;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture; (23 bytes) @ 0x00007f0d962b6c54 [0x00007f0d962b6bc0+0x0000000000000094]
J 17198 c1 com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache$$Lambda+0x00007f0b303df558.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (15 bytes) @ 0x00007f0d9634500c [0x00007f0d96344f40+0x00000000000000cc]
J 17197 c1 com.github.benmanes.caffeine.cache.LocalAsyncCache.lambda$get$2([Ljava/util/concurrent/CompletableFuture;Ljava/util/function/BiFunction;Ljava/lang/Object;Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture; (34 bytes) @ 0x00007f0d963448b4 [0x00007f0d963447e0+0x00000000000000d4]
J 17196 c1 com.github.benmanes.caffeine.cache.LocalAsyncCache$$Lambda+0x00007f0b303dfbd0.apply(Ljava/lang/Object;)Ljava/lang/Object; (23 bytes) @ 0x00007f0d963443bc [0x00007f0d963442c0+0x00000000000000fc]
J 17195 c1 com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14([Ljava/lang/Object;Ljava/util/function/Function;Ljava/lang/Object;[J[I[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node; (355 bytes) @ 0x00007f0d96341d54 [0x00007f0d96341ca0+0x00000000000000b4]
J 17194 c1 com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda+0x00007f0b303e4000.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (49 bytes) @ 0x00007f0d963415c4 [0x00007f0d96341440+0x0000000000000184]
J 16581 c1 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007f0d9625fd44 [0x00007f0d9625f700+0x0000000000000644]
J 17044 c1 com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/Function;[JZ)Ljava/lang/Object; (276 bytes) @ 0x00007f0d96304d3c [0x00007f0d96304820+0x000000000000051c]
J 16208 c1 com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(Ljava/lang/Object;Ljava/util/function/Function;ZZ)Ljava/lang/Object; (161 bytes) @ 0x00007f0d9617ab9c [0x00007f0d9617a600+0x000000000000059c]
J 16309 c1 com.github.benmanes.caffeine.cache.LocalAsyncCache.get(Ljava/lang/Object;Ljava/util/function/BiFunction;Z)Ljava/util/concurrent/CompletableFuture; (77 bytes) @ 0x00007f0d961b00fc [0x00007f0d961afee0+0x000000000000021c]
J 16951 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (80 bytes) @ 0x00007f0d962e308c [0x00007f0d962e2ee0+0x00000000000001ac]
J 17345 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getBookiesThenFreshCache(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (91 bytes) @ 0x00007f0d963a032c [0x00007f0d963a0220+0x000000000000010c]
J 16922 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getWritableBookies()Ljava/util/concurrent/CompletableFuture; (21 bytes) @ 0x00007f0d962d2d34 [0x00007f0d962d2cc0+0x0000000000000074]
J 17272 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.lambda$updatedBookies$26(Lorg/apache/pulsar/metadata/api/Notification;Lorg/apache/bookkeeper/net/BookieId;Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (396 bytes) @ 0x00007f0d9637a6d4 [0x00007f0d96379ee0+0x00000000000007f4]
J 17271 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda+0x00007f0b308e0000.get()Ljava/lang/Object; (20 bytes) @ 0x00007f0d963797e4 [0x00007f0d96379740+0x00000000000000a4]
j org.apache.pulsar.common.util.FutureUtil$Sequencer.lambda$sequential$2(Ljava/util/function/Supplier;Ljava/lang/Object;)Ljava/util/concurrent/CompletionStage;+5
J 17336 c1 org.apache.pulsar.common.util.FutureUtil$Sequencer$$Lambda+0x00007f0b308e1c00.apply(Ljava/lang/Object;)Ljava/lang/Object; (9 bytes) @ 0x00007f0d9639e034 [0x00007f0d9639dfc0+0x0000000000000074]
J 16918 c1 java.util.concurrent.CompletableFuture$UniCompose.tryFire(I)Ljava/util/concurrent/CompletableFuture; [email protected] (199 bytes) @ 0x00007f0d962cfcbc [0x00007f0d962cfa20+0x000000000000029c]
J 14239 c1 java.util.concurrent.CompletableFuture.postComplete()V [email protected] (94 bytes) @ 0x00007f0d95e664ac [0x00007f0d95e654e0+0x0000000000000fcc]
J 16897 c2 java.util.concurrent.CompletableFuture.postFire(Ljava/util/concurrent/CompletableFuture;I)Ljava/util/concurrent/CompletableFuture; [email protected] (69 bytes) @ 0x00007f0d9c933860 [0x00007f0d9c9337c0+0x00000000000000a0]
J 16918 c1 java.util.concurrent.CompletableFuture$UniCompose.tryFire(I)Ljava/util/concurrent/CompletableFuture; [email protected] (199 bytes) @ 0x00007f0d962d0364 [0x00007f0d962cfa20+0x0000000000000944]
J 17221 c1 java.util.concurrent.CompletableFuture$Completion.run()V [email protected] (7 bytes) @ 0x00007f0d9634cfb4 [0x00007f0d9634cf40+0x0000000000000074]
J 17053 c1 java.lang.Thread.run()V [email protected] (23 bytes) @ 0x00007f0d9630a29c [0x00007f0d9630a1c0+0x00000000000000dc]
v ~StubRoutines::call_stub 0x00007f0d9bd38cbf
V [libjvm.so+0x8ffb45] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x2e5
V [libjvm.so+0x9014c2] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x1d2
V [libjvm.so+0x9d4dbe] thread_entry(JavaThread*, JavaThread*)+0x8e
V [libjvm.so+0x9175e0] JavaThread::thread_main_inner()+0x1e0
V [libjvm.so+0xf5b8e8] Thread::call_run()+0xa8
V [libjvm.so+0xcea59a] thread_native_entry(Thread*)+0xda
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture;+12
Yet another crash: https://gist.github.com/lhotari/177f186d9f73297fbfbce8b026824365 in https://github.com/apache/pulsar/actions/runs/7219125983/job/19669812711?pr=21723#step:11:295
Stack: [0x00007f95c27ff000,0x00007f95c28ff000], sp=0x00007f95c28fcfd0, free space=1015k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni17736003842798679341.so+0x2b3ff9] rocksdb_get_helper(JNIEnv_*, rocksdb::DB*, rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, _jbyteArray*, int, int)+0x199
J 21109 org.rocksdb.RocksDB.get(JJ[BII)[B (0 bytes) @ 0x00007f97d9460ab4 [0x00007f97d94609c0+0x00000000000000f4]
J 21083 c1 org.rocksdb.RocksDB.get(Lorg/rocksdb/ReadOptions;[B)[B (17 bytes) @ 0x00007f97d1b5886c [0x00007f97d1b587c0+0x00000000000000ac]
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.storeGet(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+186
J 20656 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore.get(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (260 bytes) @ 0x00007f97d21b312c [0x00007f97d21b1d00+0x000000000000142c]
J 20960 c1 org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.readValueFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (38 bytes) @ 0x00007f97d20e662c [0x00007f97d20e64c0+0x000000000000016c]
J 22917 c2 com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$remap$16(Z[Ljava/lang/Object;Ljava/util/function/BiFunction;Ljava/lang/Object;[J[ILjava/lang/Object;[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node; (466 bytes) @ 0x00007f97d95b2688 [0x00007f97d95b2320+0x0000000000000368]
J 23900 c2 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007f97d98632bc [0x00007f97d9862880+0x0000000000000a3c]
J 22882 c2 org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.refresh(Ljava/lang/String;)V (36 bytes) @ 0x00007f97d95a2f98 [0x00007f97d95a29a0+0x00000000000005f8]
J 25254 c2 org.apache.pulsar.metadata.impl.AbstractMetadataStore.accept(Lorg/apache/pulsar/metadata/api/Notification;)V (405 bytes) @ 0x00007f97d9496698 [0x00007f97d94960e0+0x00000000000005b8]
J 23122 c2 org.apache.pulsar.metadata.impl.AbstractMetadataStore.accept(Ljava/lang/Object;)V (22 bytes) @ 0x00007f97d9617678 [0x00007f97d9617620+0x0000000000000058]
J 22682 c2 java.util.concurrent.CompletableFuture$AsyncSupply.run()V [email protected] (61 bytes) @ 0x00007f97d9571c44 [0x00007f97d9571a40+0x0000000000000204]
J 20533 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007f97d93d1f84 [0x00007f97d93d1f20+0x0000000000000064]
J 20537 c2 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007f97d93d2a74 [0x00007f97d93d2960+0x0000000000000114]
J 23462 c2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x00007f97d975b360 [0x00007f97d975b140+0x0000000000000220]
J 14766 c1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V [email protected] (9 bytes) @ 0x00007f97d1bcabdc [0x00007f97d1bcab40+0x000000000000009c]
J 16512 c1 io.netty.util.concurrent.FastThreadLocalRunnable.run()V (22 bytes) @ 0x00007f97d17bb60c [0x00007f97d17bb4e0+0x000000000000012c]
J 16807 c1 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x00007f97d2290c24 [0x00007f97d2290ac0+0x0000000000000164]
v ~StubRoutines::call_stub
V [libjvm.so+0x829765] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x315
V [libjvm.so+0x82af5b] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x1cb
V [libjvm.so+0x8f7843] thread_entry(JavaThread*, JavaThread*)+0xa3
V [libjvm.so+0xe5ef76] JavaThread::thread_main_inner()+0x196
V [libjvm.so+0xe629b0] Thread::call_run()+0xc0
V [libjvm.so+0xc23e91] thread_native_entry(Thread*)+0xe1
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 21109 org.rocksdb.RocksDB.get(JJ[BII)[B (0 bytes) @ 0x00007f97d9460a2f [0x00007f97d94609c0+0x000000000000006f]
J 21083 c1 org.rocksdb.RocksDB.get(Lorg/rocksdb/ReadOptions;[B)[B (17 bytes) @ 0x00007f97d1b5886c [0x00007f97d1b587c0+0x00000000000000ac]
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.storeGet(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+186
J 20656 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore.get(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (260 bytes) @ 0x00007f97d21b312c [0x00007f97d21b1d00+0x000000000000142c]
J 20960 c1 org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.readValueFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (38 bytes) @ 0x00007f97d20e662c [0x00007f97d20e64c0+0x000000000000016c]
J 22917 c2 com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$remap$16(Z[Ljava/lang/Object;Ljava/util/function/BiFunction;Ljava/lang/Object;[J[ILjava/lang/Object;[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node; (466 bytes) @ 0x00007f97d95b2688 [0x00007f97d95b2320+0x0000000000000368]
J 23900 c2 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007f97d98632bc [0x00007f97d9862880+0x0000000000000a3c]
J 22882 c2 org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.refresh(Ljava/lang/String;)V (36 bytes) @ 0x00007f97d95a2f98 [0x00007f97d95a29a0+0x00000000000005f8]
J 25254 c2 org.apache.pulsar.metadata.impl.AbstractMetadataStore.accept(Lorg/apache/pulsar/metadata/api/Notification;)V (405 bytes) @ 0x00007f97d9496698 [0x00007f97d94960e0+0x00000000000005b8]
J 23122 c2 org.apache.pulsar.metadata.impl.AbstractMetadataStore.accept(Ljava/lang/Object;)V (22 bytes) @ 0x00007f97d9617678 [0x00007f97d9617620+0x0000000000000058]
J 22682 c2 java.util.concurrent.CompletableFuture$AsyncSupply.run()V [email protected] (61 bytes) @ 0x00007f97d9571c44 [0x00007f97d9571a40+0x0000000000000204]
J 20533 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007f97d93d1f84 [0x00007f97d93d1f20+0x0000000000000064]
J 20537 c2 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007f97d93d2a74 [0x00007f97d93d2960+0x0000000000000114]
J 23462 c2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x00007f97d975b360 [0x00007f97d975b140+0x0000000000000220]
J 14766 c1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V [email protected] (9 bytes) @ 0x00007f97d1bcabdc [0x00007f97d1bcab40+0x000000000000009c]
J 16512 c1 io.netty.util.concurrent.FastThreadLocalRunnable.run()V (22 bytes) @ 0x00007f97d17bb60c [0x00007f97d17bb4e0+0x000000000000012c]
J 16807 c1 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x00007f97d2290c24 [0x00007f97d2290ac0+0x0000000000000164]
v ~StubRoutines::call_stub
Yet another crash: https://gist.github.com/lhotari/cb00532b4888acfda9053a5b4d47afee (source: https://github.com/apache/pulsar/actions/runs/7272286944/job/19814369338#step:11:343)
Stack: [0x00007f6914100000,0x00007f6914200000], sp=0x00007f69141fd470, free space=1013k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni14323513044527187158.so+0x2b3ff9] rocksdb_get_helper(JNIEnv_*, rocksdb::DB*, rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, _jbyteArray*, int, int)+0x199
j org.rocksdb.RocksDB.get(JJ[BII)[B+0
j org.rocksdb.RocksDB.get(Lorg/rocksdb/ReadOptions;[B)[B+13
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.storeGet(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.get(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+99
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.readValueFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.lambda$refresh$13(Ljava/lang/String;Ljava/lang/String;Ljava/util/concurrent/CompletableFuture;)Ljava/util/concurrent/CompletableFuture;+10
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl$$Lambda$402+0x00007f66b03feef8.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+16
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.lambda$computeIfPresent$2([Ljava/util/concurrent/CompletableFuture;Ljava/util/function/BiFunction;Ljava/lang/Object;Ljava/util/concurrent/CompletableFuture;)Ljava/util/concurrent/CompletableFuture;+13
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView$$Lambda$403+0x00007f66b03ff588.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+13
j com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$remap$16(Z[Ljava/lang/Object;Ljava/util/function/BiFunction;Ljava/lang/Object;[J[ILjava/lang/Object;[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node;+254
j com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$404+0x00007f66b03ff7c0.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+53
J 6619 c1 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007f6915e82a74 [0x00007f6915e81ce0+0x0000000000000d94]
j com.github.benmanes.caffeine.cache.BoundedLocalCache.remap(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/BiFunction;[JZ)Ljava/lang/Object;+65
j com.github.benmanes.caffeine.cache.BoundedLocalCache.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;ZZZ)Ljava/lang/Object;+64
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.computeIfPresent(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/util/concurrent/CompletableFuture;+51
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.computeIfPresent(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object;+3
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.refresh(Ljava/lang/String;)V+24
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.accept(Lorg/apache/pulsar/metadata/api/Notification;)V+67
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$accept$7(Lorg/apache/pulsar/metadata/api/Notification;Lorg/apache/pulsar/metadata/cache/impl/MetadataCacheImpl;)V+9
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$401+0x00007f66b03fe6f0.accept(Ljava/lang/Object;)V+8
J 6437 c1 java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V [email protected] (47 bytes) @ 0x00007f69155b47ec [0x00007f69155b4600+0x00000000000001ec]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.accept(Lorg/apache/pulsar/metadata/api/Notification;)V+254
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.accept(Ljava/lang/Object;)V+12
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$5(Lorg/apache/pulsar/metadata/api/Notification;Ljava/util/function/Consumer;)V+9
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$399+0x00007f66b03fa510.accept(Ljava/lang/Object;)V+8
J 6437 c1 java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V [email protected] (47 bytes) @ 0x00007f69155b47ec [0x00007f69155b4600+0x00000000000001ec]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$6(Lorg/apache/pulsar/metadata/api/Notification;)Ljava/lang/Void;+17
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$396+0x00007f66b03f86a8.get()Ljava/lang/Object;+8
j java.util.concurrent.CompletableFuture$AsyncSupply.run()V+37 [email protected]
J 4620 c1 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007f6915d6096c [0x00007f6915d60840+0x000000000000012c]
J 5982 c1 java.util.concurrent.FutureTask.run()V [email protected] (123 bytes) @ 0x00007f6915415754 [0x00007f6915414fa0+0x00000000000007b4]
J 4619 c1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007f6915d602a4 [0x00007f6915d5ffc0+0x00000000000002e4]
J 5926 c1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x00007f6915b40674 [0x00007f6915b3fd20+0x0000000000000954]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 [email protected]
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
J 6144 c1 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x00007f691544cc24 [0x00007f691544cac0+0x0000000000000164]
v ~StubRoutines::call_stub
V [libjvm.so+0x829765] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x315
V [libjvm.so+0x82af5b] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x1cb
V [libjvm.so+0x8f7843] thread_entry(JavaThread*, JavaThread*)+0xa3
V [libjvm.so+0xe5ef76] JavaThread::thread_main_inner()+0x196
V [libjvm.so+0xe629b0] Thread::call_run()+0xc0
V [libjvm.so+0xc23e91] thread_native_entry(Thread*)+0xe1
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.rocksdb.RocksDB.get(JJ[BII)[B+0
j org.rocksdb.RocksDB.get(Lorg/rocksdb/ReadOptions;[B)[B+13
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.storeGet(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.get(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+99
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.readValueFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.lambda$refresh$13(Ljava/lang/String;Ljava/lang/String;Ljava/util/concurrent/CompletableFuture;)Ljava/util/concurrent/CompletableFuture;+10
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl$$Lambda$402+0x00007f66b03feef8.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+16
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.lambda$computeIfPresent$2([Ljava/util/concurrent/CompletableFuture;Ljava/util/function/BiFunction;Ljava/lang/Object;Ljava/util/concurrent/CompletableFuture;)Ljava/util/concurrent/CompletableFuture;+13
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView$$Lambda$403+0x00007f66b03ff588.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+13
j com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$remap$16(Z[Ljava/lang/Object;Ljava/util/function/BiFunction;Ljava/lang/Object;[J[ILjava/lang/Object;[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node;+254
j com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$404+0x00007f66b03ff7c0.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+53
J 6619 c1 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007f6915e82a74 [0x00007f6915e81ce0+0x0000000000000d94]
j com.github.benmanes.caffeine.cache.BoundedLocalCache.remap(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/BiFunction;[JZ)Ljava/lang/Object;+65
j com.github.benmanes.caffeine.cache.BoundedLocalCache.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;ZZZ)Ljava/lang/Object;+64
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.computeIfPresent(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/util/concurrent/CompletableFuture;+51
j com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.computeIfPresent(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object;+3
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.refresh(Ljava/lang/String;)V+24
j org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.accept(Lorg/apache/pulsar/metadata/api/Notification;)V+67
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$accept$7(Lorg/apache/pulsar/metadata/api/Notification;Lorg/apache/pulsar/metadata/cache/impl/MetadataCacheImpl;)V+9
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$401+0x00007f66b03fe6f0.accept(Ljava/lang/Object;)V+8
J 6437 c1 java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V [email protected] (47 bytes) @ 0x00007f69155b47ec [0x00007f69155b4600+0x00000000000001ec]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.accept(Lorg/apache/pulsar/metadata/api/Notification;)V+254
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.accept(Ljava/lang/Object;)V+12
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$5(Lorg/apache/pulsar/metadata/api/Notification;Ljava/util/function/Consumer;)V+9
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$399+0x00007f66b03fa510.accept(Ljava/lang/Object;)V+8
J 6437 c1 java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V [email protected] (47 bytes) @ 0x00007f69155b47ec [0x00007f69155b4600+0x00000000000001ec]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$6(Lorg/apache/pulsar/metadata/api/Notification;)Ljava/lang/Void;+17
j org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$396+0x00007f66b03f86a8.get()Ljava/lang/Object;+8
j java.util.concurrent.CompletableFuture$AsyncSupply.run()V+37 [email protected]
J 4620 c1 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007f6915d6096c [0x00007f6915d60840+0x000000000000012c]
J 5982 c1 java.util.concurrent.FutureTask.run()V [email protected] (123 bytes) @ 0x00007f6915415754 [0x00007f6915414fa0+0x00000000000007b4]
J 4619 c1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007f6915d602a4 [0x00007f6915d5ffc0+0x00000000000002e4]
J 5926 c1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x00007f6915b40674 [0x00007f6915b3fd20+0x0000000000000954]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 [email protected]
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
J 6144 c1 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x00007f691544cc24 [0x00007f691544cac0+0x0000000000000164]
v ~StubRoutines::call_stub
@merlimat @hangc0276 Do we have any plans to update RocksDB version in Bookkeeper? We are currently on 7.9.2 and it seems that 7.10.2 is the newest 7.x: https://github.com/facebook/rocksdb/releases/tag/v7.10.2 . Newest release is 8.9.1: https://github.com/facebook/rocksdb/releases . When do we usually upgrade the RocksDB version? I'm a bit concerned about these crashes that are common in the pulsar-metadata tests. Are these a bug in Pulsar's metadata driver for RocksDB or is it a real bug in RocksDB itself?
Yet another crash, this time in broker tests.
Stack: [0x00007f68f2c00000,0x00007f68f2d00000], sp=0x00007f68f2cfdac8, free space=1014k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni10411210078074458540.so+0x2adff0] rocksdb_iterator_helper(rocksdb::DB*, rocksdb::ReadOptions, rocksdb::ColumnFamilyHandle*)+0x60
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
J 23070 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture; (21 bytes) @ 0x00007f6b5e554f6c [0x00007f6b5e554e40+0x000000000000012c]
J 23069 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/Object;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture; (23 bytes) @ 0x00007f6b5df8bd44 [0x00007f6b5df8bbc0+0x0000000000000184]
J 49063 c2 com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14([Ljava/lang/Object;Ljava/util/function/Function;Ljava/lang/Object;[J[I[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node; (355 bytes) @ 0x00007f6b66f75930 [0x00007f6b66f757a0+0x0000000000000190]
J 28833 c2 com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$502+0x00007f68f851a700.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (49 bytes) @ 0x00007f6b661163d8 [0x00007f6b661162c0+0x0000000000000118]
J 50805 c1 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007f6b5e6bf814 [0x00007f6b5e6bf1e0+0x0000000000000634]
J 50551 c2 com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(Ljava/lang/Object;Ljava/util/function/Function;ZZ)Ljava/lang/Object; (161 bytes) @ 0x00007f6b65f631e0 [0x00007f6b65f628e0+0x0000000000000900]
J 26511 c2 com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.get(Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture; (20 bytes) @ 0x00007f6b65e25680 [0x00007f6b65e25460+0x0000000000000220]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+69
J 29539 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getBookiesThenFreshCache(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (91 bytes) @ 0x00007f6b5f7bf8cc [0x00007f6b5f7bf6c0+0x000000000000020c]
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getWritableBookies()Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.lambda$updatedBookies$26(Lorg/apache/pulsar/metadata/api/Notification;Lorg/apache/bookkeeper/net/BookieId;Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+355
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$4744+0x00007f68f903fd90.get()Ljava/lang/Object;+16
J 32154 c1 org.apache.pulsar.common.util.FutureUtil$Sequencer.sequential(Ljava/util/function/Supplier;)Ljava/util/concurrent/CompletableFuture; (160 bytes) @ 0x00007f6b5f993f9c [0x00007f6b5f9932e0+0x0000000000000cbc]
J 50820 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.updatedBookies(Lorg/apache/pulsar/metadata/api/Notification;)V (146 bytes) @ 0x00007f6b5e69f42c [0x00007f6b5e69ec40+0x00000000000007ec]
J 50819 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$2017+0x00007f68f8be2c70.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007f6b5e6be6cc [0x00007f6b5e6be540+0x000000000000018c]
J 33866 c2 java.util.concurrent.CompletableFuture$AsyncSupply.run()V [email protected] (61 bytes) @ 0x00007f6b65e60ea4 [0x00007f6b65e60d00+0x00000000000001a4]
J 7662 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007f6b6525bb84 [0x00007f6b6525bb20+0x0000000000000064]
J 41512 c2 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007f6b658425e4 [0x00007f6b65842360+0x0000000000000284]
J 23159 c2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x00007f6b65b663cc [0x00007f6b65b66200+0x00000000000001cc]
J 39308 c2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V [email protected] (9 bytes) @ 0x00007f6b653e9000 [0x00007f6b653e8fa0+0x0000000000000060]
J 30635 c2 io.netty.util.concurrent.FastThreadLocalRunnable.run()V (22 bytes) @ 0x00007f6b6633bdb8 [0x00007f6b6633bd40+0x0000000000000078]
J 32703 c2 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x00007f6b664f4488 [0x00007f6b664f4420+0x0000000000000068]
v ~StubRoutines::call_stub
V [libjvm.so+0x829765] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x315
V [libjvm.so+0x82af5b] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x1cb
V [libjvm.so+0x8f7843] thread_entry(JavaThread*, JavaThread*)+0xa3
V [libjvm.so+0xe5ef76] JavaThread::thread_main_inner()+0x196
V [libjvm.so+0xe629b0] Thread::call_run()+0xc0
V [libjvm.so+0xc23e91] thread_native_entry(Thread*)+0xe1
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
J 23070 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture; (21 bytes) @ 0x00007f6b5e554f6c [0x00007f6b5e554e40+0x000000000000012c]
J 23069 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/Object;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture; (23 bytes) @ 0x00007f6b5df8bd44 [0x00007f6b5df8bbc0+0x0000000000000184]
J 49063 c2 com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14([Ljava/lang/Object;Ljava/util/function/Function;Ljava/lang/Object;[J[I[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node; (355 bytes) @ 0x00007f6b66f75930 [0x00007f6b66f757a0+0x0000000000000190]
J 28833 c2 com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$502+0x00007f68f851a700.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (49 bytes) @ 0x00007f6b661163d8 [0x00007f6b661162c0+0x0000000000000118]
J 50805 c1 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007f6b5e6bf814 [0x00007f6b5e6bf1e0+0x0000000000000634]
J 50551 c2 com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(Ljava/lang/Object;Ljava/util/function/Function;ZZ)Ljava/lang/Object; (161 bytes) @ 0x00007f6b65f631e0 [0x00007f6b65f628e0+0x0000000000000900]
J 26511 c2 com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.get(Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture; (20 bytes) @ 0x00007f6b65e25680 [0x00007f6b65e25460+0x0000000000000220]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+69
J 29539 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getBookiesThenFreshCache(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (91 bytes) @ 0x00007f6b5f7bf8cc [0x00007f6b5f7bf6c0+0x000000000000020c]
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getWritableBookies()Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.lambda$updatedBookies$26(Lorg/apache/pulsar/metadata/api/Notification;Lorg/apache/bookkeeper/net/BookieId;Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+355
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$4744+0x00007f68f903fd90.get()Ljava/lang/Object;+16
J 32154 c1 org.apache.pulsar.common.util.FutureUtil$Sequencer.sequential(Ljava/util/function/Supplier;)Ljava/util/concurrent/CompletableFuture; (160 bytes) @ 0x00007f6b5f993f9c [0x00007f6b5f9932e0+0x0000000000000cbc]
J 50820 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.updatedBookies(Lorg/apache/pulsar/metadata/api/Notification;)V (146 bytes) @ 0x00007f6b5e69f42c [0x00007f6b5e69ec40+0x00000000000007ec]
J 50819 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$2017+0x00007f68f8be2c70.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007f6b5e6be6cc [0x00007f6b5e6be540+0x000000000000018c]
J 33866 c2 java.util.concurrent.CompletableFuture$AsyncSupply.run()V [email protected] (61 bytes) @ 0x00007f6b65e60ea4 [0x00007f6b65e60d00+0x00000000000001a4]
J 7662 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007f6b6525bb84 [0x00007f6b6525bb20+0x0000000000000064]
J 41512 c2 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007f6b658425e4 [0x00007f6b65842360+0x0000000000000284]
J 23159 c2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x00007f6b65b663cc [0x00007f6b65b66200+0x00000000000001cc]
J 39308 c2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V [email protected] (9 bytes) @ 0x00007f6b653e9000 [0x00007f6b653e8fa0+0x0000000000000060]
J 30635 c2 io.netty.util.concurrent.FastThreadLocalRunnable.run()V (22 bytes) @ 0x00007f6b6633bdb8 [0x00007f6b6633bd40+0x0000000000000078]
J 32703 c2 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x00007f6b664f4488 [0x00007f6b664f4420+0x0000000000000068]
v ~StubRoutines::call_stub
full dump https://gist.github.com/lhotari/f35745b737e7107f13c8becea80f9da4 location: https://github.com/apache/pulsar/actions/runs/7299672614/job/19893235168#step:11:1590
A recent crash with Java 17.0.10 (it contains fix for JDK-8314024): https://gist.githubusercontent.com/lhotari/c21fbfdae7f3c59985480397570f9783/raw/7a73251dbd5864ebf6b5045ba1a76c5248f28033/hs_err_pid18047.log
Current thread (0x00007f1a6c3380b0): JavaThread "RocksdbMetadataStore-43-1" [_thread_in_native, id=19619, stack(0x00007f1a74fff000,0x00007f1a750ff000)]
Stack: [0x00007f1a74fff000,0x00007f1a750ff000], sp=0x00007f1a750fbf18, free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni6214973291081166669.so+0x2adff0] rocksdb_iterator_helper(rocksdb::DB*, rocksdb::ReadOptions, rocksdb::ColumnFamilyHandle*)+0x60
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
J 10001 c1 org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (637 bytes) @ 0x00007f1c85f2b3cc [0x00007f1c85f2aec0+0x000000000000050c]
J 9839 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture; (21 bytes) @ 0x00007f1c85ec3e6c [0x00007f1c85ec3d40+0x000000000000012c]
J 9838 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/Object;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture; (23 bytes) @ 0x00007f1c85ec3944 [0x00007f1c85ec37c0+0x0000000000000184]
J 9746 c1 com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache$$Lambda$423+0x00007f1a18442350.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (15 bytes) @ 0x00007f1c85e8cd74 [0x00007f1c85e8cb40+0x0000000000000234]
J 9745 c1 com.github.benmanes.caffeine.cache.LocalAsyncCache.lambda$get$2([Ljava/util/concurrent/CompletableFuture;Ljava/util/function/BiFunction;Ljava/lang/Object;Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture; (34 bytes) @ 0x00007f1c85e8bd0c [0x00007f1c85e8baa0+0x000000000000026c]
J 9612 c1 com.github.benmanes.caffeine.cache.LocalAsyncCache$$Lambda$424+0x00007f1a18442588.apply(Ljava/lang/Object;)Ljava/lang/Object; (23 bytes) @ 0x00007f1c85e3bee4 [0x00007f1c85e3bd40+0x00000000000001a4]
J 9610 c1 com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14([Ljava/lang/Object;Ljava/util/function/Function;Ljava/lang/Object;[J[I[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node; (355 bytes) @ 0x00007f1c85e372ac [0x00007f1c85e37120+0x000000000000018c]
J 9744 c1 com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$425+0x00007f1a184427d0.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (49 bytes) @ 0x00007f1c85e8153c [0x00007f1c85e812e0+0x000000000000025c]
J 8099 c1 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (654 bytes) @ 0x00007f1c85a1b714 [0x00007f1c85a1b0e0+0x0000000000000634]
J 9742 c1 com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/Function;[JZ)Ljava/lang/Object; (276 bytes) @ 0x00007f1c85e7df1c [0x00007f1c85e7d560+0x00000000000009bc]
J 9662 c1 com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(Ljava/lang/Object;Ljava/util/function/Function;ZZ)Ljava/lang/Object; (161 bytes) @ 0x00007f1c85e4f4dc [0x00007f1c85e4da80+0x0000000000001a5c]
J 9406 c1 com.github.benmanes.caffeine.cache.LocalAsyncCache.get(Ljava/lang/Object;Ljava/util/function/BiFunction;Z)Ljava/util/concurrent/CompletableFuture; (77 bytes) @ 0x00007f1c85dadf54 [0x00007f1c85dad7e0+0x0000000000000774]
J 10019 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (80 bytes) @ 0x00007f1c85f4208c [0x00007f1c85f418e0+0x00000000000007ac]
J 9920 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getBookiesThenFreshCache(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (91 bytes) @ 0x00007f1c85ef754c [0x00007f1c85ef7340+0x000000000000020c]
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getReadOnlyBookies()Ljava/util/concurrent/CompletableFuture;+12
J 9785 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.lambda$updatedBookies$26(Lorg/apache/pulsar/metadata/api/Notification;Lorg/apache/bookkeeper/net/BookieId;Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; (396 bytes) @ 0x00007f1c85ea332c [0x00007f1c85e9fda0+0x000000000000358c]
J 9784 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$919+0x00007f1a1894d1a8.get()Ljava/lang/Object; (20 bytes) @ 0x00007f1c85e9ed84 [0x00007f1c85e9ecc0+0x00000000000000c4]
J 9623 c1 org.apache.pulsar.common.util.FutureUtil$Sequencer.sequential(Ljava/util/function/Supplier;)Ljava/util/concurrent/CompletableFuture; (160 bytes) @ 0x00007f1c85e3e96c [0x00007f1c85e3d940+0x000000000000102c]
J 9364 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.updatedBookies(Lorg/apache/pulsar/metadata/api/Notification;)V (146 bytes) @ 0x00007f1c85d8dfac [0x00007f1c85d8d7c0+0x00000000000007ec]
J 9363 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$395+0x00007f1a1842ec50.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007f1c85d8d34c [0x00007f1c85d8d1c0+0x000000000000018c]
J 9982 c2 org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$415+0x00007f1a1843f468.accept(Ljava/lang/Object;)V (12 bytes) @ 0x00007f1c8d21e7d4 [0x00007f1c8d21e740+0x0000000000000094]
J 10882 c2 java.util.concurrent.CopyOnWriteArrayList.forEach(Ljava/util/function/Consumer;)V [email protected] (47 bytes) @ 0x00007f1c8d264c58 [0x00007f1c8d264ba0+0x00000000000000b8]
J 9357 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore.lambda$receivedNotification$6(Lorg/apache/pulsar/metadata/api/Notification;)Ljava/lang/Void; (28 bytes) @ 0x00007f1c85d8c774 [0x00007f1c85d8c3a0+0x00000000000003d4]
J 9356 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$$Lambda$414+0x00007f1a1843f240.get()Ljava/lang/Object; (12 bytes) @ 0x00007f1c85d8bf6c [0x00007f1c85d8bec0+0x00000000000000ac]
J 9307 c1 java.util.concurrent.CompletableFuture$AsyncSupply.run()V [email protected] (61 bytes) @ 0x00007f1c85d7b26c [0x00007f1c85d7b060+0x000000000000020c]
J 9578 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x00007f1c8d1fe704 [0x00007f1c8d1fe6a0+0x0000000000000064]
J 9092 c1 java.util.concurrent.FutureTask.run()V [email protected] (123 bytes) @ 0x00007f1c85d03254 [0x00007f1c85d02aa0+0x00000000000007b4]
J 9576 c1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x00007f1c85e10324 [0x00007f1c85e10040+0x00000000000002e4]
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 [email protected]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 [email protected]
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
j java.lang.Thread.run()V+11 [email protected]
v ~StubRoutines::call_stub
V [libjvm.so+0x829895] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x315
V [libjvm.so+0x82b08b] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x1cb
V [libjvm.so+0x8f78e3] thread_entry(JavaThread*, JavaThread*)+0xa3
V [libjvm.so+0xe5fa16] JavaThread::thread_main_inner()+0x196
V [libjvm.so+0xe633b8] Thread::call_run()+0xa8
V [libjvm.so+0xc24791] thread_native_entry(Thread*)+0xe1
Another recent crash: https://gist.github.com/lhotari/c95a1bbf42880d49b0de508f7870f94a
Stack: [0x0000720a49200000,0x0000720a49300000], sp=0x0000720a492fdab8, free space=1014k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni10433916121165253334.so+0x2adff0] rocksdb_iterator_helper(rocksdb::DB*, rocksdb::ReadOptions, rocksdb::ColumnFamilyHandle*)+0x60
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
J 53586 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture; (21 bytes) @ 0x0000720c6ab9d16c [0x0000720c6ab9d040+0x000000000000012c]
J 50411 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/Object;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture; (23 bytes) @ 0x0000720c6962c6b4 [0x0000720c6962c540+0x0000000000000174]
J 39087 c2 com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14([Ljava/lang/Object;Ljava/util/function/Function;Ljava/lang/Object;[J[I[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node; (355 bytes) @ 0x0000720c72741934 [0x0000720c72741780+0x00000000000001b4]
J 42550 c2 com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$496+0x00007209fc4f6640.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (49 bytes) @ 0x0000720c729bdcd8 [0x0000720c729bdbc0+0x0000000000000118]
J 55012 c2 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (822 bytes) @ 0x0000720c72ea1880 [0x0000720c72ea0980+0x0000000000000f00]
J 33464 c1 com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/Function;[JZ)Ljava/lang/Object; (276 bytes) @ 0x0000720c6b78e49c [0x0000720c6b78dc00+0x000000000000089c]
j com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(Ljava/lang/Object;Ljava/util/function/Function;ZZ)Ljava/lang/Object;+157
J 39494 c2 com.github.benmanes.caffeine.cache.LocalAsyncCache.get(Ljava/lang/Object;Ljava/util/function/BiFunction;Z)Ljava/util/concurrent/CompletableFuture; (77 bytes) @ 0x0000720c727e6fb4 [0x0000720c727e6e00+0x00000000000001b4]
J 21863 c2 com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.get(Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture; (20 bytes) @ 0x0000720c718feb54 [0x0000720c718feaa0+0x00000000000000b4]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+178
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getBookiesThenFreshCache(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+54
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getWritableBookies()Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.lambda$updatedBookies$26(Lorg/apache/pulsar/metadata/api/Notification;Lorg/apache/bookkeeper/net/BookieId;Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+355
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$4937+0x00007209fd15bd40.get()Ljava/lang/Object;+16
J 36324 c1 org.apache.pulsar.common.util.FutureUtil$Sequencer.sequential(Ljava/util/function/Supplier;)Ljava/util/concurrent/CompletableFuture; (160 bytes) @ 0x0000720c6ba8f9f4 [0x0000720c6ba8f040+0x00000000000009b4]
J 56975 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.updatedBookies(Lorg/apache/pulsar/metadata/api/Notification;)V (146 bytes) @ 0x0000720c6d50ace4 [0x0000720c6d50a780+0x0000000000000564]
J 56393 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$2075+0x00007209fcb96f08.accept(Ljava/lang/Object;)V (12 bytes) @ 0x0000720c6d39254c [0x0000720c6d3923c0+0x000000000000018c]
J 53585 c2 java.util.concurrent.CompletableFuture$AsyncSupply.run()V [email protected] (61 bytes) @ 0x0000720c71643ddc [0x0000720c716439c0+0x000000000000041c]
J 8303 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x0000720c7128db04 [0x0000720c7128daa0+0x0000000000000064]
J 34960 c2 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x0000720c723c7f60 [0x0000720c723c7ce0+0x0000000000000280]
J 32343 c2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x0000720c71b23694 [0x0000720c71b23440+0x0000000000000254]
J 51463 c2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V [email protected] (9 bytes) @ 0x0000720c712ab280 [0x0000720c712ab220+0x0000000000000060]
J 48582 c2 io.netty.util.concurrent.FastThreadLocalRunnable.run()V (22 bytes) @ 0x0000720c73011438 [0x0000720c730113c0+0x0000000000000078]
J 43180 c2 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x0000720c72a80588 [0x0000720c72a80520+0x0000000000000068]
v ~StubRoutines::call_stub
V [libjvm.so+0x859824] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x334
V [libjvm.so+0x85b30c] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x20c
V [libjvm.so+0x919760] thread_entry(JavaThread*, JavaThread*)+0x70
V [libjvm.so+0xed9a47] JavaThread::run()+0x127
V [libjvm.so+0xedcbcf] Thread::call_run()+0x9f
V [libjvm.so+0xc4ebb1] thread_native_entry(Thread*)+0xe1
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.rocksdb.RocksDB.iterator(JJ)J+0
j org.rocksdb.RocksDB.newIterator(Lorg/rocksdb/ReadOptions;)Lorg/rocksdb/RocksIterator;+14
j org.apache.pulsar.metadata.impl.RocksdbMetadataStore.getChildrenFromStore(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+82
J 53586 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/String;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture; (21 bytes) @ 0x0000720c6ab9d16c [0x0000720c6ab9d040+0x000000000000012c]
J 50411 c1 org.apache.pulsar.metadata.impl.AbstractMetadataStore$1.asyncLoad(Ljava/lang/Object;Ljava/util/concurrent/Executor;)Ljava/util/concurrent/CompletableFuture; (23 bytes) @ 0x0000720c6962c6b4 [0x0000720c6962c540+0x0000000000000174]
J 39087 c2 com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14([Ljava/lang/Object;Ljava/util/function/Function;Ljava/lang/Object;[J[I[Ljava/lang/Object;[Ljava/lang/Object;[Lcom/github/benmanes/caffeine/cache/RemovalCause;[Lcom/github/benmanes/caffeine/cache/Node;Ljava/lang/Object;Lcom/github/benmanes/caffeine/cache/Node;)Lcom/github/benmanes/caffeine/cache/Node; (355 bytes) @ 0x0000720c72741934 [0x0000720c72741780+0x00000000000001b4]
J 42550 c2 com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$496+0x00007209fc4f6640.apply(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (49 bytes) @ 0x0000720c729bdcd8 [0x0000720c729bdbc0+0x0000000000000118]
J 55012 c2 java.util.concurrent.ConcurrentHashMap.compute(Ljava/lang/Object;Ljava/util/function/BiFunction;)Ljava/lang/Object; [email protected] (822 bytes) @ 0x0000720c72ea1880 [0x0000720c72ea0980+0x0000000000000f00]
J 33464 c1 com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(Ljava/lang/Object;Ljava/lang/Object;Ljava/util/function/Function;[JZ)Ljava/lang/Object; (276 bytes) @ 0x0000720c6b78e49c [0x0000720c6b78dc00+0x000000000000089c]
j com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(Ljava/lang/Object;Ljava/util/function/Function;ZZ)Ljava/lang/Object;+157
J 39494 c2 com.github.benmanes.caffeine.cache.LocalAsyncCache.get(Ljava/lang/Object;Ljava/util/function/BiFunction;Z)Ljava/util/concurrent/CompletableFuture; (77 bytes) @ 0x0000720c727e6fb4 [0x0000720c727e6e00+0x00000000000001b4]
J 21863 c2 com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache.get(Ljava/lang/Object;)Ljava/util/concurrent/CompletableFuture; (20 bytes) @ 0x0000720c718feb54 [0x0000720c718feaa0+0x00000000000000b4]
j org.apache.pulsar.metadata.impl.AbstractMetadataStore.getChildren(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+178
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getBookiesThenFreshCache(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+54
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.getWritableBookies()Ljava/util/concurrent/CompletableFuture;+12
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.lambda$updatedBookies$26(Lorg/apache/pulsar/metadata/api/Notification;Lorg/apache/bookkeeper/net/BookieId;Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture;+355
j org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$4937+0x00007209fd15bd40.get()Ljava/lang/Object;+16
J 36324 c1 org.apache.pulsar.common.util.FutureUtil$Sequencer.sequential(Ljava/util/function/Supplier;)Ljava/util/concurrent/CompletableFuture; (160 bytes) @ 0x0000720c6ba8f9f4 [0x0000720c6ba8f040+0x00000000000009b4]
J 56975 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient.updatedBookies(Lorg/apache/pulsar/metadata/api/Notification;)V (146 bytes) @ 0x0000720c6d50ace4 [0x0000720c6d50a780+0x0000000000000564]
J 56393 c1 org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient$$Lambda$2075+0x00007209fcb96f08.accept(Ljava/lang/Object;)V (12 bytes) @ 0x0000720c6d39254c [0x0000720c6d3923c0+0x000000000000018c]
J 53585 c2 java.util.concurrent.CompletableFuture$AsyncSupply.run()V [email protected] (61 bytes) @ 0x0000720c71643ddc [0x0000720c716439c0+0x000000000000041c]
J 8303 c2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; [email protected] (14 bytes) @ 0x0000720c7128db04 [0x0000720c7128daa0+0x0000000000000064]
J 34960 c2 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V [email protected] (57 bytes) @ 0x0000720c723c7f60 [0x0000720c723c7ce0+0x0000000000000280]
J 32343 c2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V [email protected] (187 bytes) @ 0x0000720c71b23694 [0x0000720c71b23440+0x0000000000000254]
J 51463 c2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V [email protected] (9 bytes) @ 0x0000720c712ab280 [0x0000720c712ab220+0x0000000000000060]
J 48582 c2 io.netty.util.concurrent.FastThreadLocalRunnable.run()V (22 bytes) @ 0x0000720c73011438 [0x0000720c730113c0+0x0000000000000078]
J 43180 c2 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x0000720c72a80588 [0x0000720c72a80520+0x0000000000000068]
v ~StubRoutines::call_stub
@lhotari, thank you for your ongoing efforts in documenting this issue.
Unfortunately, the CI artifact has expired: https://github.com/apache/pulsar/actions/runs/7299672614
In RocksDB, attempting to operate on a closed database instance leads to a crash. This behavior is documented in a related issue: https://github.com/facebook/rocksdb/issues/5234
Below is a code snippet to demonstrate how to reproduce the crash:
import org.junit.rules.TemporaryFolder;
import org.rocksdb.Options;
import org.rocksdb.RocksDB;
import org.rocksdb.RocksDBException;
import org.rocksdb.RocksIterator;
import java.io.IOException;
class Scratch {
static {
RocksDB.loadLibrary();
}
public static void main(String[] args) throws IOException, RocksDBException {
TemporaryFolder tempFolder = new TemporaryFolder();
tempFolder.create();
String path = tempFolder.newFolder().toString();
Options options = new Options();
options.setCreateIfMissing(true);
RocksDB db = RocksDB.open(options, path);
byte[] key = {0x1};
byte[] value = {0x2};
db.put(key, value);
RocksIterator it = db.newIterator();
db.close();
it.value(); // crashes the JVM
}
}
In RocksDB, attempting to operate on a closed database instance leads to a crash. This behavior is documented in a related issue: facebook/rocksdb#5234
@Shawyeok yes, that's most likely the case. It's probably an issue in the shutdown sequence in tests. The scheduled jobs that might access the db should be closed before the db.