bazel icon indicating copy to clipboard operation
bazel copied to clipboard

Published 20 hours ago • verified publisher icon bazelbuild

`UnsatisfiedLinkError` with `--experimental_remote_cache_compression` on Windows

Open fmeum opened this issue 1 year ago • 0 comments

Description of the bug:

Remote builds with --experimental_remote_cache_compression crash Bazel on Windows with:

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.UnsatisfiedLinkError: 'long com.github.luben.zstd.ZstdOutputStreamNoFinalizer.recommendedCOutSize()'
	at com.github.luben.zstd.ZstdOutputStreamNoFinalizer.recommendedCOutSize(Native Method)
	at com.github.luben.zstd.ZstdOutputStreamNoFinalizer.<clinit>(ZstdOutputStreamNoFinalizer.java:29)
	at com.github.luben.zstd.ZstdOutputStream.<init>(ZstdOutputStream.java:73)
	at com.github.luben.zstd.ZstdOutputStream.<init>(ZstdOutputStream.java:53)
	at com.google.devtools.build.lib.remote.zstd.ZstdCompressingInputStream.<init>(ZstdCompressingInputStream.java:52)
	at com.google.devtools.build.lib.remote.zstd.ZstdCompressingInputStream.<init>(ZstdCompressingInputStream.java:42)
	at com.google.devtools.build.lib.remote.Chunker.maybeInitialize(Chunker.java:254)
	at com.google.devtools.build.lib.remote.Chunker.seek(Chunker.java:153)
	at com.google.devtools.build.lib.remote.ByteStreamUploader$AsyncUpload.lambda$start$0(ByteStreamUploader.java:417)
	at com.google.devtools.build.lib.remote.Retrier.executeAsync(Retrier.java:277)
	at com.google.devtools.build.lib.remote.ByteStreamUploader$AsyncUpload.lambda$start$1(ByteStreamUploader.java:411)
	at com.google.devtools.build.lib.remote.util.Utils.refreshIfUnauthenticatedAsync(Utils.java:491)
	at com.google.devtools.build.lib.remote.ByteStreamUploader$AsyncUpload.start(ByteStreamUploader.java:409)
	at com.google.devtools.build.lib.remote.ByteStreamUploader.startAsyncUpload(ByteStreamUploader.java:343)
	at com.google.devtools.build.lib.remote.ByteStreamUploader.uploadBlobAsync(ByteStreamUploader.java:261)
	at com.google.devtools.build.lib.remote.GrpcCacheClient.uploadFile(GrpcCacheClient.java:442)
	at com.google.devtools.build.lib.remote.disk.DiskAndRemoteCacheClient.lambda$uploadFile$1(DiskAndRemoteCacheClient.java:98)
	at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213)
	at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202)
	at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.ImmediateFuture.addListener(ImmediateFuture.java:47)
	at com.google.common.util.concurrent.AbstractTransformFuture.create(AbstractTransformFuture.java:39)
	at com.google.common.util.concurrent.Futures.transformAsync(Futures.java:446)
	at com.google.devtools.build.lib.remote.disk.DiskAndRemoteCacheClient.uploadFile(DiskAndRemoteCacheClient.java:97)
	at com.google.devtools.build.lib.remote.RemoteCache.lambda$uploadFile$1(RemoteCache.java:149)
	at com.google.devtools.build.lib.remote.util.RxFutures$OnceCompletableOnSubscribe.subscribe(RxFutures.java:79)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:40)
	at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2859)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle.subscribeActual(CompletableToSingle.java:37)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
	at com.google.devtools.build.lib.remote.util.AsyncTaskCache$Execution.subscribeActual(AsyncTaskCache.java:157)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
	at com.google.devtools.build.lib.remote.util.AsyncTaskCache.lambda$execute$1(AsyncTaskCache.java:309)
	at io.reactivex.rxjava3.internal.operators.single.SingleCreate.subscribeActual(SingleCreate.java:40)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableFromSingle.subscribeActual(CompletableFromSingle.java:29)
	at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2859)
	at com.google.devtools.build.lib.remote.util.RxFutures.toListenableFuture(RxFutures.java:195)
	at com.google.devtools.build.lib.remote.RemoteCache.uploadFile(RemoteCache.java:151)
	at com.google.devtools.build.lib.remote.UploadManifest.lambda$upload$0(UploadManifest.java:371)
	at com.google.devtools.build.lib.remote.util.RxFutures$OnceCompletableOnSubscribe.subscribe(RxFutures.java:79)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:40)
	at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2859)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle.subscribeActual(CompletableToSingle.java:37)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
	at io.reactivex.rxjava3.internal.operators.single.SingleResumeNext.subscribeActual(SingleResumeNext.java:39)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
	at io.reactivex.rxjava3.internal.operators.single.SingleDoFinally.subscribeActual(SingleDoFinally.java:44)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber.onNext(FlowableFlatMapSingle.java:131)
	at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapPublisher$SingleFlatMapPublisherObserver.onNext(SingleFlatMapPublisher.java:107)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.fastPath(FlowableFromIterable.java:185)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:129)
	at io.reactivex.rxjava3.internal.subscriptions.SubscriptionHelper.deferredSetOnce(SubscriptionHelper.java:202)
	at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapPublisher$SingleFlatMapPublisherObserver.onSubscribe(SingleFlatMapPublisher.java:102)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15917)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15863)
	at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapPublisher$SingleFlatMapPublisherObserver.onSuccess(SingleFlatMapPublisher.java:96)
	at io.reactivex.rxjava3.internal.operators.single.SingleDoOnSuccess$DoOnSuccess.onSuccess(SingleDoOnSuccess.java:60)
	at io.reactivex.rxjava3.internal.operators.single.SingleDoOnDispose$DoOnDisposeObserver.onSuccess(SingleDoOnDispose.java:84)
	at io.reactivex.rxjava3.internal.operators.single.SingleDoOnError$DoOnError.onSuccess(SingleDoOnError.java:52)
	at io.reactivex.rxjava3.internal.operators.single.SingleDoOnSubscribe$DoOnSubscribeSingleObserver.onSuccess(SingleDoOnSubscribe.java:77)
	at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onSuccess(SingleCreate.java:68)
	at com.google.devtools.build.lib.remote.util.RxFutures$OnceSingleOnSubscribe$1.onSuccess(RxFutures.java:155)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746)
	at com.google.common.util.concurrent.CombinedFuture$CallableInterruptibleTask.setValue(CombinedFuture.java:188)
	at com.google.common.util.concurrent.CombinedFuture$CombinedFutureInterruptibleTask.afterRanInterruptibly(CombinedFuture.java:134)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.CombinedFuture$CombinedFutureInterruptibleTask.execute(CombinedFuture.java:104)
	at com.google.common.util.concurrent.CombinedFuture.handleAllCompleted(CombinedFuture.java:62)
	at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:283)
	at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:265)
	at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42)
	at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746)
	at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:110)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746)
	at com.google.common.util.concurrent.CombinedFuture$CallableInterruptibleTask.setValue(CombinedFuture.java:188)
	at com.google.common.util.concurrent.CombinedFuture$CombinedFutureInterruptibleTask.afterRanInterruptibly(CombinedFuture.java:134)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.CombinedFuture$CombinedFutureInterruptibleTask.execute(CombinedFuture.java:104)
	at com.google.common.util.concurrent.CombinedFuture.handleAllCompleted(CombinedFuture.java:62)
	at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:283)
	at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:265)
	at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42)
	at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746)
	at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:110)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746)
	at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:110)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746)
	at com.google.devtools.build.lib.remote.util.RxFutures$CompletableFuture.set(RxFutures.java:270)
	at com.google.devtools.build.lib.remote.util.RxFutures$2.onSuccess(RxFutures.java:233)
	at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onSuccess(SingleFlatMap.java:112)
	at io.reactivex.rxjava3.internal.operators.single.SingleUsing$UsingSingleObserver.onSuccess(SingleUsing.java:154)
	at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onSuccess(SingleCreate.java:68)
	at com.google.devtools.build.lib.remote.util.RxFutures$OnceSingleOnSubscribe$1.onSuccess(RxFutures.java:155)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746)
	at io.grpc.stub.ClientCalls$GrpcFuture.set(ClientCalls.java:558)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:531)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.google.devtools.build.lib.remote.NetworkTimeInterceptor$NetworkTimeCall$1.onClose(NetworkTimeInterceptor.java:81)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:557)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:7[38](https://github.com/CodeIntelligenceTesting/jazzer/runs/7649586583?check_suite_focus=true#step:6:39))
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:[71](https://github.com/CodeIntelligenceTesting/jazzer/runs/7649586583?check_suite_focus=true#step:6:72)7)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:[133](https://github.com/CodeIntelligenceTesting/jazzer/runs/7649586583?check_suite_focus=true#step:6:134))
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

We reliably hit this on CI for https://github.com/CodeIntelligenceTesting/jazzer/pull/431. The bazel command we run is:

bazelisk build --config=ci --remote_header=x-buildbuddy-api-key=REDACTED --java_runtime_version=local_jdk_8 --disk_cache=%HOME%/bazel-disk  //...

I don't expect this to be BuildBuddy-specific, but haven't been able to test it with other backends.

Which operating system are you running Bazel on?

Windows

What is the output of bazel info release?

5.3.0rc1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

fmeum avatar Aug 03 '22 14:08 fmeum

Looks like it failed to load the zstd JNI library defined at https://cs.opensource.google/bazel/bazel/+/master:third_party/zstd-jni/zstd-jni.BUILD;l=2

Not sure if this is only a remote execution issue.

meteorcloudy avatar Sep 16 '22 14:09 meteorcloudy

Zstd support was added https://github.com/bazelbuild/bazel/pull/14041 @coeuvre Did we verify if it works on Windows?

meteorcloudy avatar Sep 16 '22 14:09 meteorcloudy

No, I think we didn't verity it on Windows.

coeuvre avatar Sep 16 '22 15:09 coeuvre

I can confirm that zstd is broken on windows and causes bazel 5.3.0 to crash. We tried to migrate our packages and held off because of this

gregoryT5 avatar Sep 16 '22 18:09 gregoryT5