scio icon indicating copy to clipboard operation
scio copied to clipboard

Deadlock in AsyncLookupDoFn

Open RustedBones opened this issue 1 year ago • 1 comments

Some dead-lock have been observed on the GrpcDoFn with the following trace

Operation ongoing in step Perform Terminator request/parDo:1 for at least 10m00s without outputting or completing in state finish in thread pool-4-thread-19 with id 78
  at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  at [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  at app//com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:561)
  at app//com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:111)
  at app//com.spotify.scio.transforms.FutureHandlers$Guava.waitForFutures(FutureHandlers.java:49)
  at app//com.spotify.scio.transforms.BaseAsyncLookupDoFn.finishBundle(BaseAsyncLookupDoFn.java:199)
  at app//com.spotify.scio.grpc.GrpcDoFn$DoFnInvoker.invokeFinishBundle(Unknown Source)

This indicates one for the future never completes, though the stub was configured with a deadline.

RustedBones avatar Jan 10 '24 14:01 RustedBones

Potentially fixed by https://github.com/spotify/scio/pull/5209. Will wait for confirmation before closing

RustedBones avatar Jan 29 '24 09:01 RustedBones