servicetalk icon indicating copy to clipboard operation
servicetalk copied to clipboard

Conversion from blocking primitives (e.g. Iterable) and deadlocks

Open Scottmitch opened this issue 3 years ago • 2 comments

additional context: https://github.com/apple/servicetalk/pull/2204

If FromIterablePublisher isn't able to unwrap to get access to the underlying Publisher deadlock is possible on consumption for the same reasons described in this PR. There are two options

Ensure the subscription and subscriber aren't using the same thread.

We can force a subscribeOn however this brings new questions:

  • offloading may not always be necessary
  • what level is the offloading applied at (e.g. do we require it during the conversion, for http we do this under the covers so we may have to hook into the execution strategy)
  • how do we infer/accept an executor (if on the concurrent.api package we don't have an executor to use, if we require one as a parameter that starts to overload operator responsibilities)
  • are there other areas in the blocking APIs with similar issues (InputStream, ..)?

Reduce default demand to 1 from PublisherAsBlockingIterable

This isn't ideal because it means demand will be very chatty and no batching is done, but will mean additional offloading isn't required.

Scottmitch avatar May 05 '22 23:05 Scottmitch

From @bondolo https://github.com/apple/servicetalk/pull/2203#discussion_r866337061

If it is not possible to recognize the source as potentially blocking on hasNext() then perhaps demand should be reduced to 1.

There doesn't seem to be a way to insert a subscribeOn here without using global executor which would be unacceptable for at least some customers.

are there other areas in the blocking APIs with similar issues?

I am concerned that there probably are and curious that none of our existing tests uncovered them. This problem reminds me of the problems users encounter with PipedOutputStream

Scottmitch avatar May 05 '22 23:05 Scottmitch

From @bondolo https://github.com/apple/servicetalk/pull/2203/commits/0cf2ac248e46f89c7bf8f081d5a266d903837cdf#r866337445

A more detailed comment on itemsToRequestNext describing problem with blocking Iterator might be helpful.

Scottmitch avatar May 06 '22 00:05 Scottmitch