manifold
manifold copied to clipboard
clj-commons
an easy way to adapt a deferred to Java CompleteabaleFuture
Also interested in this. Manifold's rationale is to help libraries stay unopinionated about concrete async abstraction. It works reasonably well for libraries consuming async objects. For libraries that produce them however, Deferred is just yet another opinionated abstraction to choose from. CompletableFuture is gaining popularity in the Java world. Would you consider Deferred to implement CompletionStage?
Something like this function should wrap it for you.
(defn deferred->completable-future [d]
(let [fut (new CompletableFuture)]
(on-realized d #(.complete fut %)
#(.completeExceptionally fut %))))
Thanks @mac01021. I assume you'd want to actually return fut.
I understand the transformation is easy enough to implement yourself. The library would have been much nicer to use if it wasn't necessary in the first place. Being able to say that Manifold runs on the platform standards could be a big selling point imho, esp. that Manifold targets library authors.
thanks, @mac01021 I was searching for a way that does not create a new CompletableFuture object but rather, extending a deferred to also be a CompletionStage.
Lately, the standard of writing async code in libraries is evolving to either:
- Using bare Java
CompletableFuture. This is the lowest common denominator and is a part of Java 8 (which is considered old now). Users can then choose between using promesa as a thin utility library or using manifold to coerce the futures or introp directly. - Using
core.asyncchannels a-la-golang style and then users should usecore.asyncrealm to solve their async tasks at hand.
The first option is both the shortest path when you write a library that wraps an existing nonblocking java one and the most non-permisive IMO.
@barkanido unlike Manifold, Promesa does not introduce any new types. Instead, it provides a Clojure API around the CompletableFuture (on JVM).
Assume you're wrapping a non-blocking Java library. If it speaks CompletableFutures already, then using Promesa doesn't change much in terms of 'permissiveness' or approachability of the resulting lib. On JVM that abstraction is the standard, even if hopelessly awkward to use.
Manifold is useful when the library doesn't speak CompletableFutures but one of many others abstractions. It's a shame that the abstraction you end up with is a custom one though. (relevant XKCD)
Would it really be worth making deferreds implement CompletionStage instead of just providing something like deferred->completable-future by default?
Probably, I just haven't done it yet because it's a bit tedious :)
There's 40+ methods in the CompletionStage interface that would have to be implemented, probably for multiple deferred implementations. (One of the annoying consequences of Clojure forbidding inheritance is discovering all the places where you have to duplicate code because types/records are crippled.)
The alternatives involve more drastic changes, but I'd consider them. One is to replace the impl of resolved deferreds like SuccessDeferred/ErrorDeferred with a single mutable deferred that's immediately resolved, and defer all calls to the underlying mutable deferred. We could also switch the existing interfaces to protocols and thus extend/use CompletableFuture itself, but that would be a breaking change.
Perhaps we could have a CompletionStageDeferred wrapper and calls to d/->deferred or d/deferred wrap the returned deferred into a CompletionStageDeferred that delegates the regular IDeferred calls to the original deferred, and provides an implementation for the CompletionStage interface as well.
That would certainly be one way to consolidate code, but at the cost of an extra layer of indirection for all Manifold deferred calls. I don't think that tradeoff is worth it.
FWIW, Zach wrote some funky macro attempt at inheritance that might help in this situation, if you can get it working.
Checkout Potemkin's def-abstract-type and deftype+ macros. It's not truly inheritance, since it's based on macros (the relevant code is physically being copied from a "parent" deftype to a "child" deftype), but it may suffice.
I've been playing with those potemkin macros in manifold, and they fit like a glove for CompletionStage. Is it be better to copy/adapt the source of those macros from potemkin, or is it better to just go and add potemkin to the manifold dependencies?
Adding a potemkin dep is fine, especially since Aleph already depends on it, but double-check, because there are a few macros that were copied from potemkin, and don't have the same bodies. I think the updated potemkin versions should work though.
I'll take note on that, I have been playing with this, and CompletionStage is ok with implement with potemkin. However, testing the 30+ methods can be a bit overwhelming since just the chain, zip and alt operators become around 27 methods in CompletionStage. OpenJDK tests these with regular tests, which makes it around 650 lines of code for those methods. I wonder if it would be acceptable to implement a "test registry" to reduce the copy-pastery (on the expense of it being considerably more complex, which is a minus)
For context, I had something like this in mind:
;; On these tests:
;; CompletionStage has many methods that mimic the chain, zip and alt
;; functions in manifold. Unfortunately, each of these has 3 different versions,
;; one for each java functional interaface, and each version has 3
;; variants/modes, a raw/same thread variant, an async variant which runs in
;; a separate thread when possible, and an async variant that runs
;; in a given executor.
(def functor-method-info
[{:methods {:raw (fn [d op _] (.thenApply ^CompletionStage d op))
:async (fn [d op _] (.thenApplyAsync ^CompletionStage d op))
:with-executor
(fn [d op ex] (.thenApplyAsync ^CompletionStage d op ex))}
:interface fn->Function
:inner-assertion #(is (= % "a test string"))
:post-assertion #(is (= % true))}
{:methods {:raw (fn [d op _] (.thenAccept ^CompletionStage d op))
:async (fn [d op _] (.thenAcceptAsync ^CompletionStage d op))
:with-executor
(fn [d op ex] (.thenAcceptAsync ^CompletionStage d op ex))}
:interface fn->Consumer
:inner-assertion #(is (= % "a test string"))
:post-assertion #(is (= % nil))}
{:methods {:raw (fn [d op _] (.thenRun ^CompletionStage d op))
:async (fn [d op _] (.thenRunAsync ^CompletionStage d op))
:with-executor
(fn [d op ex] (.thenRunAsync ^CompletionStage d op ex))}
:interface fn->Runnable
:inner-assertion #(is (= % nil))
:post-assertion #(is (= % nil))}])
(defn test-functor-success [method-info mode]
(let [previous-thread (Thread/currentThread)
[executor in-executor] (atom-executor)
was-called (atom false)
mode-assertion
(mode
{:raw #(is (identical? previous-thread (Thread/currentThread)))
:async #(is (not (identical? previous-thread (Thread/currentThread))))
:with-executor #(is @in-executor)})
method (get-in method-info [:methods mode])
{:keys [inner-assertion post-assertion]
to-java-interface :interface} method-info
d1 (d/success-deferred "a test string")
d2 (method
d1
(to-java-interface
(fn [x]
(mode-assertion)
(inner-assertion x)
(reset! was-called true)
(= x "a test string")))
executor)]
(is (= @d1 "a test string"))
(post-assertion @d2)
(is (= @was-called true))))
(deftest test-functor-methods
(testing "functor success"
(dorun (for [method-info functor-method-info
mode [:raw :async :with-executor]]
(test-functor-success method-info mode)))))
Would it be better to unroll these instead?
No, we don't have to care as much about speed in tests, so unrolling isn't needed.
However, maybe consider the alternative of copying those JDK tests over? Their test methods should work the same way on Manifold deferreds once CompletionStage is implemented. There'd still be work to be done to make Manifold polyglot, but it's doable, and may be more robust than building our own test suite.
The code I sent earlier is actually an adaptation for some of the JDK tests, they do basically the same assertions, only differences are the presence of tests for error handling (which I have omitted in my reply), and that the tests I sent also test for the thread in which the callbacks run (since that's a bit of a delicate topic in manifold). I don't think we can directly copy the JDK tests since their CompetableFuture tests are GPL2.
Gahh. Never mind, then.
We can skip the tests for which thread the deferred runs on. AFAIK, that's not part of the CompletionStage contract.
Do your tests otherwise have the same coverage?