Monix vNext — future plans

[alexandru]

Series 3.0.0 is now stable and for its duration we can no longer break binary compatibility — at least not on top of Scala 2.12+ — and I predict more features being built on top of what we have.

But we should also talk about a wish list for future versions, that can happen both soon (without breaking binary compatibility) or in a far future major version that can break binary compatibility.

This list is a work in progress, items may be added or removed, suggestions welcome.

Safe (non-binary breaking) changes

1. Provide bi-functor Task and Coeval implementations, possibly in their own sub-project?

• Not sure if we should, this needs to be discussed — I don't believe in the approach, but there are some people that do
• As an alternative, maybe a nicer EitherT would be better (e.g. one that is covariant, maybe with better performance as the one in Cats is not representative of what can be done)

2. Make Iterant covariant, for improving type inference (e.g. Iterant[+F[_], +A])
3. Better stack traces, preferably opt-in, possible solutions need to be investigated
4. ...

Binary breaking changes

Note that these aren't necessarily plans that will happen in less than a year, or even more — we are talking about a long time frame:

1. Rebuild Observable to use the Reactive Streams protocol internally, instead of the Future-based protocol that we have — but keep the Future-based one for consumption of streams and for ease of use — this might yield better performance in some cases, even if it's more challenging, but we can take inspiration from RxJava, while coming up with our own innovations and trying hard to keep source compatibility

• right now we piggyback the Reactive Streams on top of our Future-based one, but maybe it should be the other way around
• what I like about Reactive Streams is that the protocol for cancelation is clearer, whereas with Future you can end up with leaks due to not being able to unregister an "onComplete"
• one thing I really want, that RxJava doesn't do, is a stack safe flatMap implementation ... I'm not sure right now if this is a limitation of the underlying protocol or not
  • if we can make a stack safe flatMap, we can also implement Sync[Observable] and Async[Observable] at the very least 😉
  • we might be able to implement that stack safe flatMap even sooner, without breaking binary compatibility, but I remember trying it and being very challenging
• this needs experimentation before deciding whether it's a good idea or not

2. Refactor Iterant to not have so many operators that depend so much on Sync — e.g. operators should depend on Monad, MonadError, Defer, etc, as Sync should only be needed when we're suspending actual side effects

• I'm not sure if we can remove some of that dependence on Sync right now, without breaking binary compatibility
• whatever we can do sooner, we'll do sooner
• given the long time frame we are talking about, we might be talking about refactoring Iterant to use the future Cats-Effect 3.0 type class hierarchy

3. Remove all @deprecated symbols
4. ...

[Avasil]

My wishes are modest and could target 3.x.x line. I'm not in rush for 4.0.0 considering how long it took us to get to 3.0.0. And Cats-Effect 3.0 will most likely drain our resources.

• stack traces (already mentioned)
• Polymorphic Thread.interrupt support ( #983 )
• Overhaul https://monix.io/ Documentation - I feel like it is harder to navigate and looks worse than github sites or whatever is Akka doing. For instance, everything about Task is on one page and it's pretty "narrow" so there's a lot of scrolling. Might be nice to read on a phone but as our docs grow I think we should be able to split it into different sections without requiring a lot of clicks to get there. Font in code snippets could probably be better too. I'm not really knowledgeable about design but that's my subjective feelings. :D

Better stack traces, preferably opt-in, possible solutions need to be investigated

I'd say it should be opt-out. If there is a performance impact, most people won't notice it but the feature will be useful for everyone. but that's something we can discuss when we have anything

Provide bi-functor Task and Coeval implementations, possibly in their own sub-project? Not sure if we should, this needs to be discussed — I don't believe in the approach, but there are some people that do

I've grown to really like it. Well, I didn't actually use it in a project yet but I usually follow a standard pattern of F[Either[E, A]] where E is an error which is recoverable and F.raiseError is either not recoverable or when I handle all errors the same way. To me, Bifunctor is just taking this common pattern and embedding it. I don't buy performance argument but ergonomics are better (it's covariant and there is no need to wrap to EitherT to get shorcircuiting on E) and it allows cool stuff like returning UIO in an attempt.

ZIO is providing type aliases, something like Task[A] = ZIO[Any, Throwable, A] and IO[E, A] = ZIO[Any, E, A] to not require dealing with an error all the time. I don't know how well it does in practice, I assume it often infers to ZIO anyway. Different type aliases and return types in methods are extra complexity and something you need to learn even if you just want monofunctor.

I wonder how could we provide it without making it the default and still be compatible with "normal" Task.

As an alternative, maybe a nicer EitherT would be better (e.g. one that is covariant, maybe with better performance as the one in Cats is not representative of what can be done)

Covariant EitherT would be cool as well. Not sure if it's good enough in comparison to Bifunctor but it should be useful even if we have both. I'm attending ZIO Hackaton next week. Maybe there will be someone interested in contributing to Monix as well so I could guide this someone to create a prototype. It should be relatively simple if we disregard performance for now.

[alexandru]

I'm not in a hurry for a binary breaking 4.0.0 either, this is the far future we're talking about.

However we might prepare at least some stuff for it, on a separate branch maybe, because opportunities present themselves — for example if Cats or Cats-Effect break binary compatibility, then we can do so too. I don't predict that will happen soon though and it is my wish for Monix to be fairly stable going forward.

Good points on the monix.io documentation and integrating with Java's Thread.interrupt.

We'll open separate issues for these and for bifunctors and discuss implementation ideas there.

[Krever]

I will jump in only as a user but bifunctor and stacktraces are two game changers I would like to have. And similar to @Avasil, in my usecases performance is the last concern.

Other gamechanger for me was Local that allowed nice tracing implementation. Any improvements there would be nice but OTOH I didnt have any problems nor lacking features.

[MateuszKubuszok]

We are also using bifunctor approach in our project, which was achieved with things like type Result[A] = EitherT[Task, AppError, A], type DbResult[A] = Kleisli[Connection, Result, A] and it is reaallly troublesome often times, as it easily produces code like 1.asRight[AppError].pure[Task].lift (where .lift is some sad utility) among less experienced users. After a while you really miss something like

class Result[-I, +E, +A](val toKleisli: Kleisli[I, EitherT[Task, E, ?], A]) extends AnyVal {
  // tons of build-in methods
}

object Result {
  // tons of build in methods and Cats Effect type classes
}

I arrived too late to my current project so even if I try to migrate towards something more maintainable, quite a lot of code is already much more complex that it needs to be. It could be avoided if there was some obvious, recommended, build-in way of achieving that.

Similarly, to @Krever I don't really care about performance - in cases I have to handle it is a secondary concern. The biggest issue I try to fight is designing some one, obvious way of using the code that isn't confusing to other developers. As much as I find it very powerful to being able to compose things on my own, I know that it confuses the hell out of my juniors. It might be a contrib library - as long as it is "official" and "recommended" way of handling cases like this one, some next team could be spared the pain.

[alexandru]

Hi folks,

Coming back on the bifunctor implementation, lets talk about a more ergonomic alternative to EitherT first.

Can you share your thoughts on: https://github.com/monix/monix/issues/1027

[jvican]

Hey there! Great to see this discussion has started.

I'd like to say that performance is an important quality of Monix and it's quite important for me! I believe some of the items in this list are great (better stack traces are important), but I would love to see more PRs to make Monix even more performant than it is today.

I use Monix heavily in Bloop for fast scheduling of various tasks on machines with lots of cores. For example, we're heavy user of observer/observables to model many of the performance-sensitive build server internals. I haven't seen obvious performance bottlenecks with the current implementation --I haven't really profiled it either-- but the faster scheduling, context management and task creation are, the faster we will build Scala code. Building code fast is pretty much all Bloop aims to do and we rely heavily on Monix to not get in the way of that (and we're really satisfied with it so far!).

I'm planning to run some benchmarks when we upgrade to 3.0.0 to see if the latest merged optimizations have an effect in our particular use of Monix APIs. That should shed more light on whether 3.0.0 is fast enough and we would/would not benefit from further performance work. Not sure if it'll be an apple-to-apples comparison but will offer us some insights.

Until I run those experiments though, performance work always gets a :+1: from me -- I love it that the use of powerful Monix abstractions doesn't come with a (big )runtime cost. It allows people to use Monix in areas where it has traditionally not been able to model programs with such a level of abstraction (e.g. developer tools/compilers/who knows what else).

Thanks for all of the work you guys put into this library and hope that whatever you end up doing you keep pushing the limits of what Monix currently enables Scala developers to do and how they do it.

[alexandru]

Thanks @jvican, performance is always a priority.

And in case you see any regressions after an upgrade, it's a mistake that can be rectified.

[mdedetrich]

Personal priority (in order) for our use case at work is

1. Stack traces that work with Task/Coeval
2. Bifunctor (not EitherT, using that is quite painful)

[alexandru]

@mdedetrich 👍