ktor Using existing event-loop group

Using existing event-loop group

Open dave08 opened this issue 6 years ago • 14 comments

Using components from libraries that have their own event loop causes the creation of an extra (unecessary) event loop. There are currently two possibilities: the library itself using Ktor's event loop, or providing the means to re-use the library's implemented event loop.

In the case of Vert.x, its event loop group can be retrieved by using vertx.nettyEventLoopGroup(), if it could be re-used by Ktor. Or for Ktor to pass its own event loop, it should pass a Transport that is responsible for building Vert.x event loop groups (because with native transport the even loop impl is diffrent).

In vert.x, in 3.5.1 there is a feature planned, for reuse of an existing event loop group and reuse for instance ktor event loop, if ktor exposes its event loop group.

Dec 31 '17 14:12 dave08

I think vert.x host for ktor is the way to go here. We will still need HTTP message processing and all the related stuff. I can do a somewhat working stub, if you and/or @vietj can complete it.

Jan 01 '18 10:01 orangy

You mean that ktor would expose its eventloopgroup, and then the work would be wiring Vert.x to use it?

Jan 01 '18 17:01 dave08

@orangy I think this is important, since Ktor doesn't yet have any real async db, file io or event bus, etc.. components, and the true power of coroutines doesn't lie in blocking tasks.. so in the meantime it would be a very nice replacement.. I would just like to know how to get this going, I'm not too knowledgeable about these particular implementation details... so I hope I can help.

Jan 03 '18 17:01 dave08

Yes, I agree, I bookmarked it to think right after I return from snowboarding vacation :)

Jan 03 '18 18:01 orangy

I think vert.x host for ktor is the way to go here can you elaborate ? I'm not sure it means that we simply share event loop between Vert.x and Ktor (allowing to use Ktor HTTP server implementation based on Netty and use Vert.x client ecosystem) or if there would be a full implementation of Ktor SPI on top of Vert.x ?

I'm happy to contribute either way

Jan 04 '18 14:01 vietj

I was thinking about hosting ktor app as a verticle, like a servlet in an app container.

Jan 04 '18 18:01 orangy

I think that kind of defeats the purpose.. the idea for me is that in Docker Swarm, I get a lot of the Verticle functionality... for certain services, Verticles are more flexable, being able to code when deployment is done and handle failures smartly. But certain services need to be very light weight w/o all the extra infrastucture complexities... Docker Swarm handles scaling and orchestrating the scaled services in the cluster using metadata on expected resource consumption... Ktor is great as such a light weight solution, and Vert.x compliments w/ all the async services that are missing...

Jan 04 '18 21:01 dave08

I'm slightly lost here. Can you give an example of what you want to work? Like "inside a ktor handler, call this vert.x function" or something easy to understand without knowing much vert.x :)

Jan 29 '18 21:01 orangy

I'm facing a similar situation. I've built my own Netty framework for WebSockets, with some Web features that I would happily replace with Ktor. I want Ktor to reuse the NioEventLoop that I already set up for my Netty stuff. For now the only way to go with Ktor-1.0.0 is to subclass NettyApplicationEngine to force my NioEventLoop in. A better solution would be to take one or more NioEventLoop as parameters of NettyApplicationEngine.Configuration.

Netty and Project-reactor can plug themselves into an existing Executor, and Netty can even reuse some NioEventLoop across servers. So there is no waste of threads in one JVM. Ktor could offer this feature I guess.

Dec 12 '18 09:12 caillette

It's worse than I thought. Kotlin forbids to access internal classes from a class whose source file is in another module. So my CustomNettyApplicationEngine (basically a copy-paste of NettyApplicationEngine) can't use NettyChannelInitializer. Until Ktor supports a foreign EventLoopGroup my only option is to fork the git repository and do bizarre things with my project build.

Dec 13 '18 10:12 caillette

@caillette A workaround is added as per e6c058d