Request for Implementation guidance on some HTTP/2 Use-cases

[joyfulnikhil]

Jetty Version: Jetty 12

Jetty Environment: core,http2

Java Version: 17

Question We have certain use cases for which we would like to override the default behavior of the jetty. It would be helpful if we could get some input from the community to achieve this by extending or implementing some listeners or adapter classes/Interfaces of Jetty 12.

We have a Reactive Spring boot-based app that sends and receives requests on HTTP2 protocol and uses HTTP/2 Jetty as our client to send out the requests.

Problem 1: We want to build a custom DNS resolver that would cache the FQDNs and return multiple IPs associated with the FQDN. our Approach: Create an AsyncResolver that implements SocketAddressResolver and plug it into the jetty client using setSocketAddressResolver()

Problem 2: Peg Metrics for some critical HTTP/2 frames received for debugging. Our Approach: We are planning to implement the interface "Session.Listener" an implement for methods like onReset etc and then plug this implementation using addBean()

Problem 3: Is there any in-built feature in the jetty client that can keep sending HTTP PING at a regular interval to all the active sessions? Is it possible to extend or implement some Jetty classes, which would help us to send HTTP Ping on every TCP connection that hasn't been active for more than 60 seconds and close the connection if there is no reply for ping for x number of times?

Problem 4: Is there a way to set request Timeout on the jetty client at the global level and should be able to update the timeout dynamically based on the request type? e.g. all requests to abc.com should have 4 seconds, to xyz.com have 5 seconds.

Problem 5: How can we iterate through all the active destinations and close one of the connections if needed?

Problem 6: How can we customize the round-robin connection pool behavior? i.e. If the FQDN abc.com resolves to 1.1.1.1, 2.2.2.2, and 3.3.3.3. My maxConnectionPerDestnation is 4. Now in this case, I would like the jetty to create connections like the below:

1st Request for abc.com: Create Connection-1 for 1.1.1.1 2nd Request for abc.com: Create Connection-1 for 2.2.2.2 3rd Request for abc.com: Create Connection-1 for 3.3.3.3 4th Request for abc.com: Create Connection-2 for 1.1.1.1 5th Request for abc.com: Create Connection-2 for 2.2.2.2

Once maxConnectionPerDestnation is reached for all 3 destinations, then keep sending the traffic in a round-robin manner.

Problem 7: Is there any Jetty class that can help us achieve the fallback mechanism? Suppose abc.com resolves to 1.1.1.1, 2.2.2.2 and 3.3.3.3. If the connection attempt fails for an IP(2.2.2.2), then instead of failing the request try using another active connection of 2.2.2.2 or other active connections of 1.1.1.1 or 3.3.3.3

Looking forward to some guidance from the community.

[sbordet]

You have a scenario where you want to use some high-level features like connection pooling, and some low-level features like accessing directly HTTP/2 specific functionalities.

Jetty provides both APIs, but separately. If you go high-level you typically won't have access to the low-level, and viceversa.

Below answers.

1. The approach is correct.
2. (only low-level) Just implement Session.Listener in your code, no need to add the implementation as a bean (it won't do anything other than being present in the component tree).
3. (only low-level) There is no built-in "ping" class because the logic is very application-specific, exactly like the one you described. It should be trivial to write it, using HTTP2Client.scheduler.
4. No, there is no way. Jetty won't know for which domain you want to change the request timeout, so this is an application concern.
5. Can you detail this use case? Why you want to do that? Connections are closed by Jetty via idle timeout, max time, or max usage.
6. RoundRobinConnectionPool tries to provide a round-robin behavior, but that is almost impossible as detailed in the class javadocs. Furthermore, HTTP/2 is multiplexed, so a new connection is not opened until the first one saturates all the concurrent streams. You are trying to use HTTP/2 as if it was HTTP/1.1, but they work differently.
7. Request retry in case of failure is something that applications should implement, because the logic behind the retry is different for every application.

For bullets 2 and 3 that need low-level access from the high-level, it is currently not available. We could allow adding a Session.Listener instance as a bean, and compose that with the internal one created by the implementation, but as I said, this is currently not available.

[joyfulnikhil]

Thanks a lot for your response @sbordet .

For the Problem 3 and 5, what I meant was that the existing httpClient Interface doesn't have methods using which I can access the Connection or Session Objects.

For Problem 3, where we want to send HTTP PING on all active sessions, how can we get hold of all the active sessions and start sending PING Fames on that?

For Problem 5, we would like to manually delete the connections associated with a destination. Can you help us understand how can we get hold of individual connections and close the connections. As of Jetty 12 , the ConnectionPool class doesn't expose methods to achieve this.

[sbordet]

For Problem 3, where we want to send HTTP PING on all active sessions, how can we get hold of all the active sessions and start sending PING Fames on that?

Why you want to do that? If a connection is idle, it is not used, and there should be no particular reason to keep it open to waste resources.

For Problem 5, we would like to manually delete the connections associated with a destination. Can you help us understand how can we get hold of individual connections and close the connections. As of Jetty 12 , the ConnectionPool class doesn't expose methods to achieve this.

Why you want to do that?

That's by design: the high-level APIs are designed so that applications do not worry about connections because they behave very differently across HTTP versions.

HTTP/2 connections are multiplexed, and used concurrently to send requests. In general, trying to close them results in multiplexed concurrent requests to be aborted, and that is why we offer automatic closing at the right time based on idle timeout, max usage time and max usage count: the implementation will know exactly when to close them safely.

If you want fine control over the connections, you may want to write your own ConnectionPool.

If you detail the reasons you are asking these features, we can be more precise helping you (or maybe there already is a solution).

[joyfulnikhil]

Hi @sbordet ,

We have a low latency ecosystem where the E2E transaction time(across various ms) is in the range of 10-20 ms. As we use HTTP/2 for transport, we would like to avoid the cost of initiating new TCP connections as much as possible. Hence, we would like use an TCP connection as much as possible and to achieve that we would like to send PING messages at regular intervals to keep the connection alive not forever, but atleast for a significant amount of time. Therefore, problem 3 is a very important use-case for us.

Regarding the problem 5, getting access to connections and terminating them, do we have any low level APIs to achieve that?

Apart from these, I have one clarification query regarding default behavior of Jetty client and one question for support.

Support Query: Is there a way access the sent and received header and data frames in HTTP/2? We would like to copy them and then proceed for processing.

Clarification Query: Suppose the request was for FQDN abc.com. Our custom DNS Resolver resolved it to [1.1.1.1 ,2.2.2.2,3.3.3.3]. The maxConnection per destination is set to 4 on the jetty client.

1. Will Jetty client create 3 destinations , each for 1.1.1.1 , 2.2.2.2 and 3.3.3.3?

2. Will Jetty client distribute the requets for abc.com to these 3 destinations?

3. Will Jetty client round robin the requests for abc.com in this manner of C1-1.1.1.1, C1-2.2.2.2, C1-3.3.3.3, C2-1.1.1.1 ....? If not , then can you please brief us how will the default jetty connection pool will work here?

4. Will Jetty client invoke the resolve() method of Custom DNS Resolver for each request?

5. How will jetty client internally handle if one of the IPs are changed? Prev it returned 1.1.1.1 ,2.2.2.2 and 3.3.3.3. Now it started returning 1.1.1.1 ,2.2.2.2 and 4.4.4.4?

As you mentioned previously, we may implement our own connection Pool. Can you throw some more light on which interfaces to look for and how it can be integrated to Jetty Client?

[sbordet]

As we use HTTP/2 for transport, we would like to avoid the cost of initiating new TCP connections as much as possible. Hence, we would like use an TCP connection as much as possible and to achieve that we would like to send PING messages at regular intervals to keep the connection alive not forever, but atleast for a significant amount of time.

You can do the same by enlarging the idle timeout, rather than having a short idle timeout and then having to use PING to keep it open.

Regarding problem 5, it is still not clear why you want to close the connections?

Suppose the request was for FQDN abc.com. Our custom DNS Resolver resolved it to [1.1.1.1 ,2.2.2.2,3.3.3.3]. The maxConnection per destination is set to 4 on the jetty client.

1. No, Jetty will create 1 destination only because it is based on the host name abc.com, which I assume it is what you will put in the request. You are overriding this behavior however, so you probably know better.
2. No, requests will never be distributed across destination. They are distributed across connections for the same destination only if you use RoundRobinConnectionPool or RandomConnectionPool.
3. As explained in the RoundRobinConnectionPool the behavior you want is technically impossible. It may resemble it in some way, but it's not guaranteed. RoundRobinConnectionPool tries to achieve that behavior, but it is not guaranteed.
4. No, the DNS resolution is only invoked when it is necessary to create a new connection, not for each request.
5. It does not handle it. If the Jetty client needs to create a new connection, it will query the DNS and use whatever address is returned by it.

Before you embark writing your own ConnectionPool, consider that you may not need it at all: so far everything you want to do seems possible without any code modifications (perhaps only the custom DNS resolver), just using configuration.

To sum up:

• Write your own SocketAddressResolver if you need to return different IPs (note that this may not be necessary if your DNS is already configured to return different IPs for every DNS request).
• Configure HttpClient.idleTimeout with large value.
• Configure HttpClient to use the RoundRobinConnectionPool as explained in the docs.
• Call ConnectionPool.preCreateConnections() to allocate all your connections.

That should be enough to achieve what you want.

[joyfulnikhil]

Closing it as we received the guidance needed. Thanks a lot @sbordet