Is it possible to apply retry strategy?

[gigi]

The context

Apply the retry, backoff strategy per RPC call

The question

Sometimes we get an error using your library as gRPC client and Armeria as gRPC server. We saw just failed unexpected logs failed: Connection refused.

gRPC client Armeria implementation has the ability to configure retries for injected client https://line.github.io/armeria/client-retry.html

gRPC has experimental proposal https://github.com/grpc/proposal/blob/master/A6-client-retries.md

Stacktraces and logs

Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: ADDRHERE/IPHERE:50051
	Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
	at io.grpc.netty.shaded.io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)`

Additional information

So, is it possible to achieve retry behavior using your library? Thank you

[ST-DDT]

Yes, that is theoretically possible using a GrpcChannelConfigurer and similar configurers.

However, I couldn't find any useful documentation on the retry in grpc-java, so I'm not sure what else needs to be done.

@Bean
public GrpcChannelConfigurer retryChannelConfigurer() {
    return (channelBuilder, name) -> {
        channelBuilder
                    .enableRetry()
                    .maxRetryAttempts(42);
        }
    };
}

This code only enables the retry subsystem, but it doesn't actually retry according to : https://stackoverflow.com/questions/55428539/grpc-java-setting-enableretry-and-maxretryattempt-is-not-retrying . Do you have good source-code/documentation for retry logic in grpc-java itself? The inject-able stub classes are final, so there is nothing I could do to implement the retry on a stub level.

[gigi]

Sorry that I did not reply for a while) Unfortunately, I don't have any good reference except we can find by links above :)

Some references The class that holds retry logic in GrpcChannelConfigurer https://github.com/grpc/grpc-java/blob/master/core/src/main/java/io/grpc/internal/AbstractManagedChannelImplBuilder.java#L138

Armeria wraps grpc stub somehow and uses client with retry logic https://github.com/line/armeria/blob/master/core/src/main/java/com/linecorp/armeria/client/retry/RetryingRpcClient.java#L97

Some discussion about service config. I can try to configure NettyChannelBuilder later https://github.com/grpc/grpc-java/issues/5724

[freezingWu]

the retry feature , i think it's easy to reuse the spring-retry module.

[helloworlde]

the retry feature , i think it's easy to reuse the spring-retry module.

@freezingWu Could you please provide a sample of spring-retry?

[freezingWu]

hi bro, here comes the spring-boot retry module sample ,pls track it from : https://my.oschina.net/wangxincj/blog/827221

[trustin]

@gigi Does Armeria's gRPC client work OK in this case? Out of curiosity, would you mind sharing your reasoning behind using grpc-spring-boot-starter's gRPC client while using Armeria on the other side?

[ST-DDT]

@trustin This library does not provide its own client. It just manages the injection of grpc-java clients/stubs. (I assume they are the same as those used by armeria, just without any additional wrapping).

There are many ways to implement retry logic. This question is about using grpc-java's own logic to avoid the problems with the recurring unavailability of the Armeria server.

[helloworlde]

Actually, grpc-java support retry from 1.20.0, so there need a entrance config to open it. Please reference RetryingHelloWorldClient

[ST-DDT]

Thanks for the info. I'll add a corresponding entry to the documentation for easy reference.

[ST-DDT]

Note to self:

• Example retrying_service_config.json
• Service-Config docs
• Service-Config model

[alex-damico]

Hi everyone, I want to enable grpc-retry. Unfortunately, I'm having problems. Do you have an example to share (with net.devh and spring boot)?

Here you will find my sample code

Thank you for your support

[ST-DDT]

Do you have an example to share (with net.devh and spring boot)?

Unfortunately not.

Here you will find my sample code

I'm not familiar with the retry config, but from this libraries point of view this looks good to me. There seems to be an issue with your grpc-service implementation though: https://github.com/alex-damico/spring-grpc/blob/main/src/main/java/com/example/springgrpc/service/GrpcMyServer.java#L18 You never close/complete the responseStreamObserver.

Unfortunately, I'm having problems.

Could you please be a little bit more specific? What is your issue? Is there a stacktrace? If you are unsure how to configure the retry, then grpc-java itself or SO might be suited better to answer your questions.

[helloworlde]

Hi everyone, I want to enable grpc-retry. Unfortunately, I'm having problems. Do you have an example to share (with net.devh and spring boot)?

Here you will find my sample code

Thank you for your support

You can reference my sample code grpc-java-sample/retry-policy, even it is not in Spring, but the config is same.

[vikrantch-hk]

@gigi were you able to achieve this with GrpcChannelConfigurer changes

[anjeyy]

Hi everyone, i might be missing some infos. Has anyone tried resilienjce4j with the retry strategy ?

So the actual act of retrying can be put in a higher ordered function which has a retry strategy.

[guybmayor]

Hey does anyone success to use grpc retry strategy with GrpcChannelConfigurer or any other way?

[onyn]

@guybmayor, you can configure retries using defaultServiceConfig method of ManagedChannelBuilder. See example and config used in that example.

With service config it's also possible to configure load balancing.