Conjure Java Runtime Stuck On Internal Butterfly Product

[jeremyk-91]

@gsheasby reported an issue where a CJR client in the TimestampManagementService on atlasdb 0.173.2 in conjunction with a latest TimeLock could get stuck during initialization. It seems that we are getting a RetryOther exception thrown out.

com.palantir.async.initializer.AsyncInitializer: Failed to initialize {} on the attempt {} (className: TimeLockMigrator, numberOfAttempts: 4, throwable3_redirectTo: BOX_6/timelock/api) (throwable0_message: Could not contact the Timelock Server., throwable1_message: Exceeded the maximum number of allowed redirects: {requestUrl=BOX_8/timelock/api/SERVICE/timestamp-management/ping} executing GET BOX_6/timelock/api/SERVICE/timestamp-management/ping, throwable2_requestUrl: BOX_8/timelock/api/SERVICE/timestamp-management/ping)
com.palantir.common.exception.AtlasDbDependencyException: Could not contact the Timelock Server.
    at com.palantir.atlasdb.factory.startup.TimeLockMigrator.tryInitialize(TimeLockMigrator.java:77)
/* snip */
Caused by: feign.RetryableException: Exceeded the maximum number of allowed redirects: {requestUrl=BOX_8:8421/timelock/api/SERVICE/timestamp-management/ping} executing GET BOX_6/timelock/api/SERVICE/timestamp-management/ping
    at feign.FeignException.errorExecuting(FeignException.java:67)
    at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:102)
    at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76)
    at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
    at com.sun.proxy.$Proxy73.ping(Unknown Source)
    at jdk.internal.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:566)
    at com.palantir.conjure.java.ext.refresh.RefreshableProxyInvocationHandler.handleInvocation(RefreshableProxyInvocationHandler.java:61)
    at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:86)
    at com.sun.proxy.$Proxy73.ping(Unknown Source)
    at jdk.internal.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:566)
    at com.palantir.atlasdb.http.v2.FastFailoverProxy.singleInvocation(FastFailoverProxy.java:88)
    at com.palantir.atlasdb.http.v2.FastFailoverProxy.handleInvocation(FastFailoverProxy.java:78)
    at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:86)
    at com.sun.proxy.$Proxy73.ping(Unknown Source)
    at com.palantir.tritium.proxy.InstrumentedTimestampManagementRpcClient$7.ping(Unknown Source)
    at jdk.internal.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:566)
    at com.palantir.common.proxy.ExperimentRunningProxy.handleInvocation(ExperimentRunningProxy.java:91)
    at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:86)
    at com.sun.proxy.$Proxy73.ping(Unknown Source)
    at com.palantir.timestamp.RemoteTimestampManagementAdapter.ping(RemoteTimestampManagementAdapter.java:35)
    at com.palantir.timestamp.DelegatingManagedTimestampService.ping(DelegatingManagedTimestampService.java:57)
    at com.palantir.atlasdb.factory.startup.TimeLockMigrator.tryInitialize(TimeLockMigrator.java:75)
    ... 11 more
Caused by: com.palantir.logsafe.exceptions.SafeIoException: Exceeded the maximum number of allowed redirects
    at com.palantir.conjure.java.okhttp.RemotingOkHttpCall$5.visit(RemotingOkHttpCall.java:401)
    at com.palantir.conjure.java.okhttp.RemotingOkHttpCall$5.visit(RemotingOkHttpCall.java:359)
    at com.palantir.conjure.java.api.errors.QosException$RetryOther.accept(QosException.java:118)
    at com.palantir.conjure.java.okhttp.RemotingOkHttpCall$4.onResponse(RemotingOkHttpCall.java:292)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:216)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    ... 3 more
Caused by: com.palantir.conjure.java.api.errors.QosException$RetryOther: RetryOther: Requesting retry
    at com.palantir.conjure.java.api.errors.QosException.retryOther(QosException.java:72)
    at com.palantir.conjure.java.QosExceptionResponseMapper.map308(QosExceptionResponseMapper.java:65)
    at com.palantir.conjure.java.QosExceptionResponseMapper.mapResponseCode(QosExceptionResponseMapper.java:46)
    at com.palantir.conjure.java.okhttp.QosExceptionResponseHandler.handle(QosExceptionResponseHandler.java:32)
    at com.palantir.conjure.java.okhttp.RemotingOkHttpCall$4.onResponse(RemotingOkHttpCall.java:290)
    ... 5 more```