flagd
flagd copied to clipboard
open-feature
[BUG] Frequently we are encountering intermittent error with Flagd while trying to fetch flags from localhost AWS AppConfig
Observed behavior
We are running flagd as a service in our main app container.
We are storing flags over AWS AppConfig and fetching flags via localhost appconfig agent. Flagd is watching this localhost agent during boot of application, but sometime it fails to read message. I have attached error logs as they were too long.
We are using flagd image ghcr.io/open-feature/flagd:v0.12.2 and below open-feature and flagd dependencies. api 'dev.openfeature:sdk:1.14.1' api 'dev.openfeature.contrib.providers:flagd:0.11.3'
Expected Behavior
No response
Steps to reproduce
No response
can you please attach the important log lines directly to the issue rather than attaching a file, I am just not feeling comfortable downloading a file from somebody from the internet (security reasons). Only the important log lines are needed, thank you
Hello @aepfli Here are main error logs:
{"@timestamp":"2025-03-13T06:58:20.361Z","ecs.version":"1.2.0","log.level":"INFO","message":"Error initializing metadata, retrying in 0ms","process.thread.name":"Thread-6","log.logger":"dev.openfeature.contrib.providers.flagd.resolver.process.storage.connector.grpc.GrpcStreamConnector","error.type":"io.grpc.StatusRuntimeException","error.message":"CANCELLED: Failed to read message.","error.stack_trace":[{"class":"io.grpc.stub.ClientCalls","method":"toStatusRuntimeException","file":"ClientCalls.java","line":268},{"class":"io.grpc.stub.ClientCalls","method":"getUnchecked","file":"ClientCalls.java","line":249},{"class":"io.grpc.stub.ClientCalls","method":"blockingUnaryCall","file":"ClientCalls.java","line":167},{"class":"dev.openfeature.flagd.grpc.sync.FlagSyncServiceGrpc$FlagSyncServiceBlockingStub","method":"getMetadata","file":"FlagSyncServiceGrpc.java","line":290},{"class":"dev.openfeature.contrib.providers.flagd.resolver.process.storage.connector.grpc.GrpcStreamConnector","method":"observeEventStream","file":"GrpcStreamConnector.java","line":154},{"class":"dev.openfeature.contrib.providers.flagd.resolver.process.storage.connector.grpc.GrpcStreamConnector","method":"lambda$init$0","file":"GrpcStreamConnector.java","line":69},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":840}]}
{"@timestamp":"2025-03-13T06:58:20.418Z","ecs.version":"1.2.0","log.level":"ERROR","message":"Error initializing stream, retrying in 20000ms","process.thread.name":"Thread-6","log.logger":"dev.openfeature.contrib.providers.flagd.resolver.process.storage.connector.grpc.GrpcStreamConnector","error.type":"io.grpc.StatusRuntimeException","error.message":"CANCELLED: Failed to read message.","error.stack_trace":[{"class":"io.grpc.Status","method":"asRuntimeException","file":"Status.java","line":532},{"class":"io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter","method":"onClose","file":"ClientCalls.java","line":481},{"class":"io.grpc.internal.ClientCallImpl","method":"closeObserver","file":"ClientCallImpl.java","line":564},{"class":"io.grpc.internal.ClientCallImpl","method":"access$100","file":"ClientCallImpl.java","line":72},{"class":"io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed","method":"runInternal","file":"ClientCallImpl.java","line":729},{"class":"io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed","method":"runInContext","file":"ClientCallImpl.java","line":710},{"class":"io.grpc.internal.ContextRunnable","method":"run","file":"ContextRunnable.java","line":37},{"class":"io.grpc.internal.SerializingExecutor","method":"run","file":"SerializingExecutor.java","line":133},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","file":"ThreadPoolExecutor.java","line":1136},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","file":"ThreadPoolExecutor.java","line":635},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":840}]}
{"@timestamp":"2025-03-13T06:58:20.418Z","ecs.version":"1.2.0","log.level":"ERROR","message":"Error initializing metadata, retrying in 20000ms","process.thread.name":"Thread-6","log.logger":"dev.openfeature.contrib.providers.flagd.resolver.process.storage.connector.grpc.GrpcStreamConnector","error.type":"io.grpc.StatusRuntimeException","error.message":"CANCELLED: Failed to read message.","error.stack_trace":[{"class":"io.grpc.stub.ClientCalls","method":"toStatusRuntimeException","file":"ClientCalls.java","line":268},{"class":"io.grpc.stub.ClientCalls","method":"getUnchecked","file":"ClientCalls.java","line":249},{"class":"io.grpc.stub.ClientCalls","method":"blockingUnaryCall","file":"ClientCalls.java","line":167},{"class":"dev.openfeature.flagd.grpc.sync.FlagSyncServiceGrpc$FlagSyncServiceBlockingStub","method":"getMetadata","file":"FlagSyncServiceGrpc.java","line":290},{"class":"dev.openfeature.contrib.providers.flagd.resolver.process.storage.connector.grpc.GrpcStreamConnector","method":"observeEventStream","file":"GrpcStreamConnector.java","line":154},{"class":"dev.openfeature.contrib.providers.flagd.resolver.process.storage.connector.grpc.GrpcStreamConnector","method":"lambda$init$0","file":"GrpcStreamConnector.java","line":69},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":840}]}
{"@timestamp":"2025-03-13T06:58:29.871Z","ecs.version":"1.2.0","log.level":"ERROR","message":"Exception when initializing feature provider dev.openfeature.contrib.providers.flagd.FlagdProvider","process.thread.name":"main","log.logger":"dev.openfeature.sdk.ProviderRepository","error.type":"dev.openfeature.sdk.exceptions.GeneralError","error.message":"Deadline exceeded. Condition did not complete within the 10000 deadline","error.stack_trace":[{"class":"dev.openfeature.contrib.providers.flagd.resolver.common.Util","method":"busyWaitAndCheck","file":"Util.java","line":29},{"class":"dev.openfeature.contrib.providers.flagd.resolver.process.InProcessResolver","method":"init","file":"InProcessResolver.java","line":100},{"class":"dev.openfeature.contrib.providers.flagd.FlagdProvider","method":"initialize","file":"FlagdProvider.java","line":87},{"class":"dev.openfeature.sdk.FeatureProviderStateManager","method":"initialize","file":"FeatureProviderStateManager.java","line":26},{"class":"dev.openfeature.sdk.ProviderRepository","method":"initializeProvider","file":"ProviderRepository.java","line":205},{"class":"dev.openfeature.sdk.ProviderRepository","method":"prepareAndInitializeProvider","file":"ProviderRepository.java","line":176},{"class":"dev.openfeature.sdk.ProviderRepository","method":"setProvider","file":"ProviderRepository.java","line":122},{"class":"dev.openfeature.sdk.OpenFeatureAPI","method":"setProviderAndWait","file":"OpenFeatureAPI.java","line":214},{"class":"com.amway.commerce.servicechassis.toggle.appconfigdata.flagd.config.FlagdProviderConfig","method":"openFeatureApi","file":"FlagdProviderConfig.java","line":50},{"class":"com.amway.commerce.servicechassis.toggle.appconfigdata.flagd.config.FlagdProviderConfig$$SpringCGLIB$$0","method":"CGLIB$openFeatureApi$0","file":"<generated>","line":-1},{"class":"com.amway.commerce.servicechassis.toggle.appconfigdata.flagd.config.FlagdProviderConfig$$SpringCGLIB$$FastClass$$1","method":"invoke","file":"<generated>","line":-1},{"class":"org.springframework.cglib.proxy.MethodProxy","method":"invokeSuper","file":"MethodProxy.java","line":258},{"class":"org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor","method":"intercept","file":"ConfigurationClassEnhancer.java","line":348},{"class":"com.amway.commerce.servicechassis.toggle.appconfigdata.flagd.config.FlagdProviderConfig$$SpringCGLIB$$0","method":"openFeatureApi","file":"<generated>","line":-1},{"class":"jdk.internal.reflect.NativeMethodAccessorImpl","method":"invoke0","file":"NativeMethodAccessorImpl.java","line":-2},{"class":"jdk.internal.reflect.NativeMethodAccessorImpl","method":"invoke","file":"NativeMethodAccessorImpl.java","line":77},{"class":"jdk.internal.reflect.DelegatingMethodAccessorImpl","method":"invoke","file":"DelegatingMethodAccessorImpl.java","line":43},{"class":"java.lang.reflect.Method","method":"invoke","file":"Method.java","line":569},{"class":"org.springframework.beans.factory.support.SimpleInstantiationStrategy","method":"lambda$instantiate$0","file":"SimpleInstantiationStrategy.java","line":171},{"class":"org.springframework.beans.factory.support.SimpleInstantiationStrategy","method":"instantiateWithFactoryMethod","file":"SimpleInstantiationStrategy.java","line":88},{"class":"org.springframework.beans.factory.support.SimpleInstantiationStrategy","method":"instantiate","file":"SimpleInstantiationStrategy.java","line":168},{"class":"org.springframework.beans.factory.support.ConstructorResolver","method":"instantiate","file":"ConstructorResolver.java","line":653},{"class":"org.springframework.beans.factory.support.ConstructorResolver","method":"instantiateUsingFactoryMethod","file":"ConstructorResolver.java","line":645},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"instantiateUsingFactoryMethod","file":"AbstractAutowireCapableBeanFactory.java","line":1361},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"createBeanInstance","file":"AbstractAutowireCapableBeanFactory.java","line":1191},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"doCreateBean","file":"AbstractAutowireCapableBeanFactory.java","line":563},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"createBean","file":"AbstractAutowireCapableBeanFactory.java","line":523},{"class":"org.springframework.beans.factory.support.AbstractBeanFactory","method":"lambda$doGetBean$0","file":"AbstractBeanFactory.java","line":336},{"class":"org.springframework.beans.factory.support.DefaultSingletonBeanRegistry","method":"getSingleton","file":"DefaultSingletonBeanRegistry.java","line":307},{"class":"org.springframework.beans.factory.support.AbstractBeanFactory","method":"doGetBean","file":"AbstractBeanFactory.java","line":334},{"class":"org.springframework.beans.factory.support.AbstractBeanFactory","method":"getBean","file":"AbstractBeanFactory.java","line":199},{"class":"org.springframework.beans.factory.support.DefaultListableBeanFactory","method":"doResolveDependency","file":"DefaultListableBeanFactory.java","line":1573},{"class":"org.springframework.beans.factory.support.DefaultListableBeanFactory","method":"resolveDependency","file":"DefaultListableBeanFactory.java","line":1519},{"class":"org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement","method":"resolveFieldValue","file":"AutowiredAnnotationBeanPostProcessor.java","line":785},{"class":"org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement","method":"inject","file":"AutowiredAnnotationBeanPostProcessor.java","line":768},{"class":"org.springframework.beans.factory.annotation.InjectionMetadata","method":"inject","file":"InjectionMetadata.java","line":146},{"class":"org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor","method":"postProcessProperties","file":"AutowiredAnnotationBeanPostProcessor.java","line":509},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"populateBean","file":"AbstractAutowireCapableBeanFactory.java","line":1445},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"doCreateBean","file":"AbstractAutowireCapableBeanFactory.java","line":600},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"createBean","file":"AbstractAutowireCapableBeanFactory.java","line":523},{"class":"org.springframework.beans.factory.support.AbstractBeanFactory","method":"lambda$doGetBean$0","file":"AbstractBeanFactory.java","line":336},{"class":"org.springframework.beans.factory.support.DefaultSingletonBeanRegistry","method":"getSingleton","file":"DefaultSingletonBeanRegistry.java","line":307},{"class":"org.springframework.beans.factory.support.AbstractBeanFactory","method":"doGetBean","file":"AbstractBeanFactory.java","line":334},{"class":"org.springframework.beans.factory.support.AbstractBeanFactory","method":"getBean","file":"AbstractBeanFactory.java","line":199},{"class":"org.springframework.beans.factory.config.DependencyDescriptor","method":"resolveCandidate","file":"DependencyDescriptor.java","line":254},{"class":"org.springframework.beans.factory.support.DefaultListableBeanFactory","method":"doResolveDependency","file":"DefaultListableBeanFactory.java","line":1631},{"class":"org.springframework.beans.factory.support.DefaultListableBeanFactory","method":"resolveDependency","file":"DefaultListableBeanFactory.java","line":1519},{"class":"org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement","method":"resolveFieldValue","file":"AutowiredAnnotationBeanPostProcessor.java","line":785},{"class":"org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement","method":"inject","file":"AutowiredAnnotationBeanPostProcessor.java","line":768},{"class":"org.springframework.beans.factory.annotation.InjectionMetadata","method":"inject","file":"InjectionMetadata.java","line":146},{"class":"org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor","method":"postProcessProperties","file":"AutowiredAnnotationBeanPostProcessor.java","line":509},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"populateBean","file":"AbstractAutowireCapableBeanFactory.java","line":1445},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"doCreateBean","file":"AbstractAutowireCapableBeanFactory.java","line":600},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"createBean","file":"AbstractAutowireCapableBeanFactory.java","line":523},{"class":"org.springframework.beans.factory.support.AbstractBeanFactory","method":"lambda$doGetBean$0","file":"AbstractBeanFactory.java","line":336},{"class":"org.springframework.beans.factory.support.DefaultSingletonBeanRegistry","method":"getSingleton","file":"DefaultSingletonBeanRegistry.java","line":307},{"class":"org.springframework.beans.factory.support.AbstractBeanFactory","method":"doGetBean","file":"AbstractBeanFactory.java","line":334},{"class":"org.springframework.beans.factory.support.AbstractBeanFactory","method":"getBean","file":"AbstractBeanFactory.java","line":199},{"class":"org.springframework.beans.factory.config.DependencyDescriptor","method":"resolveCandidate","file":"DependencyDescriptor.java","line":254},{"class":"org.springframework.beans.factory.support.DefaultListableBeanFactory","method":"doResolveDependency","file":"DefaultListableBeanFactory.java","line":1631},{"class":"org.springframework.beans.factory.support.DefaultListableBeanFactory","method":"resolveDependency","file":"DefaultListableBeanFactory.java","line":1519},{"class":"org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement","method":"resolveFieldValue","file":"AutowiredAnnotationBeanPostProcessor.java","line":785},{"class":"org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement","method":"inject","file":"AutowiredAnnotationBeanPostProcessor.java","line":768},{"class":"org.springframework.beans.factory.annotation.InjectionMetadata","method":"inject","file":"InjectionMetadata.java","line":146},{"class":"org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor","method":"postProcessProperties","file":"AutowiredAnnotationBeanPostProcessor.java","line":509},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"populateBean","file":"AbstractAutowireCapableBeanFactory.java","line":1445},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"doCreateBean","file":"AbstractAutowireCapableBeanFactory.java","line":600},{"class":"org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory","method":"createBean","file":"AbstractAutowireCapableBeanFactory.java","line":523},{"class":"org.springframework.beans.factory.support.AbstractBeanFactory","method":"lambda$doGetBean$0","file":"AbstractBeanFactory.java","line":336},{"class":"org.springframework.beans.factory.support.DefaultSingletonBeanRegistry","method":"getSingleton","file":"DefaultSingletonBeanRegistry.java","line":307},{"class":"org.springframework.beans.factory.support.AbstractBeanFactory","method":"doGetBean","file":"AbstractBeanFactory.java","line":334},{"class":"org.springframework.beans.factory.support.AbstractBeanFactory","method":"getBean","file":"AbstractBeanFactory.java","line":199},{"class":"org.springframework.beans.factory.support.DefaultListableBeanFactory","method":"instantiateSingleton","file":"DefaultListableBeanFactory.java","line":1122},{"class":"org.springframework.beans.factory.support.DefaultListableBeanFactory","method":"preInstantiateSingleton","file":"DefaultListableBeanFactory.java","line":1093},{"class":"org.springframework.beans.factory.support.DefaultListableBeanFactory","method":"preInstantiateSingletons","file":"DefaultListableBeanFactory.java","line":1030},{"class":"org.springframework.context.support.AbstractApplicationContext","method":"finishBeanFactoryInitialization","file":"AbstractApplicationContext.java","line":987},{"class":"org.springframework.context.support.AbstractApplicationContext","method":"refresh","file":"AbstractApplicationContext.java","line":627},{"class":"org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext","method":"refresh","file":"ServletWebServerApplicationContext.java","line":146},{"class":"org.springframework.boot.SpringApplication","method":"refresh","file":"SpringApplication.java","line":752},{"class":"org.springframework.boot.SpringApplication","method":"refreshContext","file":"SpringApplication.java","line":439},{"class":"org.springframework.boot.SpringApplication","method":"run","file":"SpringApplication.java","line":318},{"class":"org.springframework.boot.SpringApplication","method":"run","file":"SpringApplication.java","line":1361},{"class":"org.springframework.boot.SpringApplication","method":"run","file":"SpringApplication.java","line":1350},{"class":"com.amway.commerce.cartbizservice.app.CartBizServiceApplication","method":"main","file":"CartBizServiceApplication.java","line":25},{"class":"jdk.internal.reflect.NativeMethodAccessorImpl","method":"invoke0","file":"NativeMethodAccessorImpl.java","line":-2},{"class":"jdk.internal.reflect.NativeMethodAccessorImpl","method":"invoke","file":"NativeMethodAccessorImpl.java","line":77},{"class":"jdk.internal.reflect.DelegatingMethodAccessorImpl","method":"invoke","file":"DelegatingMethodAccessorImpl.java","line":43},{"class":"java.lang.reflect.Method","method":"invoke","file":"Method.java","line":569},{"class":"org.springframework.boot.loader.launch.Launcher","method":"launch","file":"Launcher.java","line":102},{"class":"org.springframework.boot.loader.launch.Launcher","method":"launch","file":"Launcher.java","line":64},{"class":"org.springframework.boot.loader.launch.JarLauncher","method":"main","file":"JarLauncher.java","line":40}]}
{"@timestamp":"2025-03-13T06:58:29.876Z","ecs.version":"1.2.0","log.level":"WARN","message":"Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cartController': Unsatisfied dependency expressed through field 'cartService': Error creating bean with name 'cartServiceImpl': Unsatisfied dependency expressed through field 'featureService': Error creating bean with name 'featureServiceImpl': Unsatisfied dependency expressed through field 'openFeatureApi': Error creating bean with name 'openFeatureApi' defined in class path resource [com/amway/commerce/servicechassis/toggle/appconfigdata/flagd/config/FlagdProviderConfig.class]: Failed to instantiate [dev.openfeature.sdk.OpenFeatureAPI]: Factory method 'openFeatureApi' threw exception with message: Deadline exceeded. Condition did not complete within the 10000 deadline","process.thread.name":"main","log.logger":"org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext"}
Hello @aepfli
Update: We encountered this issue with [email protected] and gradle dependencies. implementation 'dev.openfeature:sdk:1.12.2' implementation 'dev.openfeature.contrib.providers:flagd:0.10.0'
can you please elaborate how we can reproduce this behavior, without steps to reproduce this is like a searching for a needle in the haystack. Without a basic setup, of what we need to do, or take into account, it will be really hard to help/support you
We are running app config agent as a sidecar container and flagd as a service within our main container.
On our main container startup we are configuring bean for FlagdProvider as you can see in above screenshot. And we are passing localhost appconfig agent as URI which flagd will be watching to fetch feature flags. As i can see in logs that during boot, flagd makes a request to appconfig agent to fetch profile data. AppConfig agent is able to fetch data but after passing it to flagd, flagd starts reporting error that it is not able to read message.
logs from appconfig agent:
[appconfig agent] 2025/03/13 09:22:46 INFO AppConfig Agent 2.0.4348
[appconfig agent] 2025/03/13 09:22:46 INFO serving on localhost:2772
[appconfig agent] 2025/03/13 09:22:52 INFO retrieved initial data for 'ngcom-global-dev:default:purchase' in 134ms
[appconfig agent] 2025/03/13 09:22:52 INFO retrieved initial data for 'ngcom-global-dev:default:cart' in 162ms
@ashish20790 you say you see this issue "intermittently". Could you elaborate on the timing? This might help us understand what's happening.
Do things mostly work, but you randomly get this error?
Does the error appear with any regular frequency?
Does the error resolve itself after some time?
Do you see anything unusual in flagd's (not your java process, I mean /usr/bin/flagd-build) logs? It could be cancelling responses due to errors.
One last thing... Can you report the version of protobuf-Java you're using? You might be getting it from multiple sources depending on other dependencies in your app.
Also, could you try setting io.grpc and dev.openfeature.contrib.providers log levels to DEBUG and then provide debug logs around these errors?
@ashish20790 you say you see this issue "intermittently". Could you elaborate on the timing? This might help us understand what's happening.
Do things mostly work, but you randomly get this error?
Does the error appear with any regular frequency?
Does the error resolve itself after some time?
Do you see anything unusual in flagd's (not your java process, I mean
/usr/bin/flagd-build) logs? It could be cancelling responses due to errors.One last thing... Can you report the version of protobuf-Java you're using? You might be getting it from multiple sources depending on other dependencies in your app.
Well it happens sometimes, can't say about frequency. For protobuf-java, we are using 3.25.6
OK. Please see what you can do about checking the flagd process logs and also the debug logs from the classes I mentioned. I cannot reproduce this in a simple setup, so I think it's something environmental.