HTTP Status: 500 Internal Server Error

[Tiagodge]

Bug Report

I installed the open telemetry-demo on my OpenShift Cluster and I am receiving this 500 Server Error. Once I am not running locally I needed to create a load balancer to connect externally.

Browser error trying to access the page:

rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.21.79.231:8080: connect: connection refused" could not retrieve cart main.(*frontendServer).viewCartHandler /usr/src/app/handlers.go:280 main.instrumentHandler.func1 /usr/src/app/middleware.go:120 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2047 go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux.traceware.ServeHTTP /go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/[email protected]/mux.go:145 github.com/gorilla/mux.(*Router).ServeHTTP /go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 main.(*logHandler).ServeHTTP /usr/src/app/middleware.go:94 main.ensureSessionID.func1 /usr/src/app/middleware.go:148 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2047 net/http.serverHandler.ServeHTTP /usr/local/go/src/net/http/server.go:2879 net/http.(*conn).serve /usr/local/go/src/net/http/server.go:1930 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1581

Symptom

I am having a CrashLoopBackOff error too: my-otel-demo-cart-service

1 Connecting to Redis: my-otel-demo-redis:6379,ssl=false,allowAdmin=true,abortConnect=false 2 Wasn't able to connect to redis 3 Unhandled exception. System.ApplicationException: Wasn't able to connect to redis 4 at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 89 5 at cartservice.cartstore.RedisCartStore.InitializeAsync() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 62 6 at cartservice.Startup.ConfigureServices(IServiceCollection services) in /usr/src/app/src/Startup.cs:line 38 7 at System.RuntimeMethodHandle.InvokeMethod(Object , Span`1& , Signature , Boolean , Boolean ) 8 at System.Reflection.RuntimeMethodInfo.Invoke(Object , BindingFlags , Binder , Object[] , CultureInfo ) 9 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services) 10 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection) 11 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services) 12 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services) 13 at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services, Object instance) 14 at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.<UseStartup>b__0(HostBuilderContext context, IServiceCollection services) 15 at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider() 16 at Microsoft.Extensions.Hosting.HostBuilder.Build() 17 at Program.<Main>$(String[] args) in /usr/src/app/src/Program.cs:line 19

Redis is working well: 1 1:C 09 Aug 2022 13:40:02.649 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 2 1:C 09 Aug 2022 13:40:02.657 # Redis version=7.0.4, bits=64, commit=00000000, modified=0, pid=1, just started 3 1:C 09 Aug 2022 13:40:02.658 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf 4 1:M 09 Aug 2022 13:40:02.658 * monotonic clock: POSIX clock_gettime 5 1:M 09 Aug 2022 13:40:02.663 * Running mode=standalone, port=6379. 6 1:M 09 Aug 2022 13:40:02.664 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 7 1:M 09 Aug 2022 13:40:02.664 # Server initialized 8 1:M 09 Aug 2022 13:40:02.664 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to check if the system's clocksource isn't degrading performance. 9 1:M 09 Aug 2022 13:40:02.665 * Ready to accept connections

[julianocosta89]

Hello @Tiagodge, could you try again with the new released version?

[Tiagodge]

Hello @julianocosta89 , I just tried and I continue to receive a crashloopbackoff on my-otel-demo-cart-service-855db6779c-ngrlt :

I removed the old one:

helm repo remove open-telemetry

And installed again:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

helm install my-otel-demo open-telemetry/opentelemetry-demo

The error:

1 Connecting to Redis: my-otel-demo-redis:6379,ssl=false,allowAdmin=true,abortConnect=false 2 Wasn't able to connect to redis 3 Unhandled exception. System.ApplicationException: Wasn't able to connect to redis 4 at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 89 5 at cartservice.cartstore.RedisCartStore.InitializeAsync() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 62 6 at cartservice.Startup.ConfigureServices(IServiceCollection services) in /usr/src/app/src/Startup.cs:line 38 7 at System.RuntimeMethodHandle.InvokeMethod(Object , Span`1& , Signature , Boolean , Boolean ) 8 at System.Reflection.RuntimeMethodInfo.Invoke(Object , BindingFlags , Binder , Object[] , CultureInfo ) 9 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services) 10 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection) 11 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services) 12 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services) 13 at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services, Object instance) 14 at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.<UseStartup>b__0(HostBuilderContext context, IServiceCollection services) 15 at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider() 16 at Microsoft.Extensions.Hosting.HostBuilder.Build() 17 at Program.<Main>$(String[] args) in /usr/src/app/src/Program.cs:line 19

[fatsheep9146]

Could your check the status of the redis service?

[Tiagodge]

Yes @fatsheep9146 , Redis is running and opened for connections:

my-otel-demo-redis-844988fcfb-5m2pb

1 1:C 22 Aug 2022 18:05:40.990 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 2 1:C 22 Aug 2022 18:05:40.990 # Redis version=7.0.4, bits=64, commit=00000000, modified=0, pid=1, just started 3 1:C 22 Aug 2022 18:05:40.990 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf 4 1:M 22 Aug 2022 18:05:40.991 * monotonic clock: POSIX clock_gettime 5 1:M 22 Aug 2022 18:05:40.992 * Running mode=standalone, port=6379. 6 1:M 22 Aug 2022 18:05:40.993 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 7 1:M 22 Aug 2022 18:05:40.993 # Server initialized 8 1:M 22 Aug 2022 18:05:40.993 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to check if the system's clocksource isn't degrading performance. 9 1:M 22 Aug 2022 18:05:40.993 * Ready to accept connections

[julianocosta89]

@Tiagodge is this issue still ongoing? We had some new updates in the demo and in the Helm charts, would you mind taking a look again?

[Tiagodge]

Yes, the issue was not fixed, sure. I will try the new one. But I am on vacation returning on oct 23

Thanks

[austinlparker]

This is non-blocking for 1.0

[lfranchini31]

@Tiagodge you have another services in CrashLoopBackOff error? I try the last version of this helm chart and i have 4 pods in error (my-otel-demo-cart-service with the same error, my-otel-demo-email-service, my-otel-demo-featureflag-service and my-otel-demo-frontend always in CrashLoopBackOff). I think these errors have the same source: [error] Could not create schema migrations table. This error usually happens due to the following:

• The database does not exist
• The "schema_migrations" table, which Ecto uses for managing migrations, was defined by another library
• There is a deadlock while migrating (such as using concurrent indexes with a migration_lock) To fix the first issue, run "mix ecto.create". To address the second, you can run "mix ecto.drop" followed by "mix ecto.create". Alternatively you may configure Ecto to use another table and/or repository for managing migrations: config :featureflagservice, Featureflagservice.Repo, migration_source: "some_other_table_for_schema_migrations", migration_repo: AnotherRepoForSchemaMigrations

** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2977ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

1. Ensuring your database is available and that you can connect to it
2. Tracking down slow queries and making sure they are running fast enough
3. Increasing the pool_size (although this increases resource consumption)
4. Allowing requests to wait longer by increasing :queue_target and :queue_interval See DBConnection.start_link/2 for more information (ecto_sql 3.8.2) lib/ecto/adapters/sql.ex:932: Ecto.Adapters.SQL.raise_sql_call_error/1 (elixir 1.13.3) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2 (ecto_sql 3.8.2) lib/ecto/adapters/sql.ex:1024: Ecto.Adapters.SQL.execute_ddl/4 (ecto_sql 3.8.2) lib/ecto/migrator.ex:696: Ecto.Migrator.verbose_schema_migration/3 (ecto_sql 3.8.2) lib/ecto/migrator.ex:510: Ecto.Migrator.lock_for_migrations/4 (ecto_sql 3.8.2) lib/ecto/migrator.ex:422: Ecto.Migrator.run/4 (ecto_sql 3.8.2) lib/ecto/migrator.ex:146: Ecto.Migrator.with_repo/3 (featureflagservice 0.1.0) lib/featureflagservice/release.ex:12: anonymous fn/2 in Featureflagservice.Release.migrate/0