Feature Flag Service Can't Connect to ffs_postgres

[jbhardman]

Bug Report

Which version of the demo you are using? d8ecad7b975e662d0ab9096d7952f7e14497787c

Symptom

I have tried on two different linux hosts. Using the docker compose instructions: docker compose up --force-recreate --remove-orphans --detach

I get the same problem every time. The Feature Flag Service cannot connect to ffs_postgres, and so it just continually restarts. This causes problems for everything else.

Postgres reports that it is working:

[root@gardy opentelemetry-demo]# docker ps | grep postgres
941d7440a786   postgres:16.1                                             "docker-entrypoint.s…"   52 minutes ago   Up 33 minutes (healthy)         5432/tcp                                                                                                                                                                  postgres
[root@gardy opentelemetry-demo]# docker exec -it postgres bash
postgres@941d7440a786:/$ pg_isready -U ffs -d ffs
/var/run/postgresql:5432 - accepting connections

But I just get this over and over from feature-flag-service:

Database setup or migrations failed.
Killed
Database setup or migrations failed.
Killed
Killed
Database setup or migrations failed.

What is the expected behavior?

I expect the feature-flag-service to connect to postgres and work correctly.

What is the actual behavior?

feature-flag-service fails to connect and constantly restarts

Reproduce

git clone https://github.com/open-telemetry/opentelemetry-demo.git
cd opentelemetry-demo/
docker compose up --force-recreate --remove-orphans --detach

We will close this issue if:

• The steps you provided are complex.
• If we can not reproduce the behavior you're reporting.

Additional Context

i have tried manually restarting both postgres and feature-flag-service with docker restart, but that didn't help either.

[puckpuck]

Can you do a docker compose down or make stop first?

[jbhardman]

Can you do a docker compose down or make stop first?

docker compose down works and stops everything. I've done down/up at least a dozen times on 3 different machines. Same results every time.

[jbhardman]

I just tried on my Apple Silicon Mac with Docker Desktop and for some reason it worked. It fails on all of my x86_64 linux machines.

[puckpuck]

I just tested the demo on an x86 Ubuntu 22.04.3 system and it worked as expected. Which version of Linux are you trying this on? I was also able to test it on an x86-based EKS cluster with it working again as expected.

[portertech]

Also seeing Database setup or migrations failed. on Rocky 9.2, 5.14.0-362.13.1.el9_3.x86_64.

Docker version 24.0.7, build afdd53b.

[jbhardman]

Still can't get it to run on x86_64. Only runs on my Mac.

If I remove the ENTRYPOINT line from the Dockerfile, the container will stay running so I can go in. If I run the server script, /app/bin/server, it fails with the same output.

[root@gardy opentelemetry-demo]# docker compose run featureflagservice [+] Creating 1/0 Container postgres Running 0.0s nobody@8e5503b3ec87:/app$ cd bin nobody@8e5503b3ec87:/app/bin$ ./server Killed Database setup or migrations failed. nobody@8e5503b3ec87:/app/bin$

The server script just runs another script, with this command:

./featureflagservice eval Featureflagservice.Release.migrate || { echo "Database setup or migrations failed."; exit 1; }

As you can, that is where the Database setup message comes from. What I don't know is why the first command ./featureflagservice eval Featureflagservice.Release.migrate , always says "Killed".

Any ideas?

[puckpuck]

This should be fixed now with the release of 1.7.2, and in the main branch.

[jbhardman]

Still fails with 1.7.2 on Docker on Arch Linux. [root@arch1 opentelemetry-demo-1.7.2]# docker logs feature-flag-service Database setup or migrations failed. Killed Database setup or migrations failed. Killed Killed Database setup or migrations failed. Killed Database setup or migrations failed. Killed Database setup or migrations failed. Killed Database setup or migrations failed.

[root@arch1 opentelemetry-demo-1.7.2]# uname -a Linux arch1 6.7.0-arch3-1 #1 SMP PREEMPT_DYNAMIC Sat, 13 Jan 2024 14:37:14 +0000 x86_64 GNU/Linux

[root@arch1 opentelemetry-demo-1.7.2]# pacman -Q | grep docker docker 1:24.0.7-1 docker-compose 2.24.0-1

[jbhardman]

Still fails with 1.7.2 on Docker on Arch Linux. [root@arch1 opentelemetry-demo-1.7.2]# docker logs feature-flag-service Database setup or migrations failed. Killed Database setup or migrations failed. Killed Killed Database setup or migrations failed. Killed Database setup or migrations failed. Killed Database setup or migrations failed. Killed Database setup or migrations failed.

[root@arch1 opentelemetry-demo-1.7.2]# uname -a Linux arch1 6.7.0-arch3-1 #1 SMP PREEMPT_DYNAMIC Sat, 13 Jan 2024 14:37:14 +0000 x86_64 GNU/Linux

[root@arch1 opentelemetry-demo-1.7.2]# pacman -Q | grep docker docker 1:24.0.7-1 docker-compose 2.24.0-1

@puckpuck Do you think it's a local environment thing? I can't get featureflagservice to work and All I get is "Database setup or migrations failed".

[joshleecreates]

@jbhardman That error message is coming from this line: https://github.com/open-telemetry/opentelemetry-demo/blob/b8cc89455a0a7701c0277667ea1f700ae274a0bb/src/featureflagservice/rel/overlays/bin/server#L6 which isn't actually needed anymore (since 1.7.2), but I still don't see any reason why it would fail, but that could be environment specific. If you leave the entry point and just remove that line from the shell script, does it work?

[puckpuck]

@jbhardman we did a few more changes to the feature flag service. I apologize for sounding like a broken record here, but I do believe this is fixed now. Can you please check again?

[puckpuck]

This should be fixed now, but we have also decided to move away from the current Feature Flag service towards an OpenFeature implementation, so closing this issue.