Conductor with Postgres

[ScottJOster]

Describe the bug

In a nutshell when running docker compose on compose and Postgres yaml files the server gets many bean errors including metadatadao(this on both local docker and kuber deployment).

One fix suggested advises setting a runtimeonly property in server build gradle but no matter which version of the maven Postgres I specify I still get the errors.

Can anyone advise what is the correct way to set up with Postgres ? Or is it time to switch to dynomite as it doesn't seem to support Postgres unless I'm missing obvious docs ?

I've spent a long time on this any help appreciated it seems like the question has never been answered fully .

Thanks in advance

Details Conductor version: latest / various Persistence implementation: Postgres, Queue implementation: Postgres Workflow definition:n/a Task definition:n/a Event handler definition:n/a

To Reproduce Steps to reproduce the behavior:

1. Pull docker
2. Try to build with docker compose yaml and docker postgres yaml Expected behavior Server builds with no bean errors /Postgres works

[sks1995]

both common persistence and postgres persistence dependencies should be added https://github.com/Netflix/conductor-community/tree/main/persistence/common-persistence https://github.com/Netflix/conductor-community/tree/main/persistence/postgres-persistence

[ScottJOster]

Thanks for the response - could I ask you to be more explicit however ? Are there any build steps documented for the community repo ? I have been working on dockerfiles etc from Netflix/conductor main and this community version doesn't seem to be mentioned in docs ?

Thanks again

[sks1995]

The proper steps seem to be missing for postgres/mysql. However looking at community server gradle will give some idea https://github.com/Netflix/conductor-community/blob/main/community-server/build.gradle There is blog as well: https://betterprogramming.pub/running-netflix-conductor-3-in-docker-using-elasticsearch-7-and-postgresql-b415988dd74a

[ScottJOster]

Thanks a lot much appreciated

[faradayfan]

I would like to request detailed instructions be added to the documentation on how to add properly configure the server to use postgresql for persistence. I was eventually able to discover that adding the following lines to the server/build.grade allowed me to build a version of conductor that would successfully connect to a postgresql database.

dependencies {
    ...
    runtimeOnly group: 'com.netflix.conductor', name: 'conductor-common-persistence', version: '3.13.2'
    runtimeOnly group: 'com.netflix.conductor', name: 'conductor-postgres-persistence', version: '3.13.2'
    ...
}

But it would be convenient to have a guide that outlines how to add these runtime dependencies without having to build conductor.

[ab48917]

Also, this persistence layer (either postgres/mysql provides ExecutionDAO, MetadataDAO and QueueDAO implementations. These should be flag to stop bringing the QueueDAO from mysql/postgres if we wanted to use the RedisQueueDAO provided by conductor and RedisQueueDAO shouldn't be flagged with condutor.db.type=redis-cluster, instead it should be like conductor.cache.type=redis-cluster so that we can have both persistance layer as well as queue layer (redis/dyno) separately.

[thxmike]

I would agree with faradayfan, I have been struggling for days trying to get this running and still can't get it running with POSTGRES.

[ab48917]

@thxmike I can help in case you need.

[thxmike]

@ab48917 Sure I will take all the help I can get.

[dterg]

This is still an issue for me as well, despite changing the build.grade as detailed above. conductor-server complains specifically in the /app/startup.sh:

: not founderver                  | /app/startup.sh: line 14:
: not founderver                  | /app/startup.sh: line 16:
: not founderver                  | /app/startup.sh: line 18:
: not founderver                  | /app/startup.sh: line 21: nginx
: not founderver                  | /app/startup.sh: line 22:
: No such file or directory       | /app/startup.sh: cd: line 24: can't cd to /app/libs
: not founderver                  | /app/startup.sh: line 28:
conductor-server                  | Starting Conductor server
conductor-server                  | Running Nginx in background
conductor-server                  | Property file: config-postgres.properties
conductor-server                  | config-postgres.properties
conductor-server                  | /app/startup.sh: line 41: syntax error: unexpected end of file (expecting "then")
conductor-server exited with code 2

[jeacott1]

@ScottJOster surely switching to dynomite can't be an option, it hasn't had a commit in over a year! it looks to be completely defunct. I'm looking for a new workflow orchestration engine atm, but this issue is really turning me off.