modeldb Error when setting up ModelDB via Docker Compose

On Windows 10 when running docker-compose -f docker-compose-all.yaml up I get the following:

modeldb-backend | {"thread":"main","level":"DEBUG","loggerName":"ai.verta.modeldb.utils.ModelDBHibernateUtil","message":"ModelDBHibernateUtil getSessionFactory() retrying for DB connection after 2560 millisecond ","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","instant":{"epochSecond":1594737612,"nanoOfSecond":72000000},"threadId":1,"threadPriority":5,"hostName":"03fd2e8e7248","kubernetes.podIP":""} modeldb-backend | {"thread":"main","level":"WARN","loggerName":"ai.verta.modeldb.utils.ModelDBHibernateUtil","message":"ModelDBHibernateUtil checkDBConnection() got error ","thrown":{"commonElementCount":0,"localizedMessage":"The connection attempt failed.","message":"The connection attempt failed.","name":"org.postgresql.util.PSQLException","cause":{"commonElementCount":19,"localizedMessage":"Connect timed out","message":"Connect timed out","name":"java.net.SocketTimeoutException","extendedStackTrace":"java.net.SocketTimeoutException: Connect timed out\n\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]\n\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]\n\tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333) ~[?:?]\n\tat java.net.Socket.connect(Socket.java:648) ~[?:?]\n\tat org.postgresql.core.PGStream.<init>(PGStream.java:75) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) ~[postgresql-42.2.6.jar!/:42.2.6]\n"},"extendedStackTrace":"org.postgresql.util.PSQLException: The connection attempt failed.\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:292) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.Driver.makeConnection(Driver.java:458) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.Driver.connect(Driver.java:260) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat java.sql.DriverManager.getConnection(DriverManager.java:677) ~[java.sql:?]\n\tat java.sql.DriverManager.getConnection(DriverManager.java:228) ~[java.sql:?]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.checkDBConnection(ModelDBHibernateUtil.java:483) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.checkDBConnectionInLoop(ModelDBHibernateUtil.java:325) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.createOrGetSessionFactory(ModelDBHibernateUtil.java:240) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.App.initializeServicesBaseOnDataBase(App.java:363) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.App.main(App.java:260) [classes!/:1.0-SNAPSHOT]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]\n\tat jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]\n\tat java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]\n\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:58) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\nCaused by: java.net.SocketTimeoutException: Connect timed out\n\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]\n\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]\n\tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333) ~[?:?]\n\tat java.net.Socket.connect(Socket.java:648) ~[?:?]\n\tat org.postgresql.core.PGStream.<init>(PGStream.java:75) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) ~[postgresql-42.2.6.jar!/:42.2.6]\n\t... 19 more\n"},"endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","instant":{"epochSecond":1594737622,"nanoOfSecond":77000000},"threadId":1,"threadPriority":5,"hostName":"03fd2e8e7248","kubernetes.podIP":""}

Jul 14 '20 15:07 RaduStoicescu

@RaduStoicescu : modeldb backend is unable to connect to postgres , can you verify it postgres container is up?

I am not sure how this is done on windows but this is a list of container which needs to be up for modeldb to work correctly.

 (⎈) ~/w/c/o/k/l/e/p/m/backend> docker ps
CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS              PORTS                              NAMES
5d8ad539fcf0        vertaaiofficial/modeldb-frontend:2.0.7   "docker-entrypoint.s…"   8 seconds ago       Up 7 seconds        0.0.0.0:3000->3000/tcp             modeldb-frontend
cecb257725c5        postgres                                 "docker-entrypoint.s…"   11 seconds ago      Up 10 seconds       5432/tcp                           modeldb-postgres
1d02914e811a        vertaaiofficial/modeldb-proxy:2.0.7      "/app/bin/proxy"         11 days ago         Up 8 seconds        0.0.0.0:8080->8080/tcp             modeldb-proxy
84abdfb07132        vertaaiofficial/modeldb-graphql:2.0.7    "/app/bin/graphql"       11 days ago         Up 8 seconds        0.0.0.0:4000->4000/tcp             modeldb-graphql
271c82d3f177        vertaaiofficial/modeldb-backend:2.0.7    "java -jar modeldb-1…"   11 days ago         Up 9 seconds        0.0.0.0:8085-8086->8085-8086/tcp   modeldb-backend

If postgres had problems coming up it should also show up earlier in the logs you shared.

Jul 14 '20 18:07 ravishetye

Looks like it has trouble having access to "/var/lib/postgresql/data/pgdata". I'm not sure how to create or how to set permissions for that directory inside the Docker


This user must also own the server process.


The database cluster will be initialized with locale "en_US.utf8".

The default database encoding has accordingly been set to "UTF8".

The default text search configuration will be set to "english".


Data page checksums are disabled.


fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok

creating subdirectories ... ok

selecting dynamic shared memory implementation ... posix

selecting default max_connections ... 20

selecting default shared_buffers ... 400kB

selecting default time zone ... Etc/UTC

creating configuration files ... ok

2020-07-14 19:43:17.163 UTC [82] FATAL: data directory "/var/lib/postgresql/data/pgdata" has wrong ownership

2020-07-14 19:43:17.163 UTC [82] HINT: The server must be started by the user that owns the data directory.

child process exited with exit code 1

initdb: removing contents of data directory "/var/lib/postgresql/data/pgdata"```

Jul 14 '20 19:07 RaduStoicescu

Hey @RaduStoicescu, I am a contributor to modeldb backend and have docker-compose working in my setup on windows. My set up is

Windows 10 Home single language
I have used docker toolbox (https://docs.docker.com/toolbox/toolbox_install_windows/) setup for the dockerization of applications.

this is my docker-compose-all.yaml file

version: "3.5"
services:
    modeldb-backend:
        image: vertaaiofficial/modeldb-backend:2.0.7
        container_name: "modeldb-backend"
        expose:
            - "8085"
            - "8086"
        ports:
            - "8085:8085"
            - "8086:8086"
        networks:
            - modeldb_network
        environment:
            - VERTA_MODELDB_CONFIG:/config/config.yaml
        volumes:
            - /c/Users/config:/config/
            - /c/Users/artifact-store:/artifact-store/
    modeldb-proxy:
        image: vertaaiofficial/modeldb-proxy:2.0.7
        container_name: "modeldb-proxy"
        environment:
            MDB_ADDRESS: "modeldb-backend:8085"
            SERVER_HTTP_PORT: "8080"
        expose:
            - "8080"
        ports:
            - "8080:8080"
        depends_on:
            - modeldb-backend
        networks:
            - modeldb_network
    modeldb-graphql:
        image: vertaaiofficial/modeldb-graphql:2.0.7
        container_name: "modeldb-graphql"
        environment:
            MDB_ADDRESS: "modeldb-backend:8085"
            SERVER_HTTP_PORT: "4000"
            QUERY_PATH: "/api/v1/graphql/query"
        expose:
            - "4000"
        ports:
            - "4000:4000"
        depends_on:
            - modeldb-backend
        networks:
            - modeldb_network
    modeldb-frontend:
        image: vertaaiofficial/modeldb-frontend:2.0.7
        container_name: "modeldb-frontend"
        expose:
            - "3000"
        ports:
            - "3000:3000"
        depends_on:
            - modeldb-proxy
            - modeldb-graphql
        environment:
            DEPLOYED: "yes"
            BACKEND_API_PROTOCOL: "http"
            BACKEND_API_DOMAIN: "modeldb-webapp:3000"
            MDB_ADDRESS: http://modeldb-proxy:8080
            ARTIFACTORY_ADDRESS: http://modeldb-backend:8086
            GQL_ADDRESS: http://modeldb-graphql:4000
        networks:
            - modeldb_network
    modeldb-postgres:
        image: postgres
        container_name: "modeldb-postgres"
        restart: always
        environment:
            POSTGRES_PASSWORD: root
            PGDATA: /var/lib/postgresql/data/pgdata
        networks:
            - modeldb_network
        volumes:
            - /e/Users/data:/var/lib/postgresql/data/pgdata
networks:
  modeldb_network:

where I have used shared /c/Users volume for config & artifact-store and for the Postgres data I am using other drive location like /e/Users/data . I could not use /c/Users as mapped volume for the Postgres because it is by default read-only access by the windows and you can not do any write operation on it. Postgres will create backup volume on shared volume so you needs write permission on the shared volume. Can you try to use other drive location instead of c: as a mapped volume for Postgres as shown in docker-compose-all.yaml above.

I hope you can replicate my setup and that unblocks you. If not, then let me know your Windows version and about the docker setup, and then I can help you better.

Jul 16 '20 04:07 anandjakhaniya

I changed it like this and I have the same problem

I have Windows 10 Business, version 1903

I have found this on docker forums, it looks like a common problem: https://forums.docker.com/t/data-directory-var-lib-postgresql-data-pgdata-has-wrong-ownership/17963

        image: postgres
        container_name: "modeldb-postgres"
        restart: always
        environment:
            POSTGRES_PASSWORD: root
            PGDATA: /var/lib/postgresql/data/pgdata
        networks:
            - modeldb_network
        volumes:
            - /d/users/data:/var/lib/postgresql/data/pgdata

This is the error:

modeldb-postgres    | This user must also own the server process.
modeldb-postgres    |
modeldb-postgres    | The database cluster will be initialized with locale "en_US.utf8".
modeldb-postgres    | The default database encoding has accordingly been set to "UTF8".
modeldb-postgres    | The default text search configuration will be set to "english".
modeldb-postgres    |
modeldb-postgres    | Data page checksums are disabled.
modeldb-postgres    |
modeldb-postgres    | fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
modeldb-postgres    | creating subdirectories ... ok
modeldb-postgres    | selecting dynamic shared memory implementation ... posix
modeldb-postgres    | selecting default max_connections ... 20
modeldb-postgres    | selecting default shared_buffers ... 400kB
modeldb-postgres    | selecting default time zone ... Etc/UTC
modeldb-postgres    | creating configuration files ... ok
modeldb-postgres    | running bootstrap script ... 2020-07-16 17:56:19.080 UTC [82] FATAL:  data directory "/var/lib/postgresql/data/pgdata" has wrong ownership
modeldb-postgres    | 2020-07-16 17:56:19.080 UTC [82] HINT:  The server must be started by the user that owns the data directory.
modeldb-postgres    | child process exited with exit code 1
modeldb-postgres    | initdb: removing contents of data directory "/var/lib/postgresql/data/pgdata"```

Jul 16 '20 18:07 RaduStoicescu

@RaduStoicescu : were you able to bring up Postgres? if so could you share the high-level steps so that we could add it to the modeldb documentation. If you can point us to the steps which unblocked you at a high level, someone from our team could update the documents. If you rather prefer to contribute a PR you could add a section for Windows setup under https://github.com/VertaAI/modeldb/blob/master/DEPLOY.md#Deploy-pre-published-images.

Jul 28 '20 01:07 ravishetye