jupyterhub-deploy-docker icon indicating copy to clipboard operation
jupyterhub-deploy-docker copied to clipboard

jupyterhub-db | FATAL: password authentication failed for user "postgres"

Open tanh314 opened this issue 5 years ago • 0 comments

I am receiving the following error when attempting to spin up jupyterhub.

Error

jupyterhub-db | FATAL:  password authentication failed for user "postgres"

Command Ran

docker-compose up

Troubleshooting Completed

  1. Confirmed secrets/postgres.env exists and has environment variable POSTGRES_PASSWORD set to some large hex.
  2. Confirmed docker-compose file lists secrets/postgres.env as an env_file.

Hypothesis

Doesn't postgresql://postgres:[redacted]@hub-db:None/jupyterhub state that we are trying to connect to the postgres db using the username postgres? Based on this SO post shouldn't the username postgres not have a password? According to the dockerhub documentation, the postgres image takes the POSTGRES_PASSWORD as an environment variable. How do I check if the postgres image is correctly accepting this environment file and setting the POSTGRES_PASSWORD correctly?

Logs

Recreating jupyterhub-db ... done
Recreating jupyterhub    ... done
Attaching to jupyterhub-db, jupyterhub
jupyterhub-db | 
jupyterhub-db | PostgreSQL Database directory appears to contain a database; Skipping initialization
jupyterhub-db | 
jupyterhub-db | LOG:  database system was shut down at 2020-03-09 22:31:03 UTC
jupyterhub-db | LOG:  MultiXact member wraparound protections are now enabled
jupyterhub-db | LOG:  database system is ready to accept connections
jupyterhub-db | LOG:  autovacuum launcher started
jupyterhub | [I 2020-03-10 13:51:58.365 JupyterHub app:2240] Running JupyterHub version 1.1.0
jupyterhub | [I 2020-03-10 13:51:58.366 JupyterHub app:2270] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.1.0
jupyterhub | [I 2020-03-10 13:51:58.366 JupyterHub app:2270] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-0.9.1
jupyterhub | [I 2020-03-10 13:51:58.366 JupyterHub app:2270] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.1.0
jupyterhub | [I 2020-03-10 13:51:58.367 JupyterHub app:1349] Loading cookie_secret from /data/jupyterhub_cookie_secret
jupyterhub-db | FATAL:  password authentication failed for user "postgres"
jupyterhub-db | DETAIL:  Connection matched pg_hba.conf line 95: "host all all all md5"
jupyterhub | [E 2020-03-10 13:51:58.381 JupyterHub app:1526] Failed to connect to db: postgresql://postgres:[redacted]@hub-db:None/jupyterhub
jupyterhub | [C 2020-03-10 13:51:58.382 JupyterHub app:1530] If you recently upgraded JupyterHub, try running
jupyterhub |         jupyterhub upgrade-db
jupyterhub |     to upgrade your JupyterHub database schema

Docker-Compose file

# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.

# JupyterHub docker-compose configuration file
version: "2"

services:
  hub-db:
    image: postgres:9.5
    container_name: jupyterhub-db
    restart: always
    environment:
      POSTGRES_DB: ${POSTGRES_DB}
      PGDATA: ${DB_VOLUME_CONTAINER}
    env_file:
      - secrets/postgres.env
    volumes:
      - "db:${DB_VOLUME_CONTAINER}"

  hub:
    depends_on:
      - hub-db
    build:
      context: .
      dockerfile: Dockerfile.jupyterhub
      args:
        JUPYTERHUB_VERSION: ${JUPYTERHUB_VERSION}
    restart: always
    image: jupyterhub
    container_name: jupyterhub
    volumes:
      # Bind Docker socket on the host so we can connect to the daemon from
      # within the container
      - "/var/run/docker.sock:/var/run/docker.sock:rw"
      # Bind Docker volume on host for JupyterHub database and cookie secrets
      - "data:${DATA_VOLUME_CONTAINER}"
    ports:
      - "443:443"
    links:
      - hub-db
    environment:
      # All containers will join this network
      DOCKER_NETWORK_NAME: ${DOCKER_NETWORK_NAME}
      # JupyterHub will spawn this Notebook image for users
      DOCKER_NOTEBOOK_IMAGE: ${LOCAL_NOTEBOOK_IMAGE}
      # Notebook directory inside user image
      DOCKER_NOTEBOOK_DIR: ${DOCKER_NOTEBOOK_DIR}
      # Using this run command (optional)
      DOCKER_SPAWN_CMD: ${DOCKER_SPAWN_CMD}
      # Postgres db info
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_HOST: hub-db
    env_file:
      - secrets/postgres.env
    command: >
      jupyterhub -f /srv/jupyterhub/jupyterhub_config.py

volumes:
  data:
    external:
      name: ${DATA_VOLUME_HOST}
  db:
    external:
      name: ${DB_VOLUME_HOST}

networks:
  default:
    external:
      name: ${DOCKER_NETWORK_NAME}

jupyterhub_config.py

# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.

# Configuration file for JupyterHub
import os

c = get_config()

# We rely on environment variables to configure JupyterHub so that we
# avoid having to rebuild the JupyterHub container every time we change a
# configuration parameter.

# Spawn single-user servers as Docker containers
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
# Spawn containers from this image
c.DockerSpawner.container_image = os.environ['DOCKER_NOTEBOOK_IMAGE']
# JupyterHub requires a single-user instance of the Notebook server, so we
# default to using the `start-singleuser.sh` script included in the
# jupyter/docker-stacks *-notebook images as the Docker run command when
# spawning containers.  Optionally, you can override the Docker run command
# using the DOCKER_SPAWN_CMD environment variable.
spawn_cmd = os.environ.get('DOCKER_SPAWN_CMD', "start-singleuser.sh")
c.DockerSpawner.extra_create_kwargs.update({ 'command': spawn_cmd })
# Connect containers to this Docker network
network_name = os.environ['DOCKER_NETWORK_NAME']
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.network_name = network_name
# Pass the network name as argument to spawned containers
c.DockerSpawner.extra_host_config = { 'network_mode': network_name }
# Explicitly set notebook directory because we'll be mounting a host volume to
# it.  Most jupyter/docker-stacks *-notebook images run the Notebook server as
# user `jovyan`, and set the notebook directory to `/home/jovyan/work`.
# We follow the same convention.
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan/work'
c.DockerSpawner.notebook_dir = notebook_dir
# Mount the real user's Docker volume on the host to the notebook user's
# notebook directory in the container
c.DockerSpawner.volumes = { 'jupyterhub-user-{username}': notebook_dir }
# volume_driver is no longer a keyword argument to create_container()
# c.DockerSpawner.extra_create_kwargs.update({ 'volume_driver': 'local' })
# Remove containers once they are stopped
c.DockerSpawner.remove_containers = True
# For debugging arguments passed to spawned containers
c.DockerSpawner.debug = True

# User containers will access hub by container name on the Docker network
c.JupyterHub.hub_ip = 'jupyterhub'
c.JupyterHub.hub_port = 8080

# TLS config
c.JupyterHub.port = 443
c.JupyterHub.ssl_key = os.environ['SSL_KEY']
c.JupyterHub.ssl_cert = os.environ['SSL_CERT']

# Authenticate users with GitHub OAuth
# c.JupyterHub.authenticator_class = 'oauthenticator.GitHubOAuthenticator'
# c.GitHubOAuthenticator.oauth_callback_url = os.environ['OAUTH_CALLBACK_URL']
# dummy for testing. Don't use this in production!
# c.JupyterHub.authenticator_class = 'jupyterhub.auth.DummyAuthenticator'
c.PAMAuthenticator.create_system_users = True

# Persist hub data on volume mounted inside container
data_dir = os.environ.get('DATA_VOLUME_CONTAINER', '/data')

c.JupyterHub.cookie_secret_file = os.path.join(data_dir,
    'jupyterhub_cookie_secret')

c.JupyterHub.db_url = 'postgresql://postgres:{password}@{host}/{db}'.format(
    host=os.environ['POSTGRES_HOST'],
    password=os.environ['POSTGRES_PASSWORD'],
    db=os.environ['POSTGRES_DB'],
)

# Whitlelist users and admins
c.Authenticator.whitelist = whitelist = set()
c.Authenticator.admin_users = admin = set()
c.JupyterHub.admin_access = True
pwd = os.path.dirname(__file__)
with open(os.path.join(pwd, 'userlist')) as f:
    for line in f:
        if not line:
            continue
        parts = line.split()
        # in case of newline at the end of userlist file
        if len(parts) >= 1:
            name = parts[0]
            whitelist.add(name)
            if len(parts) > 1 and parts[1] == 'admin':
                admin.add(name)

tanh314 avatar Mar 10 '20 14:03 tanh314