jupyterhub-deploy-docker
jupyterhub-deploy-docker copied to clipboard
jupyterhub-db | FATAL: password authentication failed for user "postgres"
I am receiving the following error when attempting to spin up jupyterhub.
Error
jupyterhub-db | FATAL: password authentication failed for user "postgres"
Command Ran
docker-compose up
Troubleshooting Completed
- Confirmed
secrets/postgres.envexists and has environment variablePOSTGRES_PASSWORDset to some large hex. - Confirmed docker-compose file lists
secrets/postgres.envas anenv_file.
Hypothesis
Doesn't postgresql://postgres:[redacted]@hub-db:None/jupyterhub state that we are trying to connect to the postgres db using the username postgres? Based on this SO post shouldn't the username postgres not have a password? According to the dockerhub documentation, the postgres image takes the POSTGRES_PASSWORD as an environment variable. How do I check if the postgres image is correctly accepting this environment file and setting the POSTGRES_PASSWORD correctly?
Logs
Recreating jupyterhub-db ... done
Recreating jupyterhub ... done
Attaching to jupyterhub-db, jupyterhub
jupyterhub-db |
jupyterhub-db | PostgreSQL Database directory appears to contain a database; Skipping initialization
jupyterhub-db |
jupyterhub-db | LOG: database system was shut down at 2020-03-09 22:31:03 UTC
jupyterhub-db | LOG: MultiXact member wraparound protections are now enabled
jupyterhub-db | LOG: database system is ready to accept connections
jupyterhub-db | LOG: autovacuum launcher started
jupyterhub | [I 2020-03-10 13:51:58.365 JupyterHub app:2240] Running JupyterHub version 1.1.0
jupyterhub | [I 2020-03-10 13:51:58.366 JupyterHub app:2270] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.1.0
jupyterhub | [I 2020-03-10 13:51:58.366 JupyterHub app:2270] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-0.9.1
jupyterhub | [I 2020-03-10 13:51:58.366 JupyterHub app:2270] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.1.0
jupyterhub | [I 2020-03-10 13:51:58.367 JupyterHub app:1349] Loading cookie_secret from /data/jupyterhub_cookie_secret
jupyterhub-db | FATAL: password authentication failed for user "postgres"
jupyterhub-db | DETAIL: Connection matched pg_hba.conf line 95: "host all all all md5"
jupyterhub | [E 2020-03-10 13:51:58.381 JupyterHub app:1526] Failed to connect to db: postgresql://postgres:[redacted]@hub-db:None/jupyterhub
jupyterhub | [C 2020-03-10 13:51:58.382 JupyterHub app:1530] If you recently upgraded JupyterHub, try running
jupyterhub | jupyterhub upgrade-db
jupyterhub | to upgrade your JupyterHub database schema
Docker-Compose file
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
# JupyterHub docker-compose configuration file
version: "2"
services:
hub-db:
image: postgres:9.5
container_name: jupyterhub-db
restart: always
environment:
POSTGRES_DB: ${POSTGRES_DB}
PGDATA: ${DB_VOLUME_CONTAINER}
env_file:
- secrets/postgres.env
volumes:
- "db:${DB_VOLUME_CONTAINER}"
hub:
depends_on:
- hub-db
build:
context: .
dockerfile: Dockerfile.jupyterhub
args:
JUPYTERHUB_VERSION: ${JUPYTERHUB_VERSION}
restart: always
image: jupyterhub
container_name: jupyterhub
volumes:
# Bind Docker socket on the host so we can connect to the daemon from
# within the container
- "/var/run/docker.sock:/var/run/docker.sock:rw"
# Bind Docker volume on host for JupyterHub database and cookie secrets
- "data:${DATA_VOLUME_CONTAINER}"
ports:
- "443:443"
links:
- hub-db
environment:
# All containers will join this network
DOCKER_NETWORK_NAME: ${DOCKER_NETWORK_NAME}
# JupyterHub will spawn this Notebook image for users
DOCKER_NOTEBOOK_IMAGE: ${LOCAL_NOTEBOOK_IMAGE}
# Notebook directory inside user image
DOCKER_NOTEBOOK_DIR: ${DOCKER_NOTEBOOK_DIR}
# Using this run command (optional)
DOCKER_SPAWN_CMD: ${DOCKER_SPAWN_CMD}
# Postgres db info
POSTGRES_DB: ${POSTGRES_DB}
POSTGRES_HOST: hub-db
env_file:
- secrets/postgres.env
command: >
jupyterhub -f /srv/jupyterhub/jupyterhub_config.py
volumes:
data:
external:
name: ${DATA_VOLUME_HOST}
db:
external:
name: ${DB_VOLUME_HOST}
networks:
default:
external:
name: ${DOCKER_NETWORK_NAME}
jupyterhub_config.py
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
# Configuration file for JupyterHub
import os
c = get_config()
# We rely on environment variables to configure JupyterHub so that we
# avoid having to rebuild the JupyterHub container every time we change a
# configuration parameter.
# Spawn single-user servers as Docker containers
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
# Spawn containers from this image
c.DockerSpawner.container_image = os.environ['DOCKER_NOTEBOOK_IMAGE']
# JupyterHub requires a single-user instance of the Notebook server, so we
# default to using the `start-singleuser.sh` script included in the
# jupyter/docker-stacks *-notebook images as the Docker run command when
# spawning containers. Optionally, you can override the Docker run command
# using the DOCKER_SPAWN_CMD environment variable.
spawn_cmd = os.environ.get('DOCKER_SPAWN_CMD', "start-singleuser.sh")
c.DockerSpawner.extra_create_kwargs.update({ 'command': spawn_cmd })
# Connect containers to this Docker network
network_name = os.environ['DOCKER_NETWORK_NAME']
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.network_name = network_name
# Pass the network name as argument to spawned containers
c.DockerSpawner.extra_host_config = { 'network_mode': network_name }
# Explicitly set notebook directory because we'll be mounting a host volume to
# it. Most jupyter/docker-stacks *-notebook images run the Notebook server as
# user `jovyan`, and set the notebook directory to `/home/jovyan/work`.
# We follow the same convention.
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan/work'
c.DockerSpawner.notebook_dir = notebook_dir
# Mount the real user's Docker volume on the host to the notebook user's
# notebook directory in the container
c.DockerSpawner.volumes = { 'jupyterhub-user-{username}': notebook_dir }
# volume_driver is no longer a keyword argument to create_container()
# c.DockerSpawner.extra_create_kwargs.update({ 'volume_driver': 'local' })
# Remove containers once they are stopped
c.DockerSpawner.remove_containers = True
# For debugging arguments passed to spawned containers
c.DockerSpawner.debug = True
# User containers will access hub by container name on the Docker network
c.JupyterHub.hub_ip = 'jupyterhub'
c.JupyterHub.hub_port = 8080
# TLS config
c.JupyterHub.port = 443
c.JupyterHub.ssl_key = os.environ['SSL_KEY']
c.JupyterHub.ssl_cert = os.environ['SSL_CERT']
# Authenticate users with GitHub OAuth
# c.JupyterHub.authenticator_class = 'oauthenticator.GitHubOAuthenticator'
# c.GitHubOAuthenticator.oauth_callback_url = os.environ['OAUTH_CALLBACK_URL']
# dummy for testing. Don't use this in production!
# c.JupyterHub.authenticator_class = 'jupyterhub.auth.DummyAuthenticator'
c.PAMAuthenticator.create_system_users = True
# Persist hub data on volume mounted inside container
data_dir = os.environ.get('DATA_VOLUME_CONTAINER', '/data')
c.JupyterHub.cookie_secret_file = os.path.join(data_dir,
'jupyterhub_cookie_secret')
c.JupyterHub.db_url = 'postgresql://postgres:{password}@{host}/{db}'.format(
host=os.environ['POSTGRES_HOST'],
password=os.environ['POSTGRES_PASSWORD'],
db=os.environ['POSTGRES_DB'],
)
# Whitlelist users and admins
c.Authenticator.whitelist = whitelist = set()
c.Authenticator.admin_users = admin = set()
c.JupyterHub.admin_access = True
pwd = os.path.dirname(__file__)
with open(os.path.join(pwd, 'userlist')) as f:
for line in f:
if not line:
continue
parts = line.split()
# in case of newline at the end of userlist file
if len(parts) >= 1:
name = parts[0]
whitelist.add(name)
if len(parts) > 1 and parts[1] == 'admin':
admin.add(name)