docker-galaxy-stable icon indicating copy to clipboard operation
docker-galaxy-stable copied to clipboard

Published 20 hours ago • verified publisher icon bgruening

Postgresql 9.3.16 galaxy wont start.

Open shadycuz opened this issue 6 years ago • 19 comments

What versions of postgresql is galaxy compatible with? I have also tried 9.6.2.

I'm using RDS on AWS.

When I try 9.3.16 I noticed galaxy_web and some other services failed.

root@ip-10-25-9-156:/galaxy-central# supervisorctl
autofs                           RUNNING    pid 408, uptime 0:19:53
condor                           STOPPED    Not started
cron                             RUNNING    pid 2412, uptime 0:16:32
docker                           FATAL      Exited too quickly (process log may have details)
galaxy:galaxy_nodejs_proxy       RUNNING    pid 2439, uptime 0:16:29
galaxy:galaxy_web                FATAL      Exited too quickly (process log may have details)
galaxy:handler0                  RUNNING    pid 2411, uptime 0:16:32
galaxy:handler1                  FATAL      Exited too quickly (process log may have details)
nginx                            RUNNING    pid 406, uptime 0:19:53
postgresql                       RUNNING    pid 486, uptime 0:19:49
proftpd                          RUNNING    pid 2432, uptime 0:16:30
rabbitmq                         FATAL      Exited too quickly (process log may have details)
reports                          STOPPED    Not started

So I got and get the handler log.

root@ip-10-25-9-156:/var/log/supervisor# cat handler0-stderr---supervisor-NyEUVa.log
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py:2408: SAWarning: Non-simple column elements in primary join condition for property HistoryDatasetAssociation.visible_children - consider using remote() annotations to mark the remote side.
  % self.prop
/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py:2408: SAWarning: Non-simple column elements in primary join condition for property LibraryDatasetDatasetAssociation.visible_children - consider using remote() annotations to mark the remote side.
  % self.prop
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 75, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1029, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 59, in create_or_verify_database
    migrate()
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 43, in migrate
    migrate_to_current_version( engine, db_schema )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 125, in migrate_to_current_version
    schema.runchange( ver, change, changeset.step )
  File "/galaxy_venv/local/lib/python2.7/site-packages/migrate/versioning/schema.py", line 91, in runchange
    (self.version, startver))
migrate.exceptions.InvalidVersionError: 104 is not 103
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 75, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1029, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '104' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 75, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1029, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '105' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 75, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1029, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '108' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 75, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1029, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '111' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 75, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1029, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '118' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 75, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1029, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '123' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 75, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1029, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '128' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.

Whats the DB log say?

2017-09-28 15:54:51 UTC:10.25.1.204(57450):galaxy@galaxy:[12018]:ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
2017-09-28 15:54:51 UTC:10.25.1.204(57450):galaxy@galaxy:[12018]:DETAIL: Key (typname, typnamespace)=(dataset_id_seq, 2200) already exists.
2017-09-28 15:54:51 UTC:10.25.1.204(57450):galaxy@galaxy:[12018]:STATEMENT: 
CREATE TABLE dataset (
id SERIAL NOT NULL, 
create_time TIMESTAMP WITHOUT TIME ZONE, 
update_time TIMESTAMP WITHOUT TIME ZONE, 
state VARCHAR(64), 
deleted BOOLEAN, 
purged BOOLEAN, 
purgable BOOLEAN, 
external_filename TEXT, 
_extra_files_path TEXT, 
file_size NUMERIC(15, 0), 
PRIMARY KEY (id)
)

So I run sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade I then restart galaxy_web and the failed handler and things look okay.

shadycuz avatar Sep 28 '17 20:09 shadycuz

I think the error message is this Exception: Your database has version '104' but this code expects version '134'. Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.

So you are trying to load a older Galaxy version database. But your Galaxy code is newer. We have an upgrade guide here: https://github.com/bgruening/docker-galaxy-stable#upgrading-images--toc

Galaxy can work with all versions of postgresql so this should not be the problem I think.

bgruening avatar Sep 28 '17 20:09 bgruening

Ahh, I thought it was referring back to the Postgres version. Also when I don't specify an environment variable and instead let galaxy run a local Postgres... or maybe its running SQLite? It starts fine.

shadycuz avatar Sep 28 '17 20:09 shadycuz

@bgruening This is a brand new empty postgres DB. Does that make a difference? my image is using FROM chambm/omicron-galaxy

shadycuz avatar Sep 28 '17 20:09 shadycuz

@shadycuz what do you mean with local postgresql? It is not running sqlite, if you not specifying it. If it is starting for you, do you see your old data?

bgruening avatar Sep 28 '17 20:09 bgruening

Huh, a brand new instance? This is strange the error from above indicates a difference between your Galaxy-database version and the Galaxy Source version.

bgruening avatar Sep 28 '17 20:09 bgruening

I'm testing with new EC2 machines in a jenkins build. With new RDS everytime. Strange indeed.

shadycuz avatar Sep 28 '17 20:09 shadycuz

Let's ping @chambm here as well :)

bgruening avatar Sep 28 '17 20:09 bgruening

haha =)

shadycuz avatar Sep 28 '17 20:09 shadycuz

@bgruening his docker eventually links back to this one https://github.com/chambm/omicron-galaxy/blob/master/docker/Dockerfile FROM bgruening/galaxy-stable

That is what I meant to paste the first time =)

shadycuz avatar Sep 28 '17 20:09 shadycuz

This is normal. We call this Galaxy flavor. All cool specific Galaxy environments are sharing one base image and work together to make it as smooth as possible.

bgruening avatar Sep 28 '17 20:09 bgruening

It's a weird series of errors. It seems like it's triggering the error for every migration step. Maybe one of the Galaxy core guys would have an idea? Who's the DB guru there @bgruening ?

chambm avatar Sep 28 '17 21:09 chambm

@bgruening @chambm So I tried again with the base image to make sure it wasn't something unique to @chambm Omicron Flavor.

This was my run command.

docker run --name omicron -d --restart=on-failure:10 --net=host --privileged \
                  -p 80:80 \
                  -p 8080:8080 \
                  -v /export/:/export/ \
                  -e "NONUSE=reports,slurmd,slurmctld,condor" \
                  -e GALAXY_CONFIG_FTP_UPLOAD_SITE=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4) \
                  -e GALAXY_CONFIG_CLEANUP_JOB=onsuccess \
                  -e GALAXY_CONFIG_DATABASE_CONNECTION=postgresql://galaxy:xxxxxxx@${DatabaseInstance.Endpoint.Address}:5432/galaxy \
                  -e GALAXY_CONFIG_DATABASE_ENGINE_OPTION_SERVER_SIDE_CURSORS=True \
                  -e GALAXY_CONFIG_DATABASE_ENGINE_OPTION_STRATEGY=threadlocal \
                  bgruening/galaxy-stable

This time the galaxy_web did not die but both handlers did.

root@ip-10-25-15-113:/var/log/supervisor# supervisorctl
condor                           STOPPED    Not started
cron                             RUNNING    pid 1198, uptime 1:45:46
docker                           FATAL      Exited too quickly (process log may have details)
galaxy:galaxy_nodejs_proxy       RUNNING    pid 1203, uptime 1:45:44
galaxy:galaxy_web                RUNNING    pid 153, uptime 1:49:28
galaxy:handler0                  FATAL      Exited too quickly (process log may have details)
galaxy:handler1                  FATAL      Exited too quickly (process log may have details)
nginx                            RUNNING    pid 151, uptime 1:49:28
postgresql                       RUNNING    pid 219, uptime 1:49:24
proftpd                          FATAL      Exited too quickly (process log may have details)
rabbitmq                         FATAL      Exited too quickly (process log may have details)
reports                          STOPPED    Not started
supervisor> exit

The log looks the same for handler0 and 1.

root@ip-10-25-15-113:/var/log/supervisor# cat handler0-stderr---supervisor-9yxWF_.log
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 58, in __init__
    self.config.check()
  File "/galaxy-central/lib/galaxy/config.py", line 735, in check
    self._ensure_directory( path )
  File "/galaxy-central/lib/galaxy/config.py", line 719, in _ensure_directory
    raise ConfigurationError( "Unable to create missing directory: %s\n%s" % ( path, e ) )
galaxy.exceptions.ConfigurationError: Unable to create missing directory: /export/nginx_upload_store
[Errno 13] Permission denied: '/export/nginx_upload_store'
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '0' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '5' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '9' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '15' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '21' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '27' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '33' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '39' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '46' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '53' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '60' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '68' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '76' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '84' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.
DEBUG:galaxy.app:python path is: /galaxy-central/scripts, /galaxy-central/lib, /galaxy_venv/lib/python2.7, /galaxy_venv/lib/python2.7/plat-x86_64-linux-gnu, /galaxy_venv/lib/python2.7/lib-tk, /galaxy_venv/lib/python2.7/lib-old, /galaxy_venv/lib/python2.7/lib-dynload, /usr/lib/python2.7, /usr/lib/python2.7/plat-x86_64-linux-gnu, /usr/lib/python2.7/lib-tk, /galaxy_venv/local/lib/python2.7/site-packages
WARNING:galaxy.containers:config file './config/containers_conf.yml' does not exist, running with default config
INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler0.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '92' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.

I notice at the top of the log a permissoned denied error for creating a directory, but it was eventually created? maybe the other handler?

[ec2-user@ip-10-25-15-113 ~]$ ls -la /export/
total 64
drwxr-xr-x 11 root   root    4096 Sep 28 21:30 .
dr-xr-xr-x 27 root   root    4096 Sep 28 21:27 ..
drwxr-xr-x  3 root   root    4096 Sep 27 05:48 .distribution_config
drwxr-xr-x  2 galaxy galaxy  4096 Sep 28 21:30 ftp
drwxr-xr-x  8 galaxy galaxy  4096 Sep 28 21:34 galaxy-central
drwxr-xr-x  2 root   root   16384 Sep 28 21:27 lost+found
-r--------  1 root   root      82 Sep 28 21:28 munge.key
drwx------  2 galaxy root    4096 Sep 28 21:30 nginx_upload_store
drwxr-xr-x  3   1550   1550  4096 Sep 28 21:30 postgresql
drwxr-xr-x  2 galaxy galaxy  4096 Sep 27 05:50 shed_tools
-rwxr-xr-x  1 root   root    2396 Sep 28 21:28 slurm.conf
drwxr-xr-x  3 galaxy galaxy  4096 Sep 28 21:34 tool_deps
drwxr-xr-x  3 root   root    4096 Sep 28 21:30 var

Version of the postgresql was 9.6.2, from the DB.

2017-09-28 21:30:20 UTC:10.25.15.113(54138):galaxy@galaxy:[7793]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:21 UTC:10.25.15.113(54140):galaxy@galaxy:[7794]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:23 UTC:10.25.15.113(54142):galaxy@galaxy:[7795]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:24 UTC:10.25.15.113(54144):galaxy@galaxy:[7796]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:25 UTC:10.25.15.113(54146):galaxy@galaxy:[7807]:ERROR: column galaxy_user.username does not exist at character 220
2017-09-28 21:30:25 UTC:10.25.15.113(54146):galaxy@galaxy:[7807]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:25 UTC:10.25.15.113(54146):galaxy@galaxy:[7807]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:27 UTC:10.25.15.113(54148):galaxy@galaxy:[7847]:ERROR: column galaxy_user.username does not exist at character 220
2017-09-28 21:30:27 UTC:10.25.15.113(54148):galaxy@galaxy:[7847]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:28 UTC:10.25.15.113(54148):galaxy@galaxy:[7847]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:28 UTC:10.25.15.113(54150):galaxy@galaxy:[7848]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:28 UTC:10.25.15.113(54152):galaxy@galaxy:[7849]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:30 UTC:10.25.15.113(54156):galaxy@galaxy:[7860]:ERROR: column galaxy_user.username does not exist at character 220
2017-09-28 21:30:30 UTC:10.25.15.113(54156):galaxy@galaxy:[7860]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:30 UTC:10.25.15.113(54156):galaxy@galaxy:[7860]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:32 UTC:10.25.15.113(54158):galaxy@galaxy:[7861]:ERROR: column galaxy_user.username does not exist at character 220
2017-09-28 21:30:32 UTC:10.25.15.113(54158):galaxy@galaxy:[7861]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:32 UTC:10.25.15.113(54158):galaxy@galaxy:[7861]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:34 UTC:10.25.15.113(54160):galaxy@galaxy:[7863]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:34 UTC:10.25.15.113(54162):galaxy@galaxy:[7864]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:35 UTC:10.25.15.113(54164):galaxy@galaxy:[7875]:ERROR: column galaxy_user.last_password_change does not exist at character 312
2017-09-28 21:30:35 UTC:10.25.15.113(54164):galaxy@galaxy:[7875]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:35 UTC:10.25.15.113(54164):galaxy@galaxy:[7875]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:37 UTC:10.25.15.113(54166):galaxy@galaxy:[7876]:ERROR: column galaxy_user.last_password_change does not exist at character 312
2017-09-28 21:30:37 UTC:10.25.15.113(54166):galaxy@galaxy:[7876]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:37 UTC:10.25.15.113(54166):galaxy@galaxy:[7876]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:40 UTC:10.25.15.113(54168):galaxy@galaxy:[7888]:ERROR: column galaxy_user.last_password_change does not exist at character 312
2017-09-28 21:30:40 UTC:10.25.15.113(54168):galaxy@galaxy:[7888]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:40 UTC:10.25.15.113(54168):galaxy@galaxy:[7888]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:41 UTC:10.25.15.113(54170):galaxy@galaxy:[7897]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:41 UTC:10.25.15.113(54172):galaxy@galaxy:[7898]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:42 UTC:10.25.15.113(54184):galaxy@galaxy:[7918]:ERROR: column galaxy_user.last_password_change does not exist at character 312
2017-09-28 21:30:42 UTC:10.25.15.113(54184):galaxy@galaxy:[7918]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:42 UTC:10.25.15.113(54184):galaxy@galaxy:[7918]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:45 UTC:10.25.15.113(54188):galaxy@galaxy:[7981]:ERROR: column galaxy_user.last_password_change does not exist at character 312
2017-09-28 21:30:45 UTC:10.25.15.113(54188):galaxy@galaxy:[7981]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:45 UTC:10.25.15.113(54188):galaxy@galaxy:[7981]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:47 UTC:10.25.15.113(54190):galaxy@galaxy:[7982]:ERROR: column galaxy_user.last_password_change does not exist at character 312
2017-09-28 21:30:47 UTC:10.25.15.113(54190):galaxy@galaxy:[7982]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:47 UTC:10.25.15.113(54190):galaxy@galaxy:[7982]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:48 UTC:10.25.15.113(54192):galaxy@galaxy:[7983]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:48 UTC:10.25.15.113(54194):galaxy@galaxy:[7984]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:50 UTC:10.25.15.113(54196):galaxy@galaxy:[7996]:ERROR: column galaxy_user.last_password_change does not exist at character 312
2017-09-28 21:30:50 UTC:10.25.15.113(54196):galaxy@galaxy:[7996]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:50 UTC:10.25.15.113(54196):galaxy@galaxy:[7996]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:52 UTC:10.25.15.113(54198):galaxy@galaxy:[7997]:ERROR: column galaxy_user.last_password_change does not exist at character 312
2017-09-28 21:30:52 UTC:10.25.15.113(54198):galaxy@galaxy:[7997]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:52 UTC:10.25.15.113(54198):galaxy@galaxy:[7997]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:55 UTC:10.25.15.113(54200):galaxy@galaxy:[8008]:ERROR: column galaxy_user.last_password_change does not exist at character 312
2017-09-28 21:30:55 UTC:10.25.15.113(54200):galaxy@galaxy:[8008]:STATEMENT: SELECT count(*) AS count_1 
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token 
FROM galaxy_user 
WHERE galaxy_user.email = '[email protected]') AS anon_1
2017-09-28 21:30:55 UTC:10.25.15.113(54200):galaxy@galaxy:[8008]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:56 UTC:10.25.15.113(54136):galaxy@galaxy:[7792]:ERROR: relation "ix_history_published" already exists
2017-09-28 21:30:56 UTC:10.25.15.113(54136):galaxy@galaxy:[7792]:STATEMENT: CREATE INDEX ix_history_published ON history (published)
2017-09-28 21:30:56 UTC:10.25.15.113(54136):galaxy@galaxy:[7792]:ERROR: relation "ix_stored_workflow_published" already exists
2017-09-28 21:30:56 UTC:10.25.15.113(54136):galaxy@galaxy:[7792]:STATEMENT: CREATE INDEX ix_stored_workflow_published ON stored_workflow (published)
2017-09-28 21:30:56 UTC:10.25.15.113(54136):galaxy@galaxy:[7792]:ERROR: relation "ix_page_importable" already exists
2017-09-28 21:30:56 UTC:10.25.15.113(54136):galaxy@galaxy:[7792]:STATEMENT: CREATE INDEX ix_page_importable ON page (importable)
2017-09-28 21:30:56 UTC:10.25.15.113(54204):galaxy@galaxy:[8032]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:56 UTC:10.25.15.113(54202):galaxy@galaxy:[8031]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:30:57 UTC:10.25.15.113(54206):galaxy@galaxy:[8034]:ERROR: column galaxy_user.last_password_change does not exist at character 312
2017-09-28 21:30:57 UTC:10.25.15.113(54206):galaxy@galaxy:[8034]:STATEMENT: SELECT count(*) AS count_1

mainly repeats the select command.

and then this at then end of the log

2017-09-28 21:33:57 UTC:10.25.15.113(54438):galaxy@galaxy:[8962]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:34:00 UTC:10.25.15.113(54442):galaxy@galaxy:[8970]:LOG: could not receive data from client: Connection reset by peer
2017-09-28 21:34:03 UTC:10.25.15.113(54462):galaxy@galaxy:[8978]:ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
2017-09-28 21:34:03 UTC:10.25.15.113(54462):galaxy@galaxy:[8978]:DETAIL: Key (typname, typnamespace)=(queue_id_sequence, 2200) already exists.
2017-09-28 21:34:03 UTC:10.25.15.113(54462):galaxy@galaxy:[8978]:STATEMENT: CREATE SEQUENCE queue_id_sequence
2017-09-28 21:34:12 UTC:10.25.15.113(54466):galaxy@galaxy:[9033]:LOG: could not receive data from client: Connection reset by peer

Since both handlers have the same log are they both trying to run migrations at the same time?

@bgruening any other logs I have not included that you might need? Should I open an issue in the main galaxy repo?

Thanks, Levi

shadycuz avatar Sep 28 '17 23:09 shadycuz

Also why is Postgres running in the docker? postgresql RUNNING pid 219, uptime 2:15:50

shadycuz avatar Sep 28 '17 23:09 shadycuz

both handlers appear to be fine after restarting them

 galaxy:handler0                  RUNNING    pid 1694, uptime 0:09:42
galaxy:handler1                  RUNNING    pid 1787, uptime 0:04:10

shadycuz avatar Sep 28 '17 23:09 shadycuz

@shadycuz which Galaxy version is this? How do you run it? I have never ever seen this before. Can this be an issue of your environment, Docker? What happens if you run it on your computer?

bgruening avatar Sep 29 '17 08:09 bgruening

@bgruening I posted it at the top of that long post =)

docker run --name omicron -d --restart=on-failure:10 --net=host --privileged \
                  -p 80:80 \
                  -p 8080:8080 \
                  -v /export/:/export/ \
                  -e "NONUSE=reports,slurmd,slurmctld,condor" \
                  -e GALAXY_CONFIG_FTP_UPLOAD_SITE=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4) \
                  -e GALAXY_CONFIG_CLEANUP_JOB=onsuccess \
                  -e GALAXY_CONFIG_DATABASE_CONNECTION=postgresql://galaxy:xxxxxxx@${DatabaseInstance.Endpoint.Address}:5432/galaxy \
                  -e GALAXY_CONFIG_DATABASE_ENGINE_OPTION_SERVER_SIDE_CURSORS=True \
                  -e GALAXY_CONFIG_DATABASE_ENGINE_OPTION_STRATEGY=threadlocal \
                  bgruening/galaxy-stable

shadycuz avatar Sep 29 '17 12:09 shadycuz

Can you remove the --restart=on-failure:10 please.

bgruening avatar Sep 29 '17 12:09 bgruening

@bgruening I can't run it on my local computer cause I'm trying to hit a remote RDS database, I don't know about this base image but the flavor I was using runs fine as long as I don't specify -e GALAXY_CONFIG_DATABASE_CONNECTION parameter. I am destroying the current environment and will bring up a new one shortly without the --restart flag.

shadycuz avatar Sep 29 '17 12:09 shadycuz

@bgruening same thing

root@ip-10-25-8-229:/galaxy-central# supervisorctl
condor                           STOPPED    Not started
cron                             STOPPED    Not started
docker                           STOPPED    Not started
galaxy:galaxy_nodejs_proxy       STOPPED    Not started
galaxy:galaxy_web                BACKOFF    Exited too quickly (process log may have details)
galaxy:handler0                  RUNNING    pid 148, uptime 0:02:26
galaxy:handler1                  FATAL      Exited too quickly (process log may have details)
nginx                            RUNNING    pid 145, uptime 0:02:26
postgresql                       RUNNING    pid 213, uptime 0:02:22
proftpd                          STOPPED    Not started
rabbitmq                         FATAL      Exited too quickly (process log may have details)
reports                          STOPPED    Not started

galaxy_web error log

*** WARNING: Can't find section "uwsgi" in INI configuration file /etc/galaxy/galaxy.ini ***
[uWSGI] getting INI configuration from /etc/galaxy/galaxy.ini
*** WARNING: Can't find section "uwsgi" in INI configuration file /etc/galaxy/galaxy.ini ***
[uWSGI] getting INI configuration from /etc/galaxy/galaxy.ini
*** WARNING: Can't find section "uwsgi" in INI configuration file /etc/galaxy/galaxy.ini ***
[uWSGI] getting INI configuration from /etc/galaxy/galaxy.ini
*** WARNING: Can't find section "uwsgi" in INI configuration file /etc/galaxy/galaxy.ini ***
[uWSGI] getting INI configuration from /etc/galaxy/galaxy.ini
*** WARNING: Can't find section "uwsgi" in INI configuration file /etc/galaxy/galaxy.ini ***
[uWSGI] getting INI configuration from /etc/galaxy/galaxy.ini
*** WARNING: Can't find section "uwsgi" in INI configuration file /etc/galaxy/galaxy.ini ***
[uWSGI] getting INI configuration from /etc/galaxy/galaxy.ini
*** WARNING: Can't find section "uwsgi" in INI configuration file /etc/galaxy/galaxy.ini ***

handler1 looks the same as the other times, just pasted the end of the log

INFO:galaxy.config:Logging at '10' level to '/home/galaxy/logs/handler1.log'
No handlers could be found for logger "__main__"
Traceback (most recent call last):
  File "./lib/galaxy/main.py", line 246, in <module>
    main()
  File "./lib/galaxy/main.py", line 242, in main
    app_loop(args, log)
  File "./lib/galaxy/main.py", line 114, in app_loop
    log=log,
  File "./lib/galaxy/main.py", line 102, in load_galaxy_app
    **kwds
  File "/galaxy-central/lib/galaxy/app.py", line 74, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 1025, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 97, in create_or_verify_database
    % ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '89' but this code expects version '134'.  Please backup your database and then migrate the schema by running 'sh manage_db.sh -c /etc/galaxy/galaxy.ini upgrade'.

shadycuz avatar Sep 29 '17 13:09 shadycuz