calamari
calamari copied to clipboard
Published
20 hours ago •
ceph
ceph
FATAL: Ident authentication failed for user "calamari"
Guys
I am getting this problem on CentOS6 , Centos7 , Salt version 2014 and 2015. This shows that this problem has to do with Calamari.
The problem look like , postgresql database and user are not getting created for some reason. All the relevant services are getting started but calamari-ctl initialize is throwing error when it comes to database and its user.
Could you please tell me at which step the postgresql database and user is supposed to get created , is it the responsibility of calamari-ctl initialize command to create database and user ?
Could you please verify my sequence of steps :
- install salt , salt-master , salt-minion package
- install calamari-server package ( which will install supervisor , postgresql and httpd as dependencies )
- calamari-ctl initialize command
- If everything is OK , open dashboard
Installing calamari-server package
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : httpd-2.4.6-31.el7.centos.x86_64 1/6
Installing : postgresql-9.2.10-2.el7_1.x86_64 2/6
Installing : postgresql-server-9.2.10-2.el7_1.x86_64 3/6
Installing : mod_wsgi-3.4-12.el7_0.x86_64 4/6
Installing : supervisor-3.0-1.el7.noarch 5/6
Installing : calamari-server-1.3.0.1-49_g828960a.el7.centos.x86_64 6/6
setsebool: SELinux is disabled.
setsebool: SELinux is disabled.
Redirecting to /bin/systemctl restart salt-master.service
Redirecting to /bin/systemctl stop supervisord.service
Redirecting to /bin/systemctl start supervisord.service
Redirecting to /bin/systemctl stop httpd.service
Redirecting to /bin/systemctl start httpd.service
Thank you for installing Calamari.
Please run 'calamari-ctl initialize' as root to complete the installation.
Verifying : mod_wsgi-3.4-12.el7_0.x86_64 1/6
Verifying : postgresql-server-9.2.10-2.el7_1.x86_64 2/6
Verifying : postgresql-9.2.10-2.el7_1.x86_64 3/6
Verifying : supervisor-3.0-1.el7.noarch 4/6
Verifying : calamari-server-1.3.0.1-49_g828960a.el7.centos.x86_64 5/6
Verifying : httpd-2.4.6-31.el7.centos.x86_64 6/6
Installed:
calamari-server.x86_64 0:1.3.0.1-49_g828960a.el7.centos
Dependency Installed:
httpd.x86_64 0:2.4.6-31.el7.centos mod_wsgi.x86_64 0:3.4-12.el7_0 postgresql.x86_64 0:9.2.10-2.el7_1 postgresql-server.x86_64 0:9.2.10-2.el7_1 supervisor.noarch 0:3.0-1.el7
Complete!
[root@ceph-node1 ~]# systemctl status postgresql
postgresql.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; disabled)
Active: inactive (dead)
[root@ceph-node1 ~]# supervisorctl status
carbon-cache RUNNING pid 5116, uptime 0:01:25
cthulhu STARTING
[root@ceph-node1 ~]#
[root@ceph-node1 ~]# calamari-ctl initialize
[INFO] Loading configuration..
[INFO] Starting/enabling salt...
[INFO] Starting/enabling postgres...
[ERROR] (OperationalError) could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
None None
[ERROR] We are sorry, an unexpected error occurred. Debugging information has
been written to a file at '/tmp/2015-06-25_1419.txt', please include this when seeking technical
support.
[root@ceph-node1 ~]#
[root@ceph-node1 ~]# supervisorctl status
carbon-cache RUNNING pid 5116, uptime 0:02:45
cthulhu RUNNING pid 6308, uptime 0:00:01
[root@ceph-node1 ~]#
[root@ceph-node1 ~]# service postgresql status
Redirecting to /bin/systemctl status postgresql.service
postgresql.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; disabled)
Active: inactive (dead)
[root@ceph-node1 ~]#
[root@ceph-node1 ~]#
After restarting the node
[root@ceph-node1 calamari]#
[root@ceph-node1 calamari]# calamari-ctl initialize
[INFO] Loading configuration..
[INFO] Starting/enabling salt...
[INFO] Starting/enabling postgres...
[ERROR] (OperationalError) FATAL: Ident authentication failed for user "calamari"
None None
[ERROR] We are sorry, an unexpected error occurred. Debugging information has
been written to a file at '/tmp/2015-06-25_1426.txt', please include this when seeking technical
support.
[root@ceph-node1 calamari]#
[root@ceph-node1 calamari]# service postgresql status
Redirecting to /bin/systemctl status postgresql.service
postgresql.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled)
Active: active (running) since Thu 2015-06-25 17:22:07 EEST; 6min ago
Main PID: 1322 (postgres)
CGroup: /system.slice/postgresql.service
├─1322 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
├─1358 postgres: logger process
├─1397 postgres: checkpointer process
├─1398 postgres: writer process
├─1399 postgres: wal writer process
├─1400 postgres: autovacuum launcher process
└─1401 postgres: stats collector process
Jun 25 17:22:06 ceph-node1 systemd[1]: Starting PostgreSQL database server...
Jun 25 17:22:07 ceph-node1 systemd[1]: Started PostgreSQL database server.
[root@ceph-node1 calamari]#
[root@ceph-node1 calamari]#
[root@ceph-node1 calamari]# supervisorctl status
carbon-cache RUNNING pid 4546, uptime 0:04:26
cthulhu STARTING
[root@ceph-node1 calamari]#
@GregMeno Thanks for you initial pointer on this.
Its look like postgres does not have calamari user and database.
[root@ceph-node1 calamari]#
[root@ceph-node1 calamari]# psql -U calamari
psql: FATAL: Peer authentication failed for user "calamari"
[root@ceph-node1 calamari]#
[root@ceph-node1 calamari]# psql -h localhost -U calamari calamari -W
Password for user calamari:
psql: FATAL: Ident authentication failed for user "calamari"
[root@ceph-node1 calamari]# cat /etc/calamari/calamari.conf | grep -i db
db_path = postgresql://calamari:27HbZwr*g@localhost/calamari
db_log_level = WARN
db_engine = django.db.backends.postgresql_psycopg2
db_name = calamari
db_user = calamari
db_password = 27HbZwr*g
db_host = localhost
[root@ceph-node1 calamari]# psql -h localhost -U calamari calamari -W
Password for user calamari:
psql: FATAL: Ident authentication failed for user "calamari"
[root@ceph-node1 calamari]#
[root@ceph-node1 calamari]#
[root@ceph-node1 calamari]#
[root@ceph-node1 calamari]# sudo -u postgres psql
psql (9.2.10)
Type "help" for help.
postgres=#
postgres=#
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
postgres=#
@GregMeno
As advised by you i manually run salt-call with postgres template it failed for the first time complaining about postgresql config directory already exists , because calamari-ctl initialize command would have created one , then i removed ( rm -rf /var/lib/pgsql/data ) and rerun salt-call with postgres.sls and it worked nicely.
Finally i executed calamari-ctl initialize and that succeeded.
So should we draw conclusion out of it:
- After installing calamari-server package ( calamari-server-1.3.0.1-49_g828960a.el7.centos.x86_64 ) and running command calamari-ctl initialize , its not creating calamari user and database for postgres. Do you think this needs a fix ?
[root@ceph-node1 ~]# salt-call --local state.template /opt/calamari/salt-local/postgres.sls
[INFO ] Loading fresh modules for state activity
[INFO ] Running state [postgresql-setup initdb] at time 09:27:30.830015
[INFO ] Executing state cmd.run for postgresql-setup initdb
[INFO ] Executing command 'postgresql-setup initdb' in directory '/root'
[ERROR ] Command 'postgresql-setup initdb' failed with return code: 1
[ERROR ] stdout: Data directory is not empty!
[ERROR ] retcode: 1
[ERROR ] {'pid': 14875, 'retcode': 1, 'stderr': '', 'stdout': 'Data directory is not empty!'}
[INFO ] Completed state [postgresql-setup initdb] at time 09:27:30.960253
local:
----------
ID: postgresql_initdb
Function: cmd.run
Name: postgresql-setup initdb
Result: False
Comment: Command "postgresql-setup initdb" run
Started: 09:27:30.830015
Duration: 130.238 ms
Changes:
----------
pid:
14875
retcode:
1
stderr:
stdout:
Data directory is not empty!
----------
ID: /var/lib/pgsql/data/pg_hba.conf
Function: file.replace
Result: False
Comment: One or more requisite failed: /opt/calamari/salt-local/postgres.sls.postgresql_initdb
Started:
Duration:
Changes:
----------
ID: postgresql
Function: cmd.run
Name: systemctl enable postgresql || true; systemctl stop postgresql || true; systemctl start postgresql || true
Result: False
Comment: One or more requisite failed: /opt/calamari/salt-local/postgres.sls./var/lib/pgsql/data/pg_hba.conf
Started:
Duration:
Changes:
----------
ID: calamariuser
Function: postgres_user.present
Name: calamari
Result: False
Comment: One or more requisite failed: /opt/calamari/salt-local/postgres.sls.postgresql
Started:
Duration:
Changes:
----------
ID: calamaridb
Function: postgres_database.present
Name: calamari
Result: False
Comment: One or more requisite failed: /opt/calamari/salt-local/postgres.sls.calamariuser
Started:
Duration:
Changes:
Summary
------------
Succeeded: 0 (changed=1)
Failed: 5
------------
Total states run: 5
[root@ceph-node1 ~]#
So i deleted pgsql/data
[root@ceph-node1 ~]# rm -rf /var/lib/pgsql/data
Rerun salt-call
[root@ceph-node1 ~]# salt-call --local state.template /opt/calamari/salt-local/postgres.sls
[INFO ] Loading fresh modules for state activity
[INFO ] Running state [postgresql-setup initdb] at time 09:31:28.631567
[INFO ] Executing state cmd.run for postgresql-setup initdb
[INFO ] Executing command 'postgresql-setup initdb' in directory '/root'
[INFO ] {'pid': 17783, 'retcode': 0, 'stderr': '', 'stdout': 'Initializing database ... OK'}
[INFO ] Completed state [postgresql-setup initdb] at time 09:31:42.000247
[INFO ] Running state [/var/lib/pgsql/data/pg_hba.conf] at time 09:31:42.001968
[INFO ] Executing state file.replace for /var/lib/pgsql/data/pg_hba.conf
[INFO ] File changed:
---
+++
@@ -79,11 +79,11 @@
# "local" is for Unix domain socket connections only
local all all peer
# IPv4 local connections:
-host all all 127.0.0.1/32 ident
+host all all 127.0.0.1/32 md5
# IPv6 local connections:
-host all all ::1/128 ident
+host all all ::1/128 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
#local replication postgres peer
-#host replication postgres 127.0.0.1/32 ident
-#host replication postgres ::1/128 ident
+#host replication postgres 127.0.0.1/32 md5
+#host replication postgres ::1/128 md5
[INFO ] Completed state [/var/lib/pgsql/data/pg_hba.conf] at time 09:31:42.015374
[INFO ] Running state [systemctl enable postgresql || true; systemctl stop postgresql || true; systemctl start postgresql || true] at time 09:31:42.016367
[INFO ] Executing state cmd.run for systemctl enable postgresql || true; systemctl stop postgresql || true; systemctl start postgresql || true
[INFO ] Executing command 'systemctl enable postgresql || true; systemctl stop postgresql || true; systemctl start postgresql || true' in directory '/root'
[INFO ] {'pid': 17997, 'retcode': 0, 'stderr': '', 'stdout': ''}
[INFO ] Completed state [systemctl enable postgresql || true; systemctl stop postgresql || true; systemctl start postgresql || true] at time 09:31:58.530110
[INFO ] Running state [calamari] at time 09:31:58.531188
[INFO ] Executing state postgres_user.present for calamari
[INFO ] Executing command '/bin/psql --no-align --no-readline --no-password --dbname postgres -c \'SELECT setting FROM pg_catalog.pg_settings WHERE name = \'"\'"\'server_version\'"\'"\'\' -t' as user 'postgres' in directory '/var/lib/pgsql'
[INFO ] Executing command '/bin/psql --no-align --no-readline --no-password --dbname postgres -v datestyle=ISO,MDY -c \'COPY (SELECT pg_roles.rolname as "name",pg_roles.rolsuper as "superuser", pg_roles.rolinherit as "inherits privileges", pg_roles.rolcreaterole as "can create roles", pg_roles.rolcreatedb as "can create databases", pg_roles.rolcatupdate as "can update system catalogs", pg_roles.rolcanlogin as "can login", pg_roles.rolreplication as "replication", pg_roles.rolconnlimit as "connections", pg_roles.rolvaliduntil::timestamp(0) as "expiry time", pg_roles.rolconfig as "defaults variables" , COALESCE(pg_shadow.passwd, pg_authid.rolpassword) as "password" FROM pg_roles LEFT JOIN pg_authid ON pg_roles.oid = pg_authid.oid LEFT JOIN pg_shadow ON pg_roles.oid = pg_shadow.usesysid) TO STDOUT WITH CSV HEADER\'' as user 'postgres' in directory '/var/lib/pgsql'
[INFO ] Executing command '/bin/psql --no-align --no-readline --no-password --dbname postgres -c \'SELECT setting FROM pg_catalog.pg_settings WHERE name = \'"\'"\'server_version\'"\'"\'\' -t' as user 'postgres' in directory '/var/lib/pgsql'
[INFO ] Executing command '/bin/psql --no-align --no-readline --no-password --dbname postgres -v datestyle=ISO,MDY -c \'COPY (SELECT pg_roles.rolname as "name",pg_roles.rolsuper as "superuser", pg_roles.rolinherit as "inherits privileges", pg_roles.rolcreaterole as "can create roles", pg_roles.rolcreatedb as "can create databases", pg_roles.rolcatupdate as "can update system catalogs", pg_roles.rolcanlogin as "can login", pg_roles.rolreplication as "replication", pg_roles.rolconnlimit as "connections", pg_roles.rolvaliduntil::timestamp(0) as "expiry time", pg_roles.rolconfig as "defaults variables" FROM pg_roles) TO STDOUT WITH CSV HEADER\'' as user 'postgres' in directory '/var/lib/pgsql'
[INFO ] Executing command '/bin/psql --no-align --no-readline --no-password --dbname postgres -c \'CREATE ROLE "calamari" WITH INHERIT LOGIN ENCRYPTED PASSWORD \'"\'"\'md5062e5c74664bc2059db0fb06b78676db\'"\'"\'\'' as user 'postgres' in directory '/var/lib/pgsql'
[INFO ] {'calamari': 'Present'}
[INFO ] Completed state [calamari] at time 09:32:01.464795
[INFO ] Running state [calamari] at time 09:32:01.466028
[INFO ] Executing state postgres_database.present for calamari
[INFO ] Executing command '/bin/psql --no-align --no-readline --no-password --dbname postgres -v datestyle=ISO,MDY -c \'COPY (SELECT datname as "Name", pga.rolname as "Owner", pg_encoding_to_char(encoding) as "Encoding", datcollate as "Collate", datctype as "Ctype", datacl as "Access privileges", spcname as "Tablespace" FROM pg_database pgd, pg_roles pga, pg_tablespace pgts WHERE pga.oid = pgd.datdba AND pgts.oid = pgd.dattablespace) TO STDOUT WITH CSV HEADER\'' as user 'postgres' in directory '/var/lib/pgsql'
[INFO ] Executing command '/bin/psql --no-align --no-readline --no-password --dbname postgres -c \'CREATE DATABASE "calamari" WITH ENCODING = \'"\'"\'UTF8\'"\'"\' LC_COLLATE = \'"\'"\'en_US.UTF8\'"\'"\' LC_CTYPE = \'"\'"\'en_US.UTF8\'"\'"\' TEMPLATE = template0 OWNER = "calamari"\'' as user 'postgres' in directory '/var/lib/pgsql'
[INFO ] {'calamari': 'Present'}
[INFO ] Completed state [calamari] at time 09:32:04.236382
local:
----------
ID: postgresql_initdb
Function: cmd.run
Name: postgresql-setup initdb
Result: True
Comment: Command "postgresql-setup initdb" run
Started: 09:31:28.631567
Duration: 13368.68 ms
Changes:
----------
pid:
17783
retcode:
0
stderr:
stdout:
Initializing database ... OK
----------
ID: /var/lib/pgsql/data/pg_hba.conf
Function: file.replace
Result: True
Comment: Changes were made
Started: 09:31:42.001968
Duration: 13.406 ms
Changes:
----------
diff:
---
+++
@@ -79,11 +79,11 @@
# "local" is for Unix domain socket connections only
local all all peer
# IPv4 local connections:
-host all all 127.0.0.1/32 ident
+host all all 127.0.0.1/32 md5
# IPv6 local connections:
-host all all ::1/128 ident
+host all all ::1/128 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
#local replication postgres peer
-#host replication postgres 127.0.0.1/32 ident
-#host replication postgres ::1/128 ident
+#host replication postgres 127.0.0.1/32 md5
+#host replication postgres ::1/128 md5
----------
ID: postgresql
Function: cmd.run
Name: systemctl enable postgresql || true; systemctl stop postgresql || true; systemctl start postgresql || true
Result: True
Comment: Command "systemctl enable postgresql || true; systemctl stop postgresql || true; systemctl start postgresql || true" run
Started: 09:31:42.016367
Duration: 16513.743 ms
Changes:
----------
pid:
17997
retcode:
0
stderr:
stdout:
----------
ID: calamariuser
Function: postgres_user.present
Name: calamari
Result: True
Comment: The user calamari has been created
Started: 09:31:58.531188
Duration: 2933.607 ms
Changes:
----------
calamari:
Present
----------
ID: calamaridb
Function: postgres_database.present
Name: calamari
Result: True
Comment: The database calamari has been created
Started: 09:32:01.466028
Duration: 2770.354 ms
Changes:
----------
calamari:
Present
Summary
------------
Succeeded: 5 (changed=5)
Failed: 0
------------
Total states run: 5
[root@ceph-node1 ~]#
Rerun calamari-ctl
[root@ceph-node1 ~]# calamari-ctl initialize
[INFO] Loading configuration..
[INFO] Starting/enabling salt...
[INFO] Starting/enabling postgres...
[INFO] Initializing database...
[INFO] You will now be prompted for login details for the administrative user account. This is the account you will use to log into the web interface once setup is complete.
Username (leave blank to use 'root'):
Email address: [email protected]
Password:
Password (again):
Superuser created successfully.
[INFO] Initializing web interface...
[INFO] Starting/enabling services…
@ksingh7 Yes there is a fix needed here. I am running into the same issue after a system upgrade. My hunch is that the first two states in postgres.sls need more smarts.
What I think happened in you case was that postgres may have being installed / initdb outside of calamari which cause calamari-ctl initialize to get confused
If there's a way to tell that postgres initdb is not needed, perhaps that will help. I couldn't find one, so the rule is "to start from new, remove the package, and delete /usr/lib/pgsql by hand, or else the ctl init will fail".
@GregMeno IRRC postgres has not been installed & initialized before hand ( outside calamari )
@dmick yes the short term fix to this problem is to delete postgres data director manually and rerun calamari ctl.
We should try to fix this in calamari master.
Hello,
I had this issue on RHEL 7 OSP6 because the same node was allocated as both foreman and calamari host, and both use postgres.
I fixed by commenting the following lines in /opt/calamari/salt-local/postgres.sls
#postgresql_initdb:
# cmd:
# - run
# - name: postgresql-setup initdb
# - check_cmd:
# - /bin/true
# change 'host' auth to 'md5' for local hashed-password authorization
/var/lib/pgsql/data/pg_hba.conf:
file.replace:
- pattern: host(.*)ident
- repl: host\1md5
# - require:
@michaeltchapman Seems reasonable. We never considered that calamari would co-exist well with other applications that share it's system level dependencies.
We could allow this to continue if initdb has already been done.