Operator 0.13.0 fails new installation with 'relation "conf_setting" does not exist at character 158' GUI reports 502 error

[bartowl]

ISSUE TYPE

• Bug Report

SUMMARY

As already reported by someone at the end of https://github.com/ansible/awx/issues/1618 The installation goes smooth, but postgresql initialisation fails with following error:

2021-09-28 14:14:31.589 UTC [1] LOG: database system is ready to accept connections 2021-09-28 14:14:51.042 UTC [84] ERROR: relation "conf_setting" does not exist at character 158 2021-09-28 14:14:51.042 UTC [84] STATEMENT: SELECT "conf_setting"."id", "conf_setting"."created", "conf_setting"."modified", "conf_setting"."key", "conf_setting"."value", "conf_setting"."user_id" FROM "conf_setting" WHERE ("conf_setting"."key" IN ('ACTIVITY_STREAM_ENABLED', 'ACTIVITY_STREAM_ENABLED_FOR_INVENTORY_SYNC', 'ORG_ADMINS_CAN_SEE_ALL_USERS', 'MANAGE_ORGANIZATION_AUTH', 'TOWER_URL_BASE', 'REMOTE_HOST_HEADERS', 'PROXY_IP_ALLOWED_LIST', 'LICENSE', 'REDHAT_USERNAME', 'REDHAT_PASSWORD', 'SUBSCRIPTIONS_USERNAME', 'SUBSCRIPTIONS_PASSWORD', 'AUTOMATION_ANALYTICS_URL', 'INSTALL_UUID', 'DEFAULT_EXECUTION_ENVIRONMENT', 'CUSTOM_VENV_PATHS', 'AD_HOC_COMMANDS', 'ALLOW_JINJA_IN_EXTRA_VARS', 'AWX_ISOLATION_BASE_PATH', 'AWX_ISOLATION_SHOW_PATHS', 'AWX_TASK_ENV', 'INSIGHTS_TRACKING_STATE', 'PROJECT_UPDATE_VVV', 'AWX_ROLES_ENABLED', 'AWX_COLLECTIONS_ENABLED', 'AWX_SHOW_PLAYBOOK_LINKS', 'GALAXY_IGNORE_CERTS', 'STDOUT_MAX_BYTES_DISPLAY', 'EVENT_STDOUT_MAX_BYTES_DISPLAY', 'MAX_WEBSOCKET_EVENT_RATE', 'SCHEDULE_MAX_JOBS', 'AWX_ANSIBLE_CALLBACK_PLUGINS', 'DEFAULT_JOB_TIMEOUT', 'DEFAULT_INVENTORY_UPDATE_TIMEOUT', 'DEFAULT_PROJECT_UPDATE_TIMEOUT', 'ANSIBLE_FACT_CACHE_TIMEOUT', 'MAX_FORKS', 'LOG_AGGREGATOR_HOST', 'LOG_AGGREGATOR_PORT', 'LOG_AGGREGATOR_TYPE', 'LOG_AGGREGATOR_USERNAME', 'LOG_AGGREGATOR_PASSWORD', 'LOG_AGGREGATOR_LOGGERS', 'LOG_AGGREGATOR_INDIVIDUAL_FACTS', 'LOG_AGGREGATOR_ENABLED', 'LOG_AGGREGATOR_TOWER_UUID', 'LOG_AGGREGATOR_PROTOCOL', 'LOG_AGGREGATOR_TCP_TIMEOUT', 'LOG_AGGREGATOR_VERIFY_CERT', 'LOG_AGGREGATOR_LEVEL', 'LOG_AGGREGATOR_MAX_DISK_USAGE_GB', 'LOG_AGGREGATOR_MAX_DISK_USAGE_PATH', 'LOG_AGGREGATOR_RSYSLOGD_DEBUG', 'AUTOMATION_ANALYTICS_LAST_GATHER', 'AUTOMATION_ANALYTICS_LAST_ENTRIES', 'AUTOMATION_ANALYTICS_GATHER_INTERVAL', 'SESSION_COOKIE_AGE', 'SESSIONS_PER_USER', 'DISABLE_LOCAL_AUTH', 'AUTH_BASIC_ENABLED', 'OAUTH2_PROVIDER', 'ALLOW_OAUTH2_FOR_EXTERNAL_USERS', 'LOGIN_REDIRECT_OVERRIDE', 'PENDO_TRACKING_STATE', 'CUSTOM_LOGIN_INFO', 'CUSTOM_LOGO', 'MAX_UI_JOB_EVENTS', 'UI_LIVE_UPDATES_ENABLED', 'SOCIAL_AUTH_ORGANIZATION_MAP', 'SOCIAL_AUTH_TEAM_MAP', 'SOCIAL_AUTH_USER_FIELDS', 'AUTH_LDAP_SERVER_URI', 'AUTH_LDAP_BIND_DN', 'AUTH_LDAP_BIND_PASSWORD', 'AUTH_LDAP_START_TLS', 'AUTH_LDAP_CONNECTION_OPTIONS', 'AUTH_LDAP_USER_SEARCH', 'AUTH_LDAP_USER_DN_TEMPLATE', 'AUTH_LDAP_USER_ATTR_MAP', 'AUTH_LDAP_GROUP_SEARCH', 'AUTH_LDAP_GROUP_TYPE', 'AUTH_LDAP_GROUP_TYPE_PARAMS', 'AUTH_LDAP_REQUIRE_GROUP', 'AUTH_LDAP_DENY_GROUP', 'AUTH_LDAP_USER_FLAGS_BY_GROUP', 'AUTH_LDAP_ORGANIZATION_MAP', 'AUTH_LDAP_TEAM_MAP', 'AUTH_LDAP_1_SERVER_URI', 'AUTH_LDAP_1_BIND_DN', 'AUTH_LDAP_1_BIND_PASSWORD', 'AUTH_LDAP_1_START_TLS', 'AUTH_LDAP_1_CONNECTION_OPTIONS', 'AUTH_LDAP_1_USER_SEARCH', 'AUTH_LDAP_1_USER_DN_TEMPLATE', 'AUTH_LDAP_1_USER_ATTR_MAP', 'AUTH_LDAP_1_GROUP_SEARCH', 'AUTH_LDAP_1_GROUP_TYPE', 'AUTH_LDAP_1_GROUP_TYPE_PARAMS', 'AUTH_LDAP_1_REQUIRE_GROUP', 'AUTH_LDAP_1_DENY_GROUP', 'AUTH_LDAP_1_USER_FLAGS_BY_GROUP', 'AUTH_LDAP_1_ORGANIZATION_MAP', 'AUTH_LDAP_1_TEAM_MAP', 'AUTH_LDAP_2_SERVER_URI', 'AUTH_LDAP_2_BIND_DN', 'AUTH_LDAP_2_BIND_PASSWORD', 'AUTH_LDAP_2_START_TLS', 'AUTH_LDAP_2_CONNECTION_OPTIONS', 'AUTH_LDAP_2_USER_SEARCH', 'AUTH_LDAP_2_USER_DN_TEMPLATE', 'AUTH_LDAP_2_USER_ATTR_MAP', 'AUTH_LDAP_2_GROUP_SEARCH', 'AUTH_LDAP_2_GROUP_TYPE', 'AUTH_LDAP_2_GROUP_TYPE_PARAMS', 'AUTH_LDAP_2_REQUIRE_GROUP', 'AUTH_LDAP_2_DENY_GROUP', 'AUTH_LDAP_2_USER_FLAGS_BY_GROUP', 'AUTH_LDAP_2_ORGANIZATION_MAP', 'AUTH_LDAP_2_TEAM_MAP', 'AUTH_LDAP_3_SERVER_URI', 'AUTH_LDAP_3_BIND_DN', 'AUTH_LDAP_3_BIND_PASSWORD', 'AUTH_LDAP_3_START_TLS', 'AUTH_LDAP_3_CONNECTION_OPTIONS', 'AUTH_LDAP_3_USER_SEARCH', 'AUTH_LDAP_3_USER_DN_TEMPLATE', 'AUTH_LDAP_3_USER_ATTR_MAP', 'AUTH_LDAP_3_GROUP_SEARCH', 'AUTH_LDAP_3_GROUP_TYPE', 'AUTH_LDAP_3_GROUP_TYPE_PARAMS', 'AUTH_LDAP_3_REQUIRE_GROUP', 'AUTH_LDAP_3_DENY_GROUP', 'AUTH_LDAP_3_USER_FLAGS_BY_GROUP', 'AUTH_LDAP_3_ORGANIZATION_MAP', 'AUTH_LDAP_3_TEAM_MAP', 'AUTH_LDAP_4_SERVER_URI', 'AUTH_LDAP_4_BIND_DN', 'AUTH_LDAP_4_BIND_PASSWORD', 'AUTH_LDAP_4_START_TLS', 'AUTH_LDAP_4_CONNECTION_OPTIONS', 'AUTH_LDAP_4_USER_SEARCH', 'AUTH_LDAP_4_USER_DN_TEMPLATE', 'AUTH_LDAP_4_USER_ATTR_MAP', 'AUTH_LDAP_4_GROUP_SEARCH', 'AUTH_LDAP_4_GROUP_TYPE', 'AUTH_LDAP_4_GROUP_TYPE_PARAMS', 'AUTH_LDAP_4_REQUIRE_GROUP', 'AUTH_LDAP_4_DENY_GROUP', 'AUTH_LDAP_4_USER_FLAGS_BY_GROUP', 'AUTH_LDAP_4_ORGANIZATION_MAP', 'AUTH_LDAP_4_TEAM_MAP', 'AUTH_LDAP_5_SERVER_URI', 'AUTH_LDAP_5_BIND_DN', 'AUTH_LDAP_5_BIND_PASSWORD', 'AUTH_LDAP_5_START_TLS', 'AUTH_LDAP_5_CONNECTION_OPTIONS', 'AUTH_LDAP_5_USER_SEARCH', 'AUTH_LDAP_5_USER_DN_TEMPLATE', 'AUTH_LDAP_5_USER_ATTR_MAP', 'AUTH_LDAP_5_GROUP_SEARCH', 'AUTH_LDAP_5_GROUP_TYPE', 'AUTH_LDAP_5_GROUP_TYPE_PARAMS', 'AUTH_LDAP_5_REQUIRE_GROUP', 'AUTH_LDAP_5_DENY_GROUP', 'AUTH_LDAP_5_USER_FLAGS_BY_GROUP', 'AUTH_LDAP_5_ORGANIZATION_MAP', 'AUTH_LDAP_5_TEAM_MAP', 'RADIUS_SERVER', 'RADIUS_PORT', 'RADIUS_SECRET', 'TACACSPLUS_HOST', 'TACACSPLUS_PORT', 'TACACSPLUS_SECRET', 'TACACSPLUS_SESSION_TIMEOUT', 'TACACSPLUS_AUTH_PROTOCOL', 'SOCIAL_AUTH_GOOGLE_OAUTH2_KEY', 'SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET', 'SOCIAL_AUTH_GOOGLE_OAUTH2_WHITELISTED_DOMAINS', 'SOCIAL_AUTH_GOOGLE_OAUTH2_AUTH_EXTRA_ARGUMENTS', 'SOCIAL_AUTH_GOOGLE_OAUTH2_ORGANIZATION_MAP', 'SOCIAL_AUTH_GOOGLE_OAUTH2_TEAM_MAP', 'SOCIAL_AUTH_GITHUB_KEY', 'SOCIAL_AUTH_GITHUB_SECRET', 'SOCIAL_AUTH_GITHUB_ORGANIZATION_MAP', 'SOCIAL_AUTH_GITHUB_TEAM_MAP', 'SOCIAL_AUTH_GITHUB_ORG_KEY', 'SOCIAL_AUTH_GITHUB_ORG_SECRET', 'SOCIAL_AUTH_GITHUB_ORG_NAME', 'SOCIAL_AUTH_GITHUB_ORG_ORGANIZATION_MAP', 'SOCIAL_AUTH_GITHUB_ORG_TEAM_MAP', 'SOCIAL_AUTH_GITHUB_TEAM_KEY', 'SOCIAL_AUTH_GITHUB_TEAM_SECRET', 'SOCIAL_AUTH_GITHUB_TEAM_ID', 'SOCIAL_AUTH_GITHUB_TEAM_ORGANIZATION_MAP', 'SOCIAL_AUTH_GITHUB_TEAM_TEAM_MAP', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_URL', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_API_URL', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_KEY', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_SECRET', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_ORGANIZATION_MAP', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_TEAM_MAP', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_ORG_URL', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_ORG_API_URL', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_ORG_KEY', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_ORG_SECRET', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_ORG_NAME', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_ORG_ORGANIZATION_MAP', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_ORG_TEAM_MAP', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_TEAM_URL', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_TEAM_API_URL', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_TEAM_KEY', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_TEAM_SECRET', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_TEAM_ID', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_TEAM_ORGANIZATION_MAP', 'SOCIAL_AUTH_GITHUB_ENTERPRISE_TEAM_TEAM_MAP', 'SOCIAL_AUTH_AZUREAD_OAUTH2_KEY', 'SOCIAL_AUTH_AZUREAD_OAUTH2_SECRET', 'SOCIAL_AUTH_AZUREAD_OAUTH2_ORGANIZATION_MAP', 'SOCIAL_AUTH_AZUREAD_OAUTH2_TEAM_MAP', 'SAML_AUTO_CREATE_OBJECTS', 'SOCIAL_AUTH_SAML_SP_ENTITY_ID', 'SOCIAL_AUTH_SAML_SP_PUBLIC_CERT', 'SOCIAL_AUTH_SAML_SP_PRIVATE_KEY', 'SOCIAL_AUTH_SAML_ORG_INFO', 'SOCIAL_AUTH_SAML_TECHNICAL_CONTACT', 'SOCIAL_AUTH_SAML_SUPPORT_CONTACT', 'SOCIAL_AUTH_SAML_ENABLED_IDPS', 'SOCIAL_AUTH_SAML_SECURITY_CONFIG', 'SOCIAL_AUTH_SAML_SP_EXTRA', 'SOCIAL_AUTH_SAML_EXTRA_DATA', 'SOCIAL_AUTH_SAML_ORGANIZATION_MAP', 'SOCIAL_AUTH_SAML_TEAM_MAP', 'SOCIAL_AUTH_SAML_ORGANIZATION_ATTR', 'SOCIAL_AUTH_SAML_TEAM_ATTR') AND "conf_setting"."user_id" IS NULL) ORDER BY "conf_setting"."id" ASC

ENVIRONMENT

• AWX version: as in 0.13.0 by default
• Operator version: 0.13.0
• Kubernetes version: 1.20.9
• AWX install method: azure AKS

STEPS TO REPRODUCE

followed installation guide

EXPECTED RESULTS

WEB UI shows

ACTUAL RESULTS

WEB UI returns 502, container log from postgresql returns bunch of errors starting with as above

ADDITIONAL INFORMATION

AWX-OPERATOR LOGS

[shanemcd]

Seems like migrations didnt finish. Can you look at the logs for the operator and see if there's anything useful in there?

[alexeyostrovsky-rabove]

Any news here? Same issue for 0.14 and 0.15

[alexeyostrovsky-rabove]

Did not found any errors about migration in operator logs

[alexeyostrovsky-rabove]

Here is log file operator.log

[pashok2398]

Had the same issue (fresh install without migration)

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  ingress_type: ingress
  hostname: paveld-awx.xxx.xxx
  ingress_tls_secret: tls-cert

I just did kubectl delete pod awx-demo-postgres-0 to restart the pg and its worked fine. Looks like its trying to do migration by default, is there and flag to avoid it? Thanks

[kaitoy]

Same issue for 0.15.

[WeetA34]

Hello,

I had a similar issue with fresh install. i had to put image_pull_policy: Always

It's probably related to quay.io/ansible/awx-ee:latest.

[cmanzur]

Same errors running quay.io/ansible/awx:19.5.1 and quay.io/ansible/awx-ee:latest and postgres:12 from scratch

[WeetA34]

ok. It's probably a different problem I had. In my case, it was clearly due to an outdated awx-ee:latest image due to docker image caching and image pull policy set to IfNotPresent by default.

[Delrynn]

Still the same issue with 0.16.1.

I've tried deleting the postgres pod, and then I see the following in the web container:

2022-02-15 16:36:08,276 INFO spawned: 'awx-rsyslogd' with pid 240
2022-02-15 16:36:08,276 INFO spawned: 'awx-rsyslogd' with pid 240
2022-02-15 16:36:08,286 INFO exited: awx-rsyslogd (exit status 1; not expected)
2022-02-15 16:36:08,286 INFO exited: awx-rsyslogd (exit status 1; not expected)
rsyslogd: could not open config file '/var/lib/awx/rsyslog/rsyslog.conf': No such file or directory [v8.2102.0-6.el8 try https://www.rsyslog.com/e/2104 ]
rsyslogd: run failed with error -2104 (see rsyslog.h or try https://www.rsyslog.com/e/2104 to learn what that number means)

If I then delete the main awx pod, I'm back to it waiting for a database migration but looking for a different value on the DB pod:

2022-02-15 16:38:01.024 UTC [52] ERROR:  column main_organization.default_environment_id does not exist at character 309
2022-02-15 16:38:01.024 UTC [52] STATEMENT:  SELECT "main_organization"."id", "main_organization"."created", "main_organization"."modified", "main_organization"."description", "main_organization"."created_by_id", "main_organization"."modified_by_id", "main_organization"."name", "main_organization"."custom_virtualenv", "main_organization"."max_hosts", "main_organization"."default_environment_id", "main_organization"."admin_role_id", "main_organization"."execute_role_id", "main_organization"."project_admin_role_id", "main_organization"."inventory_admin_role_id", "main_organization"."credential_admin_role_id", "main_organization"."workflow_admin_role_id", "main_organization"."notification_admin_role_id", "main_organization"."job_template_admin_role_id", "main_organization"."execution_environment_admin_role_id", "main_organization"."auditor_role_id", "main_organization"."member_role_id", "main_organization"."read_role_id", "main_organization"."approval_role_id" FROM "main_organization" WHERE "main_organization"."name" = 'Default'
2022-02-15 16:38:05.783 UTC [58] ERROR:  column main_instancegroup.is_container_group does not exist at character 129
2022-02-15 16:38:05.783 UTC [58] STATEMENT:  SELECT "main_instancegroup"."id", "main_instancegroup"."name", "main_instancegroup"."created", "main_instancegroup"."modified", "main_instancegroup"."is_container_group", "main_instancegroup"."credential_id", "main_instancegroup"."pod_spec_override", "main_instancegroup"."policy_instance_percentage", "main_instancegroup"."policy_instance_minimum", "main_instancegroup"."policy_instance_list" FROM "main_instancegroup"

Then after a few minutes the web container shows the rsyslog.conf error.

my awx.yml:

---
 apiVersion: awx.ansible.com/v1beta1
 kind: AWX
 metadata:
   name: awx-demo
 spec:
   ingress_type: ingress
   hostname: awx-demo.cybr.com
   ingress_tls_secret: awx-ingress-tls
   service_type: NodePort
   nodeport_port: 30080
   web_resource_requirements:
     requests:
       cpu: 200m
       memory: 200Mi
   task_resource_requirements:
     requests:
       cpu: 200m
       memory: 200Mi
   ee_resource_requirements:
     requests:
       cpu: 200m
       memory: 200Mi

halp!

[jehutywong]

Version: 0.25.0 i was facing the same. But looks like the error can be ignored. After ~7mins waiting, the db will be reinitialized.

[DFazeli]

Same errors running quay.io/ansible/awx:21.6.0 and quay.io/ansible/awx-ee:latest and postgres:12 and postgres:12 from scratch

ERROR:  relation "django_migrations" does not exist at character 124
 2022-11-14 18:26:17.421 UTC [98] STATEMENT:  SELECT "django_migrations"."id", "django_migrations"."app", "django_migrations"."name", "django_migrations"."applied" 
2022-11-14 18:26:19.053 UTC [99] ERROR:  relation "conf_setting" does not exist at character 158                                                                   
2022-11-14 18:26:19.053 UTC [99] STATEMENT:  SELECT "conf_setting"."id", "conf_setting"."created", "conf_setting"."modified", "conf_setting"."key", "conf_setting" 2022-11-14 18:26:23.272 UTC [101] ERROR:  relation conf_setting" does not exist at character 158

My demo.yaml

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: tag-0-29-0
spec:
  service_type: ClusterIP
  postgres_image: repo.local/infra/postgres
  postgres_image_version: "13"
  postgres_storage_class: "nfs-client"
  redis_image: repo.local/infra/redis
  redis_image_version: "7"
  projects_storage_class: "nfs-client"
  projects_storage_size: "2Gi"
  control_plane_ee_image: repo.local/infra/awx-ee:latest
  ee_images:
    - name: awx-ee
      image: repo.local/infra/awx-ee:latest
  image: repo.local/infra/awx
  image_version: "21.6.0"
  init_container_image: repo.local/infra/awx-ee
  init_container_image_version: "latest" 

[goldyfruit]

Same here with Postgres 13 and the operator version 1.3.0. I has to delete the postgres pod (just the pod, not the pvc).

[ngotzmann]

I have the same issue with postgres 13 and the operator version 2.2.1. But deleting the postgres pod or wainting a few hours didnt solve the issue. I also tried out older operator versions, but got the same issue...

Does anyone had success in solving it?

[kuchynkap]

Hello,

I have the same problem using AWX Operator 2.4.0, Postgres 13, AWX 22.5.0 - fresh install. Waiting does not help, neither deleting the postgres pod.

Has anyone figured this out, please?

Thank you!

[vasanthtmurali]

Hello All,

I am facing the same issue. If someone already resolved this please mention the steps.

Thank you !

[Aaron199]

I have the same problem using AWX Operator 2.7.0, Postgres 13, AWX 23.3.0