Relation "am_usage_uploaded_files" does not exist

[maron546]

Description: We have installed a distribuited deployment (Pattern 2), with two Analytics Worker instances. Everything seems working fine but, database side, the following error occurrs periodically:

2021-03-24 14:17:07 UTC [24783-14] xxx@WSO2AM_STATS_DB STATEMENT:  SELECT 1 FROM AM_USAGE_UPLOADED_FILES
2021-03-24 14:17:07 UTC [24787-7] xxx@WSO2AM_STATS_DB ERROR:  relation "am_usage_uploaded_files" does not exist at character 15

Below the database configurations in the deployment.yaml file of the Analytics Worker:

wso2.datasources:
  dataSources:
    # carbon metrics data source
    - name: WSO2_METRICS_DB
      description: The datasource used for dashboard feature
      jndiConfig:
        name: jdbc/WSO2MetricsDB
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:h2:${sys:carbon.home}/wso2/dashboard/database/metrics;AUTO_SERVER=TRUE'
          username: wso2carbon
          password: wso2carbon
          driverClassName: org.h2.Driver
          maxPoolSize: 30
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

    - name: WSO2_PERMISSIONS_DB
      description: The datasource used for dashboard feature
      jndiConfig:
        name: jdbc/PERMISSION_DB
        useJndiReference: true
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:postgresql://xxx/WSO2AM_PERMISSIONS_DB'
          username: xxx
          password: xxx
          driverClassName: org.postgresql.Driver
          maxPoolSize: 10
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

    - name: GEO_LOCATION_DATA
      description: "The data source used for geo location database"
      jndiConfig:
        name: jdbc/GEO_LOCATION_DATA
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:h2:${sys:carbon.home}/wso2/worker/database/GEO_LOCATION_DATA;AUTO_SERVER=TRUE'
          username: wso2carbon
          password: wso2carbon
          driverClassName: org.h2.Driver
          maxPoolSize: 50
          idleTimeout: 60000
          validationTimeout: 30000
          isAutoCommit: false

    - name: APIM_ANALYTICS_DB
      description: Datasource used for APIM Analytics
      jndiConfig:
        name: jdbc/APIM_ANALYTICS_DB
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:postgresql://xxx/WSO2AM_STATS_DB'
          username: xxx
          password: xxx
          driverClassName: org.postgresql.Driver
          maxPoolSize: 50
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

    #Main datasource used in API Manager
    - name: AM_DB
      description: Main datasource used by API Manager
      jndiConfig:
        name: jdbc/AM_DB
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:postgresql://xxx/apim_db'
          username: xxx
          password: xxx
          driverClassName: org.postgresql.Driver
          maxPoolSize: 10
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false


    - name: WSO2AM_MGW_ANALYTICS_DB
      description: "The datasource used for APIM MGW analytics data."
      jndiConfig:
        name: jdbc/WSO2AM_MGW_ANALYTICS_DB
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:h2:${sys:carbon.home}/wso2/worker/database/WSO2AM_MGW_ANALYTICS_DB;AUTO_SERVER=TRUE'
          username: wso2carbon
          password: wso2carbon
          driverClassName: org.h2.Driver
          maxPoolSize: 50
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

    - name: WSO2_CLUSTER_DB
      description: The MySQL datasource used for Cluster Coordination
      jndiConfig:
        name: jdbc/WSO2_CLUSTER_DB
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: "jdbc:postgresql://xxx/WSO2_CLUSTER_DB"
          username: xxx
          password: xxx
          driverClassName: org.postgresql.Driver
          maxPoolSize: 50
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

And for the Analytics Dashboard:

wso2.datasources:
  dataSources:
    # Dashboard data source
    - name: WSO2_DASHBOARD_DB
      description: The datasource used for dashboard feature
      jndiConfig:
        name: jdbc/DASHBOARD_DB
        useJndiReference: true
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:postgresql://xxx/WSO2AM_DASHBOARD_DB'
          username: xxx
          password: xxx
          driverClassName: org.postgresql.Driver
          maxPoolSize: 20
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

    - name: BUSINESS_RULES_DB
      description: The datasource used for dashboard feature
      jndiConfig:
        name: jdbc/BUSINESS_RULES_DB
        useJndiReference: true
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:postgresql://xxx/WSO2AM_BUSINESS_RULES_DB'
          username: xxx
          password: xxx
          driverClassName: org.postgresql.Driver
          maxPoolSize: 20
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

    # carbon metrics data source
    - name: WSO2_METRICS_DB
      description: The datasource used for dashboard feature
      jndiConfig:
        name: jdbc/WSO2MetricsDB
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:h2:${sys:carbon.home}/wso2/dashboard/database/metrics;AUTO_SERVER=TRUE'
          username: wso2carbon
          password: wso2carbon
          driverClassName: org.h2.Driver
          maxPoolSize: 20
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

    - name: WSO2_PERMISSIONS_DB
      description: The datasource used for dashboard feature
      jndiConfig:
        name: jdbc/PERMISSION_DB
        useJndiReference: true
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:postgresql://xxx/WSO2AM_PERMISSIONS_DB'
          username: xxx
          password: xxx
          driverClassName: org.postgresql.Driver
          maxPoolSize: 10
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

    #Data source for APIM Analytics
    - name: APIM_ANALYTICS_DB
      description: Datasource used for APIM Analytics
      jndiConfig:
        name: jdbc/APIM_ANALYTICS_DB
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:postgresql://xxx/WSO2AM_STATS_DB'
          username: xxx
          password: xxx
          driverClassName: org.postgresql.Driver
          maxPoolSize: 50
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

    #   # Sample datasource used in API Manager
    - name: AM_DB
      description: Main datasource used by API Manager
      jndiConfig:
        name: jdbc/AM_DB
      definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:postgresql://xxx/apim_db'
          username: xxx
          password: xxx
          driverClassName: org.postgresql.Driver
          maxPoolSize: 10
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

The AM_DB datasource is the same database which is defined under [database.apim_db] configuration in the deployment.toml file of the WSO2 API Manager.

Affected Product Version: WSO2 AM 3.1.0 WSO2 Analytics 3.1.0 WSO2 IS 5.10.0

OS, DB, other environment details and versions: Rancher PostgreSQL 9.5