job_type field was missing from .elastic-connectors-sync-jobs-v1

[ppf2]

We had a situation where we discovered that the job_type field was missing from .elastic-connectors-sync-jobs-v1's mapping definition. This resulted in term queries that would schedule/fetch sync jobs by a job_type silently failing (since the job_type was not defined as a term in index mappings, the term query wouldn't match any results from the index).

The workaround was to delete the full .elastic-connectors-sync-jobs-v1 index so that the Connector Service would re-create it with the correct mappings on startup.

This is created to determine:

1. If this is a potential index migration issue.
2. If this could be a result of the end user directly updating .elastic-connectors-sync-jobs-v1. If so, what kind of safeguard we can put into place.
3. Should the Connector Service detect corrupted system index schemas and self-heal or provide the necessary feedback to administrators with suggested remediation actions, etc...

cc: @artem-shelkovnikov @jedrazb

[artem-shelkovnikov]

During our investigation we were unable to understand the core reason of it due to lack of logging.

So we won't be able to understand why it happens, but we can instead add more logs/safeguards in-place to alleviate this.

I tried of thinking of the reasons why it could happen and failed. It was definitely not a user issue - they cannot delete mappings, so mappings failed to apply for some reason.

Should the Connector Service detect corrupted system index schemas and self-heal or provide the necessary feedback to administrators with suggested remediation actions, etc...

We would definitely love that - not sure how? Self-heal is not possible here, but we can provide feedback or diag tooling on our side until we switch to APIs with system indices, which seems to happen soon. Therefore, probably it's not worth to address now, only mark this as a known rare issue and see how it goes once we switch to APIs

[jedrazb]

I reproduced the issue:

• Create new cloud deployment <8.9 (I used 8.8.2)
• Create a connector, schedule a sync job, it can fail. It's important for the sync job index with mappings to be created
• Upgrade to 8.9+ (I upgraded to 8.13.0)
• I'm experiencing exact same issues as our users. It's now impossible to start syncs, sync job history is not shown.
  • It's due to the fact that job_type is missing in mappings (as keyword) and our logic assumes we can do keyword-specific queries.
  • During the stack upgrade the index mappings are not updated. See docs: change in template don't change the actual mappings, templates are only used upon index creation
  • Deleting sync jobs index (sync job history) fixes the issue as now the correct mappings would be applied from the index template added in 8.9. After deleting index it's sufficient to manually schedule a sync job for the index to be recreated with correct mappings.

The impact is limited to users that scheduled at least one sync job in <8.9 version and upgraded their stack.

Our index mappings use dynamic: false setting (docs) that means that: New fields (like job_type added in migration to 8.9) are ignored. These fields will not be indexed or searchable, but will still appear in the _source field of returned hits. These fields will not be added to the mapping, and new fields must be added explicitly. This is why job_type is still visible when inspecting the source of docs but is absent from the mapping.

[seanstory]

My suggestion for next steps:

• [x] document as a known issue and with the curl command workaround to add the field
• [x] update the existing ent-search migration that attempts to add the job_type field to documents to first execute an Update Mapping API request to add the job_type field to the mapping of an existing index (this will help customers on <8.10 versions that update to >=8.14)
• [x] scan our migrations to see if this same mistake has occurred elsewhere since we moved our index definitions to templates in Elasticsearch

[jedrazb]

curl command workaround to add the field

Verified that following command fixes the bug for subsequent syncs

PUT .elastic-connectors-sync-jobs-v1/_mapping
{
  "properties": {
    "job_type": {
      "type": "keyword"
    }
  }
}

[jedrazb]

scan our migrations to see if this same mistake has occurred elsewhere since we moved our index definitions to templates in Elasticsearch

I took the mappings diff between index versions created in 8.8 and the migrated to 8.13, and index versions created in 8.13.

We are missing following mappings in old version: For .elastic-connectors-sync-jobs-v1

• connector.sync_cursor (object) (not an issue, since data is still in source)
• job_type (keyword) big issue, breaking our logic

For .elastic-connectors-v1 we are missing,

• features.incremental_sync.properties.enabled (can be issue if we use boolean queries on that)
• last_access_control_sync_error keyword
• last_access_control_sync_scheduled_at date
• last_access_control_sync_status keyword
• last_incremental_sync_scheduled_at date
• scheduling nested mapping needs to be redeclared to account for different sync types ... (I don't think you can update existing mapping via PUT _mapping) EDIT: actually you can, since the new keys don't override old structure types
• sync_cursor object

Overall the only urgent thing is to add job_type keyword to migration as our logic assumes it's keyword. All other missing mappings are non-critical since the data is still present in the source, the only effect of missing mappings is that we can't run queries on those fields.

[seanstory]

Thanks for looking into it. While not critical, seems like this is a good time to go ahead and fix all of those, just to prevent any future issues.

[jedrazb]

Sharing requests that should be added in the migration to update the mappings, I will followup to add this to 8.14 migration logic

Connector index

PUT .elastic-connectors-v1/_mapping
{
  "properties": {
    "features": {
      "properties": {
        "incremental_sync": {
          "properties": {
            "enabled": {
              "type": "boolean"
            }
          }
        }
      }
    },
    "last_access_control_sync_error": {
      "type": "keyword"
    },
    "last_access_control_sync_scheduled_at": {
      "type": "date"
    },
    "last_access_control_sync_status": {
      "type": "keyword"
    },
    "last_incremental_sync_scheduled_at": {
      "type": "date"
    },
    "scheduling": {
      "properties": {
        "access_control": {
          "properties": {
            "enabled": {
              "type": "boolean"
            },
            "interval": {
              "type": "text"
            }
          }
        },
        "full": {
          "properties": {
            "enabled": {
              "type": "boolean"
            },
            "interval": {
              "type": "text"
            }
          }
        },
        "incremental": {
          "properties": {
            "enabled": {
              "type": "boolean"
            },
            "interval": {
              "type": "text"
            }
          }
        }
      }
    },
    "sync_cursor": {
      "type": "object"
    }
  }
}

Sync job index

PUT .elastic-connectors-sync-jobs-v1/_mapping
{
  "properties": {
    "job_type": {
      "type": "keyword"
    },
    "connector": {
      "properties": {
        "sync_cursor": {
          "type": "object"
        }
      }
    }
  }
}

[leemthompo]

• [x] document as a known issue and with the curl command workaround to add the field

[jedrazb]

ent-search module would take care of updating the mappings when migration to ES 8.14+ this issue should be fixed going forward