Pat Whelan

1719 instances of `Transform has failed` errors on serverless over the last 90 days: - 14 of `The object cannot be set twice` -> https://github.com/elastic/elasticsearch/issues/107215 - 25 of `Failed to...

> To help focus investigations, I would suggest we look at the behaviour of transforms whilst nodes leave and join the cluster. I believe it is likely we have some...

Will look into this as well: https://github.com/elastic/elasticsearch/issues/100891 It's likely we don't have to worry about some of these inconsistencies during a rollout if we can handle the rollout

Related to `data too large`: https://github.com/elastic/elasticsearch/issues/60391

> [endpoint.metadata_united-default-8.14.0] transform has failed; experienced: [Insufficient memory for search after repeated page size reductions to [0], unable to continue pivot, please simplify job or increase heap size on data...

> [endpoint.metadata_current-default-8.14.0] transform has failed; experienced: [task encountered irrecoverable failure: org.elasticsearch.index.shard.IllegalIndexShardStateException: CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED]].

@nerophon that seems to be an issue with the index. From searching around, it seems that it is [corrupted](https://discuss.elastic.co/t/java-io-eofexception-read-past-eof-niofsindexinput-error/12436). Do you know if the index is a Transform internal index,...

A few new ones ``` Caused by: java.lang.IllegalArgumentException: field [message] not present as part of path [message] ``` Doesn't seem to reoccur There are still a lot of WARNS due...

I'm not sure if this is the issue, but this looks like it can be an issue: [doNextSearch](https://github.com/prwhelan/elasticsearch/blob/main/x-pack/plugin/transform/src/test/java/org/elasticsearch/xpack/transform/transforms/TransformIndexerStateTests.java#L196) - Thread 1 (test thread) runs [maybeTriggerAsyncJob](https://github.com/prwhelan/elasticsearch/blob/main/x-pack/plugin/transform/src/test/java/org/elasticsearch/xpack/transform/transforms/TransformIndexerStateTests.java#L596), this starts Thread 2. -...

> > Do not use date filters which are less than a date value as this conflicts with logic applied at each checkpoint execution and data may be missed. >...