OpenSearch Error - Not sure where to debug

[jlaura]

Running v2.3.0 on AWS OpenSearch. I am attempting to add additional items to an existing index and am seeing the following:

["error":{"name":"ResponseError","meta":{"body":{"error":{"root_cause":[{"type":"illegal_state_exception","reason":"Expected] valid sequence number for replicate op but was unassigned"}],"type":"illegal_state_exception","reason":"Expected valid sequence number for replicate op but was unassigned"},"status":500},"statusCode":500,

It looks like this error is related to OpenSearch and shards, so I ran GET _cat/shards?h=index,shard,prirep,state,unassigned.reason in the OS Dashboard as per this AWS article and it looks like everything is as expected.

Output for the collection I am attempting to add to:

mro_ctx_controlled_usgs_dtms                                 3 r STARTED 
mro_ctx_controlled_usgs_dtms                                 3 p STARTED 
mro_ctx_controlled_usgs_dtms                                 2 p STARTED 
mro_ctx_controlled_usgs_dtms                                 2 r STARTED 
mro_ctx_controlled_usgs_dtms                                 1 p STARTED 
mro_ctx_controlled_usgs_dtms                                 1 r STARTED 
mro_ctx_controlled_usgs_dtms                                 4 r STARTED 
mro_ctx_controlled_usgs_dtms                                 4 p STARTED 
mro_ctx_controlled_usgs_dtms                                 0 r STARTED 
mro_ctx_controlled_usgs_dtms                                 0 p STARTED

I can verify that no unassigned shards exist using: GET _cluster/allocation/explain?pretty.

Other Checks:

• too many shards
• skewed shards across nodes
• disk space issues
• Node issues: I also scaled the cluster, adding another node via the AWS UI (not the serverless.yml).
• Data creation issues: I manually created a new index
• STAC item validation issues: Item validate (v1.0.0) at staclint.com
• Something specific to the item: Tried a few new items
• Something specific to the collection: Tested updating an item in another collection (and it worked!)

Any chance that this error has been observed by any other devs or users? It looks like it is related to pushing an item to an existing collection, but the error is opaque to me.