Elasticsearch reindex command cannot be run on HK cluster

[louischan-oursky]

2024/05/01 15:38:26 App): reindexing

Failed to update last_indexed_at of user . pq: unexpected Parse response 'C'

Error: sql: transaction has already been committed or rolled back

[linear[bot]]

DEV-1270 Elasticsearch reindex command cannot be run on HK cluster

[tung2744]

I suspect there is some race condition in this line:

https://github.com/authgear/authgear-server/blob/main/cmd/authgear/elasticsearch/reindex.go#L229-L231

The problem maybe UpdateLastIndexedAt is using the same db handle used by QueryPage, which internally assigning a same variable in HookHandle tx. I guess maybe they are running at the same time causing the problem.
Maybe we should UpdateLastIndexedAt in a queue.

[fungc-io]

@tung2744 do we still observe this problem in staging?