Issue with updating status of collections mentioned in direct-read-namespaces to directreads collection

[sachinnagesh]

Hi @rwynn,

Monstache version for issue : v6.7.11

It is expected for collections mentioned in direct-read-namespaces, once collections data gets copied from MongoDB to ElasticSearch. it should update collections in directreads collection. But this is not happening in case of v6.7.11. It's working fine with v6.7.10.

Is there any specific change in this regard for v6.7.11?

[rwynn]

Hi @sachinnagesh ,

Monstache had an issue where if the application was forcibly closed it could erroneously mark completion by writing to the directreads collection (a false positive).

The work around implemented for this was to write the directreads collection at the end of the process after checking that direct reads were not in progress on the read side and the final flush call to Elasticsearch had completed.

If you run monstache with -exit-after-direct-reads you should see this collection being written after seeing the message "Direct reads completed". Please let me know if that is not the case.

[mix4242]

Hi @rwynn,

I am unfortunately observing the behaviour you mention in the last sentence:

I have upgraded from version 6.7.7 to 6.7.11. My montache 'initial' sync process that syncs the existing data & is then supposed to exit (config has exit-after-direct-reads = true), does not do so, but instead keeps running:

Log extract

2023-03-19T22:19:00.880Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:19:00 Started monstache version 6.7.11"
2023-03-19T22:19:00.880Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:19:00 Go version go1.18.3"
2023-03-19T22:19:00.880Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:19:00 MongoDB go driver v1.10.3"
2023-03-19T22:19:00.880Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:19:00 Elasticsearch go driver 7.0.31"
2023-03-19T22:19:00.881Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:19:00 Successfully connected to MongoDB version 4.4.13"
2023-03-19T22:19:00.891Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:19:00 Successfully connected to Elasticsearch version 7.17.0"
2023-03-19T22:19:00.894Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:19:00 Joined cluster APP-init"
2023-03-19T22:19:00.896Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:19:00 Starting work for cluster APP-init"
2023-03-19T22:19:00.896Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:19:00 Skipping direct reads for namespaces marked complete: ["APP-es.collectionA"]"
2023-03-19T22:19:00.896Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:19:00 Listening for events"

After it says Listening for events it never prints Direct reads completed or anything else for that matter, until I terminate it:

Log extract 2

2023-03-19T22:28:02.897Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:28:02 Starting clean shutdown"
2023-03-19T22:28:02.897Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:28:02 Stopping all workers"
2023-03-19T22:28:02.899Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:28:02 Shutting down"
2023-03-19T22:28:02.906Z [MonstacheRunner] Init Sync: "Process completed successfully."
2023-03-19T22:28:02.906Z [MonstacheRunner] Init Sync: "Completed after: 00:09:02.0097980"

When I downgrade to version 6.7.10, I see correct behaviour i.e. termination after direct-reads:

Log extract 3

2023-03-19T22:18:26.424Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:18:26 Resuming work for cluster APP-init"
2023-03-19T22:18:26.425Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:18:26 Skipping direct reads for namespaces marked complete: ["APP-es.collectionA"]"
2023-03-19T22:18:26.425Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:18:26 Listening for events"
2023-03-19T22:18:26.425Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:18:26 Direct reads completed"
2023-03-19T22:18:26.426Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:18:26 Stopping all workers"
2023-03-19T22:18:26.426Z [MonstacheRunner] Init Sync: "INFO 2023/03/19 22:18:26 Shutting down"
2023-03-19T22:18:26.431Z [MonstacheRunner] Init Sync: "Process completed successfully."
2023-03-19T22:18:26.432Z [MonstacheRunner] Init Sync: "Completed after: 00:00:00.0059530"

Is it perhaps because in my case it has previously completed direct reads (Skipping direct reads for namespaces marked complete) & so it doesn't terminate because of that?

[sachinnagesh]

@rwynn We are not using -exit-after-direct-reads. We just reverted back to 6.7.10 as with 6.7.11 it was not updating collection in directreads collection to mark it as completed. Every time when service was getting deployed, it was again copying whole collection to es index.