Changing the _id does not delete it.

[lineaheshsh]

hello lewin

I am facing another problem.

Currently, my monstache configuration is as follows.

[monstache.toml] mongo-url = "mongodb://{id}:{password}@{ip}:{port}/{db}?authSource=admin"

elasticsearch-urls = [{ES_HOSTS}] elasticsearch-max-conns = 4 elasticsearch-max-seconds = 5 elasticsearch-max-bytes = 8000000

namespace-regex = '^db.' change-stream-namespaces = ["collectionA", "collectionB", "collectionC"]

dropped-collections = false dropped-databases = false

gzip = true stats = true index-stats = true resume = true resume-name = "replica" resume-strategy = 0 verbose = true index-as-update = true

[[mapping]] namespace = "collectionA" index = "collectionA"

[[mapping]] namespace = "collectionB" index = "collectionA"

[[mapping]] namespace = "collectionC" index = "collectionC"

[[script]] namespace = "collectionA" routing = true script = """ module.exports = function(doc) { if ( doc.rset != "success-data-import" ) { if ( doc.meta.isdel == "N" ) { return false; } var timestamp = new Date(); doc.meta.timestamp = timestamp; doc.meta._meta_monstache = {id: doc.meta.id, routing: doc.meta.id, index: "collectionA" }; return doc.meta; } } """

[[script]] namespace = "collectionB" routing = true script = """ module.exports = function(doc) { if ( doc.rset != "success-data-import" ) { var timestamp = new Date(); doc.meta.timestamp = timestamp; doc.meta._meta_monstache = {id: doc.meta.id, routing: doc.meta.goodsid, index: "collectionA" }; return doc.meta; } } """

My configuration is indexing into parent/child structure using join of Elasticsearch.

The current structure of MongoDB indexes data in doc.meta, and id values ​​also exist in doc.meta.

ex) collectionA Mongo Data { '_id' : ObjectId("030000001800276518000247"), 'core': "" 'meta': { "id": "1800276518000247", "join": "good", "a" : "a", ..... } }

I have data with this structure.

Realizing that the _id and _routing values ​​of Elasticsearch must match for join indexing, the following settings were added.

doc.meta._meta_monstache = {id: doc.meta.id, routing: doc.meta.id, index: "indexname" };

My problem is that the id value is not used as the Mongo Objectid, so it is not deleted when data is deleted.

Is there any good way for this?

[lineaheshsh]

I will summarize the above contents and raise it again.

Currently I'm not using MongoDB's ObjectId as the elasticsearch document_id.

Currently financed id with _meta_monstache settings.

The problem is that the delete sync is not in progress after the overwrite.

I manually found and applied the delete-strategy = 1 option and it hasn't been run yet.

I'm waiting for a good way.

[lineaheshsh]

re open

[lineaheshsh]

In case of update,

{"update":{"_index":"ticket","_id":"2100139521000058","routing":"2100139521000058"}},

The _id value is set well, but

For Delete, not found occurs as follows:

{"delete":{"_index":"ticket","_id":"030000002100139521000058","routing":"2100139521000058"}}, {"took":21,"errors":false,"items":[{"delete":{"_index":"ticket_mon_20210713","_type":"_doc","_id":"030000002100139521000058","_version ":1,"result":"not_found","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":500434,"_primary_term":1,"status ":404}}]},

[neilchen12]

Monstache version: 6.7.7 MongoDB version: 4.2.10 Elasticsearch version: 7.10.1

I have the same problem, after overried the output.ID, when delete the doc in MongoDB, reports Failed to find unique document ... for deletion using index pattern ... error, it still exists in Elasticsearch, try to set delete-strategy = 1 doesn't work either. This may be caused by the doDelete() function restricts req.id to op.id.

Is there any way to delete in this case ?

@rwynn please help

[rwynn]

I think if you implement func Process(input*monstachemap.ProcessPluginInput) error you could detect the delete event and then use the elastic client to issue a custom delete.

[rwynn]

Here is an example of how you can delete using the client inside a Process function: https://github.com/olivere/elastic/blob/release-branch.v7/delete_test.go#L50