go-zero icon indicating copy to clipboard operation
go-zero copied to clipboard

使用mongodb和缓存获取数据错乱

Open themoonbear opened this issue 3 years ago • 19 comments

4B1FDC46-FCAE-4537-85CC-12ED1758A7A3 `

themoonbear avatar Aug 19 '22 07:08 themoonbear

Because too many requests to access the same non-exist data will bring down the service.

kevwan avatar Aug 19 '22 09:08 kevwan

我的英文不太好, 还是使用中文表达吧. 从生成的代码看, 插入的时候,数据没有进入缓存, 获取的时候也没有从缓存中获取, 那这个缓存层在哪里起作用的呢? 有没有例子可以参考一下

themoonbear avatar Aug 20 '22 02:08 themoonbear

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


My English is not very good, so I should use Chinese expression. From the generated code, when inserting, the data does not enter the cache, nor does it get from the cache when retrieving, so where does this cache layer work? Yes There is no example to refer to

Issues-translate-bot avatar Aug 20 '22 02:08 Issues-translate-bot

Delete when insert, and queries checks cache first.

kevwan avatar Aug 20 '22 14:08 kevwan

Delete when insert, and queries checks cache first. 看源码有看到, 在插入的时候删除了占位符的缓存, 但是每次调用FindOne的时候也是用的占位符, 那不是还是从数据库里取数据吗? 就是这块没看懂.

themoonbear avatar Aug 20 '22 14:08 themoonbear

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


Delete when insert, and queries checks cache first. Looking at the source code, you can see that the cache of the placeholder is deleted when it is inserted, but the placeholder is also used every time FindOne is called. Isn't that still fetching data from the database? I just didn't understand this one.

Issues-translate-bot avatar Aug 20 '22 14:08 Issues-translate-bot

One minute for placeholder cache.

kevwan avatar Aug 20 '22 14:08 kevwan

在插入的时候 if !data.ID.IsZero() {} 这段代码是bug吧? 应该是空的时候才生成一个吧?

themoonbear avatar Aug 21 '22 06:08 themoonbear

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


When inserting if !data.ID.IsZero() {}, is this code a bug? It should be generated when it is empty, right?

Issues-translate-bot avatar Aug 21 '22 06:08 Issues-translate-bot

Generated code?

kevwan avatar Aug 21 '22 06:08 kevwan

是的, 包括在调用FindOne的时候使用空ID做为KEY, 都应该会导致问题吧. 如果1号记录被第一次调用的时候,空ID被设置了1号记录,第二次2号来获取的时候,还使用空ID,那不从缓存层把1号记录返回了吗?

themoonbear avatar Aug 21 '22 10:08 themoonbear

InsertOne 中的 key 是删除缓存,不管 id 是否存在,对业务都不影响,FindOne 不可能有空 ID,因为已经做了id 校验

The key in function InsertOne is used to delete redis cache, it has no business effect whether the id exists or not, and the key in function FindOne, it is impossible that the case as you said it will return a empty data if the ID is empty, because there has a validation for ID before invoke FindOne.

kesonan avatar Aug 23 '22 02:08 kesonan

InsertOne 中的 key 是删除缓存,不管 id 是否存在,对业务都不影响,FindOne 不可能有空 ID,因为已经做了id 校验

The key in function InsertOne is used to delete redis cache, it has no business effect whether the id exists or not, and the key in function FindOne, it is impossible that the case as you said it will return a empty data if the ID is empty, because there has a validation for ID before invoke FindOne.

你好,我理解设计的目的,但从用goctl生成的代码看,我不确定这样的代码逻辑是否正确,所有想确认一下是否是goctl生成的代码是错误的。 从生成代码看,FindOne使用的是空ID下去查的,你说了已做了ID校验请问是在哪一步?

themoonbear avatar Aug 25 '22 01:08 themoonbear

空 id 在调用 ObjectIDFromHex 时就会报错,这里设计没有问题的哈,你可以写单测试一下哈

It will report an error if you given an empty id string and call ObjectIDFromHex, you can write a unit test to verify FindOne method.

kesonan avatar Sep 01 '22 14:09 kesonan

你好,我说的是截图里划红框的位置,缓存的KEY,截图代码是用goctl生成的,没有做过改动。在插入时判断插入数据是否有ID,如果有要重新生成一个,如果没有则缓存ID是空ID,执行过程中会删除空ID对应的缓存。在查询时,是用空ID到缓存里查,没有则从数据库里加到缓存中,这样这条数据是以空ID做为key存在缓存中的。

themoonbear avatar Sep 07 '22 02:09 themoonbear

Yes, it should be in cache. I've talked couple times in my presentations.

If we don't put the placeholder in cache, the cache cluster might be flooded for not-existing data.

kevwan avatar Sep 12 '22 11:09 kevwan

Yes, it should be in cache. I've talked couple times in my presentations.

If we don't put the placeholder in cache, the cache cluster might be flooded for not-existing data.

之前较忙没有继续跟进这个问题,占位符的设计目的无论从论坛和你的英文回复中我都清楚了,但并没有解决我遇到的问题,第一:调用InsertOne方法插入数据后如何知道该条记录在MongoDB中的ID值。第二:数据库中有两条数据,在调用FindeOne时由于第一条被占位符缓存,第二条取到的是第一条的数据。 Dingtalk_20220930143202 Dingtalk_20220930141503 Dingtalk_20220930141651

themoonbear avatar Sep 30 '22 06:09 themoonbear

@anqiansong 你好,可否根据我的测试结果给与解答?

themoonbear avatar Sep 30 '22 06:09 themoonbear

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


Hello @anqiansong, can you follow up on my test results and give me an answer?

Issues-translate-bot avatar Sep 30 '22 06:09 Issues-translate-bot