oio-sds
oio-sds copied to clipboard
Ambiguity of consecutive remove and create events
ISSUE TYPE
- Feature Idea
- Potential Bug
COMPONENT NAME
OpenIO event interface
SDS VERSION
openio 4.3.1
SUMMARY
Henceforward a create event is storage.content.new
and remove event is storage.content.deleted
.
Currently, there are three scenarios where consecutive remove and create events are sent with no way of differentiating between the following cases:
- An object is removed and a new one is created.
- An object is replaced (no object removal call was issued from a client)
- A new object chunk is written.
For example, this is an example of a create event. In all above cases, the fields are equal (except size
and mtime
). Interestingly, ctime
is also unchanged in case 2, although it is technically a new object with a new ctime
. Is this a bug or intented?:
storage.content.new json: {
"data" :
[
{
"ctime" : 1553594179,
"deleted" : false,
"header" : "DCF93055FC840500CFE0EECEA531E335",
"mtime" : 1553594318,
"name" : "test.mp3",
"type" : "aliases",
"version" : 1553594179451362
},
{
"chunk-method" : "plain/nb_copy=1",
"hash" : "",
"id" : "DCF93055FC840500CFE0EECEA531E335",
"mime-type" : "application/octet-stream",
"policy" : "SINGLE",
"size" : 5811819,
"type" : "contents_headers"
}
],
"event" : "storage.content.new",
"job_id" : "140470",
"part" : 0,
"parts" : 1,
"request_id" : "E5353587114D3D271E6690A75B7974B8",
"url" :
{
"account" : "delvefs",
"content" : "DCF93055FC840500CFE0EECEA531E335",
"id" : "093597B972937B6D59781FF7ED86AF559C6ED06CCBA749A4F4B4A00A872CC96A",
"ns" : "OPENIO",
"path" : "test.mp3",
"user" : "abc"
},
"when" : 1553594318771714
}
For our use case, we need to somehow differentiate between all cases. As this is a feature idea, we suggest the following changes to the event interface for the above cases:
- An object is removed and a new one is created. : Send one remove and one create request (no change required as it is current functionality).
-
An object is replaced (no object removal call was issued from a client) : Either send a create event with a specific flag (alternatively change
ctime
) or send astorage.content.update
event, which modifyctime
,mtime
, andsize
. -
A new object chunk is written. : Since only the chunk is modified and not the whole object replaced on each write, consecutive remove and create events do not seem approriate for this case. Perhaps
storage.content.update
could be used here as well, or alternatively usestorage.chunk.new
to reflect thesize
andmtime
update of the object.
I am aware that such changes might not be possible due to how OpenIO is processing the events itself or due to backwards compatibility, however, if there is another solution to this issue, it would be very useful to us.
STEPS TO REPRODUCE
Follow above described cases
Hello. I did some investigation. Here is a recap.
Operation | Event type | Contains user metadata | Contains chunks | Used by |
---|---|---|---|---|
Object create | storage.content.new | yes | no | Replicator |
Object delete | storage.content.deleted | no | yes | Content cleaner, Replicator |
Object create (overwrite) (1) | storage.content.deleted | no | yes | Content cleaner, Replicator |
Object create (overwrite) (2) | storage.content.new | yes | no | Replicator |
Object append | storage.content.append | no (can't be changed) | no | Replicator |
Object update (chunk mod) (1) | storage.content.deleted | no (missing version) | yes (partial) | Content cleaner, Replicator |
Object update (chunk mod) (2) | storage.content.new | yes (partial) | no | Replicator |
Object change property | storage.content.update | yes | no | Replicator |
Object truncate (1) | storage.content.deleted | no (missing header) | yes (partial) | Content cleaner, Replicator |
Object truncate (2) | storage.content.new | yes (partial) | no | Replicator |
As you pointed out, it is not possible (or hard) to know which operation triggered the event. For OpenIO SDS internal operations, we did not need to distinguish them.
I see two possible evolutions:
- Create new types of events for object overwrite, update and truncation, which could include both contents of
storage.content.new
andstorage.content.deleted
events. - Keep the existing events, but add a new field telling which kind of operation triggered the event emission. This solution is easier to implement, and does not break compatibility.
Hi Florent, thanks for the detailed reply and considering the change!
I don't think it is necessary to create new types of events. Keeping the existing events and adding some additional information, as you suggested in Option 2, is sufficient and would help significantly.