ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

iterate_type with prefix can return incomplete results with S3 storage

Open alexowens90 opened this issue 2 years ago • 2 comments
trafficstars

https://github.com/man-group/ArcticDB/blob/master/cpp/arcticdb/storage/s3/s3_storage-inl.hpp#L359-L360

Underlying cause is that the index type (which comes before the symbol name in the S3 path) is hard-coded based on whether the keytype is ref or atom. This is incorrect, for example, with symbol list keys, where __add__ and __delete__ keys have "Unknown" index type, but compacted keys have "Timestamp".

Short term - should confirm that no current usages of iterate_type are returning incomplete sets of keys.

Longer term - fix to check all possible index types for the provided key type.

alexowens90 avatar Apr 11 '23 08:04 alexowens90

I don't think anything should have "Unknown" index type, it seems like that's the bug

willdealtry avatar Apr 11 '23 09:04 willdealtry

At least for the purposes of iterate_type, it's assumed that all ref keys have unknown index type

alexowens90 avatar Apr 11 '23 13:04 alexowens90