noobaa-core icon indicating copy to clipboard operation
noobaa-core copied to clipboard

Fixed map_common_prefixes queries

Open dannyzaken opened this issue 2 years ago • 0 comments

Signed-off-by: Danny Zaken [email protected]

Explain the changes

  • Changed pl/sql function map_common_prefixes to fetch all required entries in one call.
  • instead of querying for 1000 keys and then extracting the common prefixes from it (can be very few common prefixes if there are a lot of objects in that "dir"), it iterates over the objectmds until reaching the max keys or the end.
  • To avoid unnecessary scanning of objects - if we find a common prefix, we set the marker to be the next possible value for a common prefix (by incrementing the last character by 1)
  • These changes rely on the DB having collation of LC_COLLATE = 'C'. it will not work on DBs with utf8 collation

Issues: Fixed #xxx / Gap #xxx

  1. More cleanup can be done after this change. for example, it is probably unnecessary to redo the list_object queries in the object_server, since now it should return all of the required count https://github.com/noobaa/noobaa-core/blob/5b6bcb568bbb699e735523067b83a37e622e8db9/src/server/object_services/object_server.js#L1028-L1031

Testing Instructions:

  • [ ] Doc added/updated
  • [ ] Tests added

dannyzaken avatar Aug 11 '22 04:08 dannyzaken