datachain icon indicating copy to clipboard operation
datachain copied to clipboard

Track created at date

Open shcheklein opened this issue 9 months ago • 4 comments

Since we list versions (do we?) we can use the first version in a versioned bucket as a created date. Let's add it to the File object.

shcheklein avatar Mar 08 '25 20:03 shcheklein

Good idea. But we should careful with this - it will require group-by / partition-by every time you work with files which is quite an overhead for large bucket.

I'd suggest postponing this until a clear need or we find a way to optimize this.

dmpetrov avatar Mar 10 '25 17:03 dmpetrov

I'd suggest postponing this until a clear need

it comes from a user request (priority - need clarification)

it will require group-by / partition-by every time you work with files

not sure I understand the point, tbh? could you clarify please?

shcheklein avatar Mar 10 '25 17:03 shcheklein

Maybe this can be calculated right in the listing process itself if it is guaranteed that we always get all versions of a file in the same "batch" when traversing through cloud storages (this needs to be checked). Then we would avoid re-calculating it every time with group by on the fly.

ilongin avatar Mar 13 '25 15:03 ilongin

Folks, let's deprioritize it until we confirm that modification time isn't sufficient - decreasing the priority

dmpetrov avatar Mar 18 '25 05:03 dmpetrov