s3gw icon indicating copy to clipboard operation
s3gw copied to clipboard

RFC: sfs file tree: store metadata in MP / version files

Open irq0 opened this issue 1 year ago • 4 comments

What needs to be done

Store metadata (e.g object id, mp id, commit time, owner) with the MP part / object version file in the SFS dir tree.

Why it needs to be done

It would help a fsck tool and a human debugging.

Design Space

Header / footer

Let file data start at a fixed offset. Append data after file data.

A fixed offset header needs to take alignment into account as SFS will never read it.

Can be implemented so that head -1 works.

XAttrs

Store everything in a single / separate xattrs.

xattrs have performance / overhead on the filesystem that needs to be taken into account. Backing up / copying the datastore may loose xattrs.

irq0 avatar Aug 10 '23 14:08 irq0

Is this intended to duplicate, supplement, or replace the metadata that's already present in the sqlite database?

tserong avatar Aug 11 '23 05:08 tserong

Duplicate. Put all information necessary with the file to make it self describing

irq0 avatar Aug 11 '23 07:08 irq0

Seems reasonable, and definitely helpful to humans debugging.

For fsck it adds another thing to check: that file metadata matches db metadata :-P

tserong avatar Aug 11 '23 08:08 tserong

This should be defined in an ADR

jhmarina avatar Sep 19 '23 13:09 jhmarina