etl
etl copied to clipboard
metadata: display and presentation fields need to be automatically propagated on operations
Context
- We currently propagate and merge some/many metadata fields when you do common operations between
Table
columns - Since then, we have added more fields, and these do not get automatically propagated yet
One-liner
Fields like title_public
, or topic_tags
should be automatically propagated on operations, with some reasonable logic.
Motivation
- Wherever possible, we want to save work and make maintaining all this metadata easier
- We should try to propagate what additional fields we can
Technical notes
- Currently, when doing an operation between two indicators, say
c = a + b
, thepresentation
field in the metadata is propagated toc
only ifa.metadata.presentation
is identical tob.metadata.presentation
. - This means that many public titles or topic tags will not be automatically propagated (even though topic tags are now mandatory). We would like to improve the logic of this propagation.
- Since
presentation
is a dictionary, it needs a different merge strategy to plain strings
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Note that you need to do a rebuild after doing this, and then check the output of datadiff
, which will be crazy big and probably not super useful.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.