etl icon indicating copy to clipboard operation
etl copied to clipboard

metadata: display and presentation fields need to be automatically propagated on operations

Open pabloarosado opened this issue 1 year ago • 3 comments

Context

  • We currently propagate and merge some/many metadata fields when you do common operations between Table columns
  • Since then, we have added more fields, and these do not get automatically propagated yet

One-liner

Fields like title_public, or topic_tags should be automatically propagated on operations, with some reasonable logic.

Motivation

  • Wherever possible, we want to save work and make maintaining all this metadata easier
  • We should try to propagate what additional fields we can

Technical notes

  • Currently, when doing an operation between two indicators, say c = a + b, the presentation field in the metadata is propagated to c only if a.metadata.presentation is identical to b.metadata.presentation.
  • This means that many public titles or topic tags will not be automatically propagated (even though topic tags are now mandatory). We would like to improve the logic of this propagation.
  • Since presentation is a dictionary, it needs a different merge strategy to plain strings

pabloarosado avatar Oct 30 '23 09:10 pabloarosado

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Feb 13 '24 02:02 stale[bot]

Note that you need to do a rebuild after doing this, and then check the output of datadiff, which will be crazy big and probably not super useful.

larsyencken avatar Mar 07 '24 10:03 larsyencken

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar May 07 '24 01:05 stale[bot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Jul 06 '24 15:07 stale[bot]