mimir icon indicating copy to clipboard operation
mimir copied to clipboard

Make Value Repair its own operator

Open okennedy opened this issue 6 years ago • 0 comments

"Value Repair" is the "repair" part of the Missing Value lens. It would be helpful to make this its own operator for two reasons:

  1. Missing Value Repair is something that can happen in multiple places (missing value lens, type inference lens)
  2. Missing Value Repair can be implemented in different ways, each involving a very different query structure: Imputation can be handled by projection, Remove bad data is a selection, Interpolation is a windowed aggregate,

Proposed structure:

ApplyValueRepair(
  model: String, // Referencing ModelManager
  input: Operator,
  target: Var, 
  constraint: Constraint  
)

Proposed semantics: Return a table based on [input] that (1) has an identical schema, (2) is "minimally different", and (3) for which [constraint] applied to [target] is satisfied for every row. Permitted edits include:

  • Delete rows
  • Modify [target]

InlineVGTerms (should be renamed now) should use [model] to select a repair to replace ApplyValueRepair with.

okennedy avatar Dec 20 '18 00:12 okennedy