lightning support automatically fixing the data duplication

[glorv]

Feature Request

Describe your feature request related problem:

Lightning should support auto-detecting data duplication and automatically fix the data duplication.

Describe the feature you'd like:

Fix the data duplication by removing all the rows that are in an inconsistent state. Thus, after lightning importing, data in the cluster is in a compatible status.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy: