tidb-lightning icon indicating copy to clipboard operation
tidb-lightning copied to clipboard

Published 20 hours ago • verified publisher icon pingcap

[5.0] Online import

Open kennytm opened this issue 4 years ago • 1 comments

Description

Lightning currently requires the cluster to be offline and empty because

  1. SST import violates ACID
  2. Import mode trades space amplification for write speed
  3. Region-split works better on empty ranges
  4. Cluster performance is reduced.

We may support online import based on pingcap/br#87.

Category

  • Feature
  • Performance

Value

Value description

This is needed for data-hub and data middle platform cases. A lot of cases, users have not only real-time data but also third party data, to import and analyze together with transactional data. This might as well be merged together with batch scenario and speed them up all together.

Value score

  • (TBD) / 5

Workload estimation

  • 30 person-day

Time

GanttStart: YYYY-MM-DD GanttDue: YYYY-MM-DD

kennytm avatar May 28 '20 20:05 kennytm

discussed with @ilovesoup and after a collection of scenarios, there are no specific scenarios that require this feature, so we can lower our priority to P2 and prioritize it later when we encounter a specific scenario.

also /cc @kennytm @overvenus @scsldb

IANTHEREAL avatar Jul 01 '20 04:07 IANTHEREAL