delta icon indicating copy to clipboard operation
delta copied to clipboard

Use IndexedFile for DeltaSource admission control instead of FileAction

Open bart-samwel opened this issue 1 year ago • 0 comments

Which Delta project/connector is this regarding?

  • [X] Spark
  • [ ] Standalone
  • [ ] Flink
  • [ ] Kernel
  • [ ] Other (fill in here)

Description

This is a small refactoring. The DeltaSource has admission control (e.g. maxFilesPerTrigger), and the code is based on FileAction objects. However, all the calling code really has IndexedFile objects, and has to do manual conversions to deal with the fact that not all IndexedFile objects have file actions. This PR simplifies this by making the admission control based on IndexedFile.

How was this patch tested?

Existing unit tests.

Does this PR introduce any user-facing changes?

No.

bart-samwel avatar Nov 21 '23 10:11 bart-samwel