feat: add tqdm progress bar when training kmeans with CPU

[westonpace]

[westonpace]

I have mixed feelings about this PR. The actual kmeans training is several calls deep. In order to get progress to it we have to pass around a lot of tramp data. I don't think this is the right approach to take but it does offer a good starting point for the discussion:

• A simpler approach would be to just use log!(...) statements.
• If this were a higher level language I might suggest an "async local" or some kind of aspect oriented approach but I don't know that Rust has gotten many fans of the concept yet.
• We could also make progress reporting static (e.g. like logging or tracing).

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 81.48148% with 15 lines in your changes missing coverage. Please review.

Project coverage is 79.97%. Comparing base (bc4d863) to head (575a073). Report is 2 commits behind head on main.

Files	Patch %	Lines
rust/lance/src/index/vector/ivf.rs	72.50%	10 Missing and 1 partial :warning:
rust/lance-linalg/src/kmeans.rs	88.00%	3 Missing :warning:
rust/lance-index/src/vector/kmeans.rs	50.00%	0 Missing and 1 partial :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2557      +/-   ##
==========================================
- Coverage   80.01%   79.97%   -0.05%     
==========================================
  Files         209      210       +1     
  Lines       59821    59891      +70     
  Branches    59821    59891      +70     
==========================================
+ Hits        47868    47900      +32     
- Misses       9129     9144      +15     
- Partials     2824     2847      +23     

Flag	Coverage Δ
unittests	79.97% <81.48%> (-0.05%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[wjones127]

The actual kmeans training is several calls deep. In order to get progress to it we have to pass around a lot of tramp data.

Would you feel better about it if there were more passed down with it? I think all the same levels we want to pass down progress, we also want to pass down CPU threadpools, memory limit, and such. Though I suppose these could all just be static Arc<Mutex<T>> inside lance-core.

[westonpace]

Would you feel better about it if there were more passed down with it? I think all the same levels we want to pass down progress, we also want to pass down CPU threadpools, memory limit, and such. Though I suppose these could all just be static Arc<Mutex<T>> inside lance-core.

@wjones127

:grimacing: I think I would feel worse :laughing:

I guess the question is whether users need per-operation limits / logging / status or if per-process limits / logging / status are sufficient?

I was pondering a little what per-process progress bars might look like and I think it could be handled ok. The "begin" method could return something much like a span. The trait would then receive a UUID for the operation. Something like tqdm could use the position argument to report multiple progress bars at once.