24000 images, memory leak caused crashes during construction.

[lichangy]

24000 images, memory leak caused crashes during construction. The "prediction" method of MLModel leads to a continuous increase in memory.

[mazzzystar]

You're right. I tried to write a version using Claude 3.5 Sonnet that controls memory, for example, checking the current memory after each batch and then updating the batch size. This seems to be able to control memory usage. However, the problem is that the indexing speed becomes significantly slower.

If you don't mind the indexing speed slowing down, you can give it a try.

[antgiant]

I would be willing to give it a try.