feat(ml): dynamic batching

[mertalev]

Description

Models are optimized for processing inputs in batches, but until now the ML service has processed each input individually without any batching. This is particularly consequential when using acceleration devices; the overhead of moving data repeatedly to and from a device can undermine its performance. On the other hand, batching adds complexity to server code, particularly in the case of failures.

This PR aggregates request payloads within a certain timeframe into a single model input, improving performance for acceleration devices while being self-contained in the ML service. This does come at a cost: dynamic batching is less efficient than fixed batching, and in the case of CPUs, the overhead it adds outweighs the performance benefit. Because of this, it is disabled by default and can be enabled by setting MACHINE_LEARNING_MAX_BATCH_SIZE greater than 1.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
immich	⬜️ Ignored (Inspect)	Visit Preview		Oct 23, 2023 11:32pm

[mertalev]

Closing as this branch is very stale. It's better to make a new PR when this feature is revisited.