Add support for the reduce operator

[tchaton]

🚀 Feature

Motivation

It is quite frequent users need to perform a reduce operation across a dataset. LitData could expose a custom map operator with state forwarding and automated distributed final reduction.

Pitch

Alternatives

Additional context

[deependujha]

Can you please provide any example, when someone would want to use reduce operator? This will help better in designing this feature.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bhimrazy]

Let's keep this — it's worth exploring and discussing it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.