Add training mode compression for zstd

[tchaton]

🚀 Feature

Motivation

https://github.com/facebook/zstd

https://python-zstandard.readthedocs.io/en/latest/dictionaries.html

Pitch

Alternatives

Additional context

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[deependujha]

• From zstd's readme:

---

Also, chatgpt says:

---

The graph shared has 10K different json files of roughly 1KB each.

LitData chunks on an average will be 64MB or more, but definitely not in a few KBs.

---

should I close the issue, or give it a try?

[bhimrazy]

Let’s keep this open for now — could be fun to try out for learning purposes.

Also, cool to see zstd coming to the Python 3.14 stdlib! 🚀 https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-pep784