feat: add support for packing tokenized datasets

[kmehant]

What does this PR do?

Fixes #1848

Before submitting

• [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [x] Did you read the contributor guideline, Pull Request section?
• [x] Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case. #1848
• [ ] Did you make sure to update the documentation with your changes? Here are the documentation guidelines.
• [x] Did you write any new necessary tests?

Who can review?

@qgallouedec

Anyone from the community!