[DATA] Update data preprocessing

[KB-Ding]

PR types

New features

PR changes

APIs

Description

support megatron dataset.

[paddle-bot[bot]]

Thanks for your contribution!

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 0.64516% with 462 lines in your changes are missing coverage. Please review.

Project coverage is 62.78%. Comparing base (7c6772b) to head (ed8fbe0). Report is 824 commits behind head on develop.

Files	Patch %	Lines
paddlenlp/data/indexed_dataset.py	0.00%	460 Missing :warning:
paddlenlp/trainer/trainer.py	60.00%	2 Missing :warning:

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #6223      +/-   ##
===========================================
- Coverage    63.16%   62.78%   -0.39%     
===========================================
  Files          529      530       +1     
  Lines        77184    77645     +461     
===========================================
- Hits         48755    48746       -9     
- Misses       28429    28899     +470     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[github-actions[bot]]

This Pull Request is stale because it has been open for 60 days with no activity. 当前Pull Request 60天内无活动，被标记为stale。

[github-actions[bot]]

This Pull Request is stale because it has been open for 60 days with no activity. 当前Pull Request 60天内无活动，被标记为stale。