Investigate high-performance data serialization
This issue should be done at the end of project https://github.com/asyml/forte/projects/3.
Is your feature request related to a problem? Please describe. Forte currently supports a simple serialization via JsonPickle. This format is fairly readable but the performance for loading and serialization are not optimized. Furthermore, we don't have control over the whole serialization process, which stops us from performing checking during the serialization
This issue needs to be further clarified after the main data structure is defined.
Describe the solution you'd like
- Investigate the alternatives and create a rough estimation
- Implement a different serialization that uses the new method.
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context
- This is part of the data efficiency project
- This PR should be made to the
new_datapackbranch.