forte icon indicating copy to clipboard operation
forte copied to clipboard

Investigate high-performance data serialization

Open hunterhector opened this issue 4 years ago • 0 comments

This issue should be done at the end of project https://github.com/asyml/forte/projects/3.

Is your feature request related to a problem? Please describe. Forte currently supports a simple serialization via JsonPickle. This format is fairly readable but the performance for loading and serialization are not optimized. Furthermore, we don't have control over the whole serialization process, which stops us from performing checking during the serialization

This issue needs to be further clarified after the main data structure is defined.

Describe the solution you'd like

  1. Investigate the alternatives and create a rough estimation
  2. Implement a different serialization that uses the new method.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context

  • This is part of the data efficiency project
  • This PR should be made to the new_datapack branch.

hunterhector avatar Mar 30 '21 20:03 hunterhector