Split large mlperf_log_trace JSON files into 25MB chunks

[Mostelk]

Need a post processing script in the App, that checks the size of the mlperf_log_trace, and splits it into smaller sizes for upload results to github, for example https://github.com/mlcommons/mobile_results_v3.1/tree/main/closed/Qualcomm_Innovation_Center/results/SD7PG2/mobilenetEdgeTPU/Offline/log_accuracy splits the json files to chunks of 25Mb https://github.com/mlcommons/mobile_results_v3.1/tree/main/closed/Samsung/results/object_detection-performance/mlperf_log_trace_split splits them into chunks < 50 MB

need to apply to the v4 submission branch https://github.com/mlcommons/mobile_app_open/tree/submission-v4.0

[anhappdev]

Should this feature be in the app?

On a computer, it's quite easy to zip and split the big file, though. For example, on macOS I can zip and split the file with this command:

zip -s 25m new.zip big_file.json

But doing that in the mobile app is not trivial.

[Mostelk]

@anhappdev it is nice to have, not a must, I know split can do it. There is some reference here as well https://stackoverflow.com/questions/37761543/how-i-can-split-file-in-android-case-the-file-is-large-to-upload-itsound-file

[freedomtan]

Should this feature be in the app?

On a computer, it's quite easy to zip and split the big file, though. For example, on macOS I can zip and split the file with this command:

zip -s 25m new.zip big_file.json

But doing that in the mobile app is not trivial.

I don't remember the status of zip, but I think the old unix command split is available on Android.

[anhappdev]

I don't remember the status of zip, but I think the old unix command split is available on Android.

But the solution must be cross-platform and work on Android, iOS and Windows.

[freedomtan]

generally, use some libz (zip, gzip, or compression form) or any libraries we can use to do what the old unix split does. @freedomtan to dig more

or we can just add it to loadgen. Do the splitting during log generation time?