prime-simplereport icon indicating copy to clipboard operation
prime-simplereport copied to clipboard

[SPIKE] Investigate upper bound of CSV export features

Open mindyatwork opened this issue 8 months ago • 2 comments

Background

With the new data retention policy, we need to enable users to export their test and patient data. Back of the napkin product usage shows us that

  • the maximum test results download size is ~88mb for one facility
  • the maximum patients download size is ~17.5mb for one facility

The question is - if we want to enable facilities to export their data, can we support direct download? Or does there need to be an async experience?

Ask

Investigate if we can support direct downloads to help us decide if we'll need to build an async download experience instead. Please check the behavior at these thresholds:

  • 100mb (upper bound with buffer)
  • 88mb (expected upper bound)
  • 20mb (98% of all facilities fall under this threshold for test data. All patient data falls under this)
  • 10mb (97% of all facilities fall under this threshold for test data. 98% patient data falls under this)

There is an existing test data download feature that you can use to test.

Additional context

mindyatwork avatar Apr 21 '25 23:04 mindyatwork

I confirmed that the upper bound numbers are still accurate and don't need to change.

mindyatwork avatar Apr 23 '25 17:04 mindyatwork

Please see the Spike doc for this ticket.

Overall Conclusion: An async process will be required to address files larger than 10 mb

arinkulshi-skylight avatar May 01 '25 15:05 arinkulshi-skylight

Thanks @arinkulshi-skylight ! I reviewed the doc. From my perspective, this ticket is good to close. I'll take this finding back and think about next steps.

mindyatwork avatar May 02 '25 19:05 mindyatwork

Next steps:

  1. Understand if we can increase the timeout
  2. See if we can easily compress the files before download.

arinkulshi-skylight avatar May 14 '25 18:05 arinkulshi-skylight