querybook icon indicating copy to clipboard operation
querybook copied to clipboard

fix: filestore performance fix

Open najma-kunju opened this issue 1 year ago • 5 comments

najma-kunju avatar Apr 07 '23 08:04 najma-kunju

About my PR:

Previously, the write method in FileUploader class was opening the file in append mode and writing data into it for each record of the result set of the query. Which perhaps was talking a lot of time if the result set is huge and is also not a good practice to increase the file operation.

So to increase the performance, we did the below:

  • Initialised self._chunks as an empty list in start method of FileUploader class.
  • Kept appending each record of the query result to the above list in write method of FileUploader class.
  • At the end of the process, Converted the list elements to string using join in end method of FileUploader class.
  • Opened the file in write mode and wrote the above string value to file in end method of FileUploader class.

Thanks for your time.

ghost avatar Apr 11 '23 06:04 ghost

@najma-kunju there are still lint and python test failures

jczhong84 avatar May 09 '23 23:05 jczhong84

@jczhong84, have fixed the current 2 lint errors

ghost avatar May 10 '23 05:05 ghost

@najma-kunju it's still failing

jczhong84 avatar May 13 '23 00:05 jczhong84

Hi @jczhong84, Now the checks have passed. Thanks for the help throughout PRs 1221 and 1237.

ghost avatar May 14 '23 16:05 ghost