querybook
querybook copied to clipboard
fix: filestore performance fix
About my PR:
Previously, the write method in FileUploader class was opening the file in append mode and writing data into it for each record of the result set of the query. Which perhaps was talking a lot of time if the result set is huge and is also not a good practice to increase the file operation.
So to increase the performance, we did the below:
- Initialised self._chunks as an empty list in start method of FileUploader class.
- Kept appending each record of the query result to the above list in write method of FileUploader class.
- At the end of the process, Converted the list elements to string using join in end method of FileUploader class.
- Opened the file in write mode and wrote the above string value to file in end method of FileUploader class.
Thanks for your time.
@najma-kunju there are still lint and python test failures
@jczhong84, have fixed the current 2 lint errors
@najma-kunju it's still failing
Hi @jczhong84, Now the checks have passed. Thanks for the help throughout PRs 1221 and 1237.