google-api-python-client icon indicating copy to clipboard operation
google-api-python-client copied to clipboard

Published 20 hours ago • verified publisher icon googleapis

Memory leak - issue while downloading large files

Open hexadivine opened this issue 3 years ago • 2 comments

Issue : next_chunk method stores downloaded content to 'fd' file object. This downloaded content is stored in ram. As the file size increases, it consumes more RAM to store downloaded data.

Reference - https://github.com/googleapis/google-api-python-client/blob/5399bd3bf34ef31302a77ff484ef0067a18709a6/googleapiclient/http.py#L755

Related issues: #512 #198

hexadivine avatar Feb 28 '22 04:02 hexadivine

Hi @parthea,

Please check the pull request. I have added "MediaGenBaseDownload" class which will yield fetched chunks in order to keep RAM consumption linear(equal to chunksize), with the backward compatibility.

hexadivine avatar Mar 28 '22 18:03 hexadivine

One of my commit did not had verified tag, and also causing it to fail CLA check. I will have to create new PR.

hexadivine avatar Jul 29 '24 11:07 hexadivine