Calling HTTPFileSystem.get on large files looks problematic
Open
Koncopd
opened this issue 10 months ago
•
4 comments
The problem is that when HTTPFileSystem.get is called, it checks if it is a directory here, but to check that this is a directory, it downloads the whole body of the file here, and then again streams the file.