s3fs icon indicating copy to clipboard operation
s3fs copied to clipboard

Is writting thread safe?

Open mporracindie opened this issue 6 years ago • 4 comments

Hi, I would like to know if writing from different threads to the same open connection is safe? What happens if I keep calling write while it needs to flush?

mporracindie avatar Oct 10 '19 14:10 mporracindie

What exactly do you mean by open connection?

martindurant avatar Oct 10 '19 14:10 martindurant

So if I do something like:

with s3.open('mybucket/new-file', 'wb') as f:

And then call f.write() from multiple threads at the same time at a high rate would it be able to handle it?

mporracindie avatar Oct 11 '19 12:10 mporracindie

I don't believe its ever been tested. Each upload of a block will happen in only one thread, but it is conceivable that the various attributes of the file instance have some race conditions.

martindurant avatar Oct 11 '19 13:10 martindurant

It's not, files have been getting "chopped off" midway while I'm attempting to write many files at once.

tumut avatar Jan 23 '20 23:01 tumut