Radim Řehůřek

> so just raise an exception when one fout.write(buff) is called with a buff > 5GB? smart_open's promise is to handle large uploads (and downloads) transparently. So instead of raising...

@JamalRahman @pythric @ivanhigueram could you help with a fix, prepare the PR?

@lambdamusic thanks for reporting. What is the actual error your seeing (versus the expected result)? You mention some Python crash (segfault?), but then show some traceback? A reproducible example would...

Sorry, I'm still not clear. If Python crashed (OS killed the process), how did you get the traceback in the OP? "Crashing" and "responding to CTRL-C" are mutually exclusive.

OK, so it looks like some other Python process crashed (not the main script you were running; possibly some of forked multiprocessing processes). This might be tricky to debug. Any...

I'm thinking it might be RAM-related. `for key, content in s3_iter_bucket(…)` will keep both `key` and `content` in RAM. Did you observe any unusual RAM spikes, when running your script?...

@lambdamusic can you replicate the issue when you replace sending the downloaded data to ES with a simple `sleep(5.0)`? (= to simulate consuming the downloaded object queue more slowly than...

Thanks for looking into this @petedannemann . > Writing via smart-open is slower. Do you know the cause?

Might be related to limitations of `moto` which reportedly fails to propagate keys to forked processes: https://github.com/RaRe-Technologies/smart_open/pull/635#issue-702702227

@mpenkov what was the issue?