Christine Smit

On it. Let me see that s3fs has to say. Thank you.

First, I don't think s3fs is doing directories listings on the variable path. It only does directory listings at the higher level with the .zgroup. So the problem does not...

I think that the next thing I'd be curious to try is to use s3fs directly to repeatedly overwrite an object in s3 and see what happens. I think I'll...

@rabernat - We unfortunately do not have a final size for the array. Data keeps coming in until the mission is over. We did try pre-allocating an enormous array that...

@joshmoore - that is an interesting idea, but my bucket did not have versioning. This is good to know, however, if we ever decide to turn versioning on.

@martindurant - Will do. Not sure if I have time to run it today, but I will get back to you.

@martindurant - I appended to an existing array using a new process and it was just as slow as appending had been at the end of the first process. So...

I am stumped. I haven't had a ton of time to spend on this issue over the last 6 weeks, but I've tried to replicate the timing issue by just...

So I don't know how to replicate this timing issue outside of zarr.

Yeah. You actually did mention that before. As I said in https://github.com/zarr-developers/zarr-python/issues/938#issuecomment-1017655172, the logs do show that zarr lists the root directory repeatedly, which makes total sense since my script...