Max number of objects supported?

[dblacknc]

I am using Warp in K8s mode with 22 clients. A mix test with total ~250M 4KB objects, or about 11.3M per client, appeared to normally finish the prepare phase in all client, then it just exited - nothing further in the log, and didn't move to the benchmark phase. I can try looking at the code, and thought to ask - what's the expected max for the total object count and/or the per-client count?

[harshavardhana]

@dblacknc what is the server component here?

[dblacknc]

The object storage (S3) server is a third-party vendor (not MinIO) that I don't feel free to specify in this forum, sorry. With this same vendor a wide variety of other object sizes and tests have worked well. I'm planning to drop back to just one Warp client (from 22) and testing it first with 1M objects, increasing by 1M until (I expect) it'll break before I get past 11M. That's assuming the Warp master process isn't itself failing because the previous total of 250M objects was too high.

[klauspost]

250M objects may run you out of memory. You will need a good deal of memory to deal with that many.

[harshavardhana]

warp is benchmarking tool not a data generation tool - if you wish to only generate data it must be either custom written or use some other tool to do so.

warp has some amount of memory state that is not suitable to create 250M objects, it would have OOMed on the client side.