bundleSize is not a configurable WriterOption

[lsankar4033]

Currently, bundle size is defaulted to 256: https://github.com/jwhitbeck/dendrite/blob/master/dendrite/java-src/dendrite/java/Options.java#L343. If the write thread is outrun by the writing thread, it's possible that the batchQueue gets filled. This is a problem if the machine running dendrite doesn't have enough memory.

Another method to solve this problem might be to apply backpressure via the API (i.e. making certain write calls more expensive when a flush is necessary).

[harob]

I believe I also have a case where I would want bundle size to be configurable. I am writing out many rows per file to many files, and when I hit ~800 open FileWriters my process OOMs, even with a 20GB heap size. AFAICT the only collection that is growing is the map of FileWriters.

[jwhitbeck]

The batchQueue is a BlockingQueue that can't hold more than 10 bundles and will start blocking beyond that. There is no particular reason for this value and we could probably lower that to 1 bundle. This would lower the memory usage in the case where the calling thread outruns the dendrite writing thread and shouldn't have any noticeable write throughput impact. I would rather have a good hardcoded value than expose more internals in the public API. Would that address both of your needs?

[jwhitbeck]

Made the batch queue size change here 7ab029242cf292e28d3ea4aa286f501e2e92ee57

[lsankar4033]

Fair enough. This covers my issue. If @harob agrees, we can close this out.