Reduce CPU time spent in ingest path

[brancz]

Currently, a major amount of CPU time is spent in the ingestion path of Parca/arcticdb, and garbage collection.

https://pprof.me/999a52c/

There are a couple of things we can and should try:

• Set parquet.ColumnBufferCapacity of a parquet writer to the maximum amount that we may write to the buffer during ingestion. I suspect we are wildly over-allocating when splitting data.
• Tune parquet.PageBufferSize. Same reason as above. We can probably have arcticDB auto-tune itself here based on previous write sizes.
• Reuse parquet.Writer. When done with a writer we can call .Reset on it and reuse it to write the next buffer. If the above two don't help, this is probably the last resort. This one is also more complicated as we can't reuse writers for different schemas, so we would need to keep a record of the schemas and create a sync.Pool per schema that we see. I suspect this will eventually have to be done to get GC time under control (we'll need to measure whether the saving in CPU time of GC is worth it, as we'd be spending time on resetting the writers).

[cyriltovena]

FrostDB uses a lot of OpenFile for reading row in memory. I think https://github.com/segmentio/parquet-go/pull/255 will help.

I went for the pool per schema/dynamic columns as I think over allocating is fine as long as you can reuse.

So far this what I have.

❯ benchstat before.txt after.txt
name                                       old time/op    new time/op    delta
_Table_Insert_10Rows_10Iters_10Writers-16     460ms ±14%     586ms ±12%  +27.44%  (p=0.008 n=5+5)

name                                       old alloc/op   new alloc/op   delta
_Table_Insert_10Rows_10Iters_10Writers-16     794MB ±30%     293MB ±59%  -63.15%  (p=0.008 n=5+5)

name                                       old allocs/op  new allocs/op  delta
_Table_Insert_10Rows_10Iters_10Writers-16     1.49M ±52%     0.76M ±24%  -49.11%  (p=0.016 n=5+5)

I'm still playing around though.

[brancz]

Nice! We should also be able to re-use writers and use a sync.Pool on a per schema basis for those as well. I think that will optimize away another fair bit.

[cyriltovena]

That's what I actually do.

[thorfour]

Closing as I believe this has been addressed with many changes, notably the new index.