Batch input to duckdb?

[jeberly]

Is there an existing method to do batch inserts into duckdb? Or any plans for something like that?

I would like to get logs into duckdb so we can analyze log data in a simple to host/maintain solution.

[mihaitodor]

Hey @jeberly, unfortunately, the DuckDB Go driver links against the DuckDB binaries through CGo, which is a deal-breaker for Benthos, since the project goal is to keep the main build CGo-free. However, you can create a custom plugin where you import Benthos as a library along with your plugin to create a custom binary.

Since their library is compatible with database/sql, ideally you should be able to inject a new driver for the various sql_* components in Benthos, but that's not supported right now, since the driver list can't be customised and there's no way to specify the placeholder format or if a transaction statement should be used or not. Right now, the simplest approach is to copy whatever component(s) you need from here and adjust them as needed to create your own plugin.

[jeberly]

Thanks! Good idea with copying an existing plugin.

[artemklevtsov]

Also you can use command process with the duckdb cli.

[rasnes]

Waiting for someone to implement a DuckDB driver with purego so we can add it to Redpanda Connect ⌛ 😊

[Flamefork]

Waiting for someone to implement a DuckDB driver with purego so we can add it to Redpanda Connect ⌛ 😊

https://github.com/fpt/go-pduckdb