loicalleyne

On a high level I'm thinking the first step is that output writer would need a new configuration option for schema where the writer will accept a schema passed by...

On a high level I'm thinking the first step is that output writer would need a new configuration option for schema where the writer will accept a schema passed by...

I wrote a proof of concept that can take a map and output what I think is compliant YAML schema for the parquet encoder and also the beginning of a...

As far as I know Parquet only supports a union of ["primitive_type", "null"] or ["null", "primitive_type"] https://github.com/apache/parquet-format/pull/44 In that scenario you provided my code would infer the following `- {...

As an example of what gets inferred from an arbitrary map, here are the input and output examples: Input: ``` var exampleMap = map[string]interface{}{ "request": map[string]interface{}{ "datetime": "2021-07-27 02:59:59", "ip":...

Just saw https://github.com/benthosdev/benthos/commit/07ed81b150778a362e25e52428c59a05ca21369b, I'd like to work on enabling parquet output without having to specify a schema in the config file - I was wondering: If input messages are structured,...

@sdressler I've written a package that allows you to generate an Arrow schema from an arbitrary map[string]interface{}, maybe it's helpful. github.com/loicalleyne/arrow_schemagen

Any chance of this being merged? As far as I can tell the tool in unusable without this PR.

How should a package be emitting logs in a way that makes it ok to use in Benthos? Someone submitted an issue last year to add an option to disable...

☹️ Looked at the simpleforce code again, Salesforce has yet another custom auth flow which doesn't seem to fit with the current http_client auth, templates alone might not make this...