Support to add new fields to struct/dict column types

[rdy5644]

Currently the Sturct type of columns cannot be updated and it doesn't accept variable fields like for one row I need only x and for some other row I need y.

[changhiskhan]

I'm not sure we'll want to support implicitly changing the table schema like this. LanceDB isn't designed to be schema-less like this. In production, I think it's likely to create some surprises that will be hard to track down. Is there no way for you to know the schema ahead of time? Or if we added an API for you to explicitly add a sub-column, does that work?

[changhiskhan]

Another possibility - would you be ok storing the metadata as a JSON string? Or do you need to be able to run filters on metadata fields?

[wjones127]

Or do you need to be able to run filters on metadata fields?

To be clear--this isn't all or nothing. You can have both a few fields that are required and can be used in filters, and then a remaining schemaless JSON string field for the rest of your metadata.

[textangel]

Have hit this issue lately, also https://github.com/lancedb/lancedb/issues/850. Nested dict support would be great

[xpluscal]

Whats the general stance from the team here?

We're also dealing with highly varied data where metadata filter querying would be ideal. This currently forces us to play in 2 DB mode aka with pg + lance or similar.

Would love to find a way to do just lance.