jitsu icon indicating copy to clipboard operation
jitsu copied to clipboard

Support Array types in Destinations

Open xtreding opened this issue 4 years ago • 1 comments

Problem

At present Jitsu saves fields of array type as strings into destinations. Some destinations have JSON column types (e.g. Postgres has jsonb). So Jitsu can support typing.ARRAY format and map it to jsonb in Postgres case and other JSON types in other destinations. It helps users to use such columns in SQL queries in a more convenient way.

Solution

  • Add typing.ARRAY to Jitsu types tree (typing/datatype.go and typing/converter.go).
  • Add parsing rules in schema/flattener.go and typing/datatype.go TypeFromValue().
  • Enrich all destination adapters with typing.ARRAY -> destination type mappings (e.g. adapters/postgres.go SchemaToPostgres)
  • Add tests

xtreding avatar Oct 19 '21 16:10 xtreding

Clickhouse also has array type columns and can be very convenient to store part of fields/subfields in an array column type.

amadrizwan avatar Oct 20 '21 14:10 amadrizwan