jitsu
jitsu copied to clipboard
jitsucom
When syncing a source to a destination not all columns get copied
Version: Jitsu Next @ latest
Source postgres table:
Destination postgres table:
The source postgres table contains only one row whose scope and workspaceId fields are null (these fields are missing from the destination table). So it seems to be only happening when a column in all rows are null.
I assume this is an issue with Airbyte but I still want to check in.
Going into the json_schema fetched after a call to /discover I get the following for this source table:
{...
"json_schema": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"scope": {
"type": "string"
},
"userId": {
"type": "string"
},
"createdAt": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_without_timezone"
},
"updatedAt": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_without_timezone"
},
"preferences": {
"type": "string"
},
"workspaceId": {
"type": "string"
}
}
}
}
I'm not sure if it's related that none of the fields have got any property to indicate nullable.
It's not a bug, it's how Jitsu is designed at the moment. The connector sends data to Bulker as an array (stream) of JSON object, and Bulker omits all null values. Also, it create columns on demand, so as a result the columns are not being created. But once the column will have a non-null value at least in one record coming in, it will be created.
That being said, I agree that the current design is not ideal. We will make a change that will remove null value deletion if the data is coming from Jitsu Connectors