ClickHouse.Client icon indicating copy to clipboard operation
ClickHouse.Client copied to clipboard

[FEAT] add ClickHouseBulkCopyColumnMapping

Open rferraton opened this issue 8 months ago • 2 comments

Problem statement

Currently ColumnNames in the ClikHouseBulkCopy builder cannot be use to map source and target columns like other bulkcopy client do (mssql, mysql, oracle, hana).

This feature is very convienient to manage not position aligned source and target (missing column in the source or in the target).

Suggested solution

Here an exemple for mssql :

private static void MapColumnsByName(string targetschema, string targettable, SqlBulkCopy sbc, List<string> targetColumns, DbDataReader sdr)
{
	SqlBulkCopyColumnMapping BulkCopyColumnMapping;
	var columnSchema = sdr.GetColumnSchema();
	int targetColumnsCount = targetColumns.Count;
	int mappedColumns = 0;
	sbc.ColumnMappings.Clear();

	// map the source columns that exists in the targetColumns list
	foreach (var column in columnSchema)
	{
		if (targetColumns.Contains(column.ColumnName.ToLowerInvariant()))
		{
			BulkCopyColumnMapping = new SqlBulkCopyColumnMapping(column.ColumnName, column.ColumnName);
			sbc.ColumnMappings.Add(BulkCopyColumnMapping);
			mappedColumns++;
		}
		else
		{
			Program.logger.Warning($"Column {column.ColumnName} not found in target table {targetschema}.{targettable}. The column will be ignored");
		}
	}
}

Considered alternatives

No response

Additional information

No response

rferraton avatar Mar 27 '25 00:03 rferraton

There is an init-only property ColumnNames which can be used to pass the eponymous parameter on object init:

using var bulkCopy = new ClickHouseBulkCopy(connection)
{
    DestinationTableName = targetTable,
    ColumnNames = ["value"]
};
await bulkCopy.InitAsync();

Can you try this feature to see if it answers your purposes?

DarkWanderer avatar Mar 27 '25 19:03 DarkWanderer

I did, but the issue with ColumnNames is :

  • it is an init only attribut (not so painfull but...)
  • when you don't have a plain mapping (all targeted columns mapped). Then the bulk loader fails. Which is not the case with other bulkcopy that have a xxxBulkCopyColumnMapping. They allons partial mapping which is very convenant.

rferraton avatar Mar 27 '25 20:03 rferraton