clickhouse-odbc icon indicating copy to clipboard operation
clickhouse-odbc copied to clipboard

Int64 columns are loaded as binary columns in PowerBI

Open nudles opened this issue 1 year ago • 2 comments
trafficstars

ClickHouse version: 24.7.1.2092 ODBC version: 1.2.1.20220905 PowerBI version: 2024 July Table DDL

CREATE TABLE `Store`
(
	location Int64,
	name String
)
ENGINE = MergeTree()
ORDER BY (location);

After creating the table, I tried to load it from PowerBI via ODBC driver. However, only the 'name' column is visible. The LocationID column is missing. img_v3_02cs_45b7bade-8286-4bf9-b169-5281c624achu

img_v3_02cs_bfff7591-5277-4fd1-abe3-51b27305a2hu

In the transform view, it shows that the 'location' column's type is binary. I changed it to number manually. img_v3_02cs_7d548409-266f-46e7-b082-fe0cbebde0hu

Then I can see the location column

img_v3_02cs_a4d7363a-e3f2-4171-be16-861c9cf8e0hu

The question is how can we change ODBC driver to fix the data type mapping? Thank you!

nudles avatar Jul 17 '24 03:07 nudles

I also tried columns of other data types, including Int32, UInt32, UInt64. Only Int32 column can be displayed. All other columns are recongized as Binary column by PowerBI.

nudles avatar Jul 17 '24 10:07 nudles

Hi @traceon , could you please give some hints on how to debug/fix this issue? Thank you!

nudles avatar Jul 18 '24 03:07 nudles

The problem seems to come from the fact that SQLGetTypeInfo returns the same type name for different types—signed and unsigned. That behavior is defined here. For example, both Int64 and UInt64 return the type name BIGINT.

At least in my testing, when I remove this type name overlap—either by completely removing unsigned types or by assigning unique names to each type—the problems seem to go away, and all columns appear as whole numbers in Power BI.

Using unique names for each type is indeed the correct approach. In fact, the ODBC documentation requires that the type names returned by SQLGetTypeInfo match the type names used in CREATE TABLE or ALTER TABLE clauses when defining column types: ODBC Documentation for SQLGetTypeInfo.

However, this change might break some backward compatibility, so further investigation is needed.

Additionally, the SQLColumns function should return a dataset with columns of a specific type, as described
here. This makes sense because Power BI should not check the types in the resulting dataset; instead, it should start
reading the values as described in the standard.

slabko avatar Mar 07 '25 13:03 slabko

@nudles, am I correct in understanding that you can change the type of the location column to Whole Number in the Power BI Query Editor? While it's annoying, there is a workaround for the problem for the time being.

Image

slabko avatar Mar 07 '25 13:03 slabko

@slabko do we need to release #478 to close the issue ?

mshustov avatar May 19 '25 07:05 mshustov

@mshustov Yes, we need to release the fix.

slabko avatar May 21 '25 06:05 slabko

Fixed in 1.4.1.20250523

Note: BI caches data definitions. This means that even if you update the driver and try to load a table previously accessed with the older driver, Power BI may still use the data definitions it received from the older version. To prevent issues such as numeric columns being recognized as binary data, you need to reload the table definitions by clicking the small reload button in the upper-right corner.

Image

slabko avatar May 23 '25 16:05 slabko