parquet-python icon indicating copy to clipboard operation
parquet-python copied to clipboard

TypeError: cannot convert 'int' object to bytes

Open hoday opened this issue 5 years ago • 2 comments

I had an error when trying to open a parquet file:

Traceback (most recent call last): File "/local/workplace/lib/python3.6/site-packages/lambda_handlers/parquet_test.py", line 57, in lambda_handler for row in parquet.reader(fin): File "/local/workplace/lib/python3.6/site-packages/parquet/init.py", line 470, in reader dict_items) File "/local/workplace/lib/python3.6/site-packages/parquet/init.py", line 340, in read_data_page if schema_element.converted_type is not None else read_values File "/local/workplace/lib/python3.6/site-packages/parquet/converted_types.py", line 68, in convert_column return [Decimal(intbig(unscaled)) * scale_factor for unscaled in data] File "/local/workplace/lib/python3.6/site-packages/parquet/converted_types.py", line 68, in return [Decimal(intbig(unscaled)) * scale_factor for unscaled in data] File "/local/workplace/lib/python3.6/site-packages/parquet/converted_types.py", line 42, in intbig return int.from_bytes(data, 'big', signed=True) TypeError: cannot convert 'int' object to bytes

hoday avatar May 10 '20 08:05 hoday

I got a similar error and determined it was choking on a column of type FIXED_LEN_BYTE_ARRAY. I was able to work around this by excluding the column.

Traceback (most recent call last): File ".\parquetexport3.py", line 102, in for row in parquet.DictReader(fileIn, columns=targetColumns): File "C:\Users\m\AppData\Local\Programs\Python\Python38-32\lib\site-packages\parquet_init_.py", line 415, in DictReader for row in reader(file_obj, columns): File "C:\Users\m\AppData\Local\Programs\Python\Python38-32\lib\site-packages\parquet_init_.py", line 464, in reader values = read_data_page(file_obj, schema_helper, page_header, cmd, File "C:\Users\m\AppData\Local\Programs\Python\Python38-32\lib\site-packages\parquet_init_.py", line 334, in read_data_page read_values = convert_column(read_values, schema_element)
File "C:\Users\m\AppData\Local\Programs\Python\Python38-32\lib\site-packages\parquet\converted_types.py", line 66, in convert_column return [Decimal(intbig(unscaled)) * scale_factor for unscaled in data] File "C:\Users\m\AppData\Local\Programs\Python\Python38-32\lib\site-packages\parquet\converted_types.py", line 66, in return [Decimal(intbig(unscaled)) * scale_factor for unscaled in data] File "C:\Users\m\AppData\Local\Programs\Python\Python38-32\lib\site-packages\parquet\converted_types.py", line 40, in intbig return int.from_bytes(data, 'big', signed=True) TypeError: cannot convert 'int' object to bytes

mauryfs avatar Dec 03 '20 21:12 mauryfs

Thanks!

hoday avatar Dec 04 '20 06:12 hoday