aws-sdk-pandas

aws-sdk-pandas copied to clipboard

fix(athena): fixed write_iceberg which could fire a ICEBERG_TOO_MANY_OPEN_PARTITIONS Athena error

6

### Feature or Bugfix - Bugfix ### Detail - Athena: fixed write_iceberg which could fire a ICEBERG_TOO_MANY_OPEN_PARTITIONS Athena error ### Relates - None ### Details This is not really a...

erwan-simon

Athena query throws error with message "AttributeError: 'pyarrow._parquet.FileMetaData' object has no attribute 'total_byte_size'"

2

### Describe the bug Using the AWS Wrangler SDK to query a table using Athena results in error message "AttributeError: 'pyarrow._parquet.FileMetaData' object has no attribute 'total_byte_size'". This error happens when...

leo4ever

Unable to use copy_from_files to load into a redshift table w/ an identity column.

1

### Describe the bug In reference to this [issue](https://github.com/aws/aws-sdk-pandas/issues/1110), it appears we are still unable to run copy_from_files when attempting to copy parquet data into a redshift table that has...

nlm4145

Augment dataframes with metadata from the origin file

1

Right now, a third party process saves out json files into an S3 bucket for us. The filename looks like `-.json.gz`, and each file is the output of an endpoint...

Samreay

Double carriage return when using \r\n as line terminator

2

### Describe the bug It seems the old issue [Issue 415](https://github.com/aws/aws-sdk-pandas/issues/415) is back: Passing in \r\n as the line terminator when calling to_csv yields a CSV file with an extra...

davidcava

`athena.to_parquet` fails when `mode=overwrite_partitions` and `partition_cols` contains something like `hour(timestamp_col)`.

3

### Describe the bug When using `s3.to_parquet` to update a parquet file that is partitioned by a time interval or a timestamp "attribute" (such as year, month, hour, etc.), the...

useredsa

Intermittent NETWORK_CONNECTION Error During s3.read_parquet_table Operation

1

### Describe the bug Hi, I'm encountering an intermittent issue when using the s3.read_parquet_table function in my ETL pipeline. The pipeline reads Parquet files from S3 every 5 minutes (modin,...

DimitarSirakov

Athena read_sql_query error when workgroup encryption is turned ON using CSE_KMS

1

### Describe the bug Running any query on Athena fails with the error message "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x86 in position 2: invalid start byte " when workgroup...

leo4ever

fix: support json type in athena2pyarrow

8

### Feature or Bugfix - Feature - Bugfix - Refactoring ### Detail - - ### Relates - By submitting this pull request, I confirm that my contribution is made under...

einavfiterTB

Iceberg commit errors thrown when using overwrite partition somehow fails to clean up temp table

2

### Describe the bug When concurrent processes attempt to update the same iceberg table and one hits an ICEBERG_COMMIT_ERROR, sometimes the temp_table created by delete_from_iceberg_table fails to get cleaned up....

snakingfire