aws-sdk-pandas
aws-sdk-pandas copied to clipboard
aws
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Par...
### Describe the bug When I run a SPARQL SELECT query against Neptune using AWS SDK for Pandas, if one of the values is null, I get an error: File...
### Describe the bug Hi, I've used wr.s3.read_parquet to load my parquet file from s3. I have a very huge file in s3. A single file has 1 Million rows,...
### Describe the bug ``` wr.athena.to_iceberg( df=df, database='test_database', table='my_table2', table_location='s3://bucket-testing1/my_table2/', temp_path=f's3://bucket-testing1/temp_path/', keep_files=True ) ``` For parallel writing, If keep_files=True then it is resulting the **duplicates** and I tried appending the...
Leveraging the Arrow ADBC Driver has a number of advantages: - A single, universal driver would replace all existing, proprietary drivers (mssql, postgres...) - Better performance due to not processing...
**Describe the solution you'd like** Integration with this: https://aws.amazon.com/about-aws/whats-new/2023/10/amazon-redshift-integration-secrets-manager/
### Describe the bug if a column is null there should be a fallback data type (varchar) I'm using: ``` wr.redshift.copy_from_files( path=path, con=con, table=file_name.replace(".parquet", ""), schema="staging", parquet_infer_sampling = 1, varchar_lengths_default...
Hi, I'm working with an Aurora PostgreSQL database - and I had to remove the "pg8000_native.literal" from this line to properly work with chunks: https://github.com/aws/aws-sdk-pandas/blob/602e8a78d6560ea16be1f342061c0e1493327df4/awswrangler/postgresql.py#L106 If I understand the documentation...
I'm using awswrangler==3.4.2 `wr.athena.to_iceberg` to ingest ~100 text files from S3 into an Iceberg table using parallel lambdas ``` wr.athena.to_iceberg( df=df, database=database, table=table_name, partition_cols=partition_cols, table_location=s3_location, temp_path=f"s3://{bucket}/{database}/temp/{table_name}", keep_files=False, ) ``` but...
### Describe the bug Calling `we.s3.to_deltalake()` throws the following error: ``` self._table = RawDeltaTable( ^^^^^^^^^^^^^^ TypeError: argument 'storage_options': 'NoneType' object cannot be converted to 'PyString' ``` ### How to Reproduce...
Most of existing [DynamoDB tests](https://github.com/aws/aws-sdk-pandas/blob/main/tests/unit/test_dynamodb.py) test for data frame shape, but not the content. This has caused us to miss issues like #2614 #2605. Filing this to work on DynamoDB...
