Ibis-Athena: Dataset error while using save_dataset.save(data)

[uday-dasari]

We are using Athena as backend and Ibis Version: 3.3.0 to save the table dataset using the below code -

save_dataset = TableDataset(
    table_name="table_name",
    connection={
        "backend": "athena",
        "s3_staging_dir": "s3://bucket-name/...",
        "schema_name": "schema_name",
    },
    save_args={"materialized": "table", "overwrite": None} 

save_dataset.save(data)

This is resulting in a dataset error.

DatasetError: Failed while saving data to dataset TableDataset(backend=athena, load_args={}, materialized=table, save_args={}, table_name='table_name'). TABLE_NOT_FOUND: line 17:6: Table does not exist. You may need to manually clean the data at s3 location.

But after creating a table before saving into it, this below error pops up -

DatasetError: Failed while saving data to dataset TableDataset(backend=athena, load_args={}, materialized=table, save_args={}, table_name='table_name'). TABLE_ALREADY_EXISTS: line 1:1: Destination table already exists.

Tried passing different save_args as well, but that did not resolve the issue - to overwrite as True or materialized as insert etc. The goal here is to load the dataset from one DB and save into another. It was successfully loaded but facing trouble with saving it.

[cpcloud]

Thanks for the issue!

What is TableDataset? Are you using another library on top of Ibis here?

Also, Ibis 3.3.0 is not a version of Ibis that exists.

You probably installed the other Ibis which is not our Ibis.

You need to install our Ibis with pip install ibis-framework.