astro-sdk
astro-sdk copied to clipboard
issue with deserialization of timestamp datatype
Describe the bug The below task fails with current version of code in main.Refer to https://astronomer.slack.com/archives/C03868KGF2Q/p1669809348854679
@aql.dataframe(columns_names_capitalization="original")
def load_and_group_covid_data():
"""
Loads data from a COVID data REST API and then groups values based on the months.
:return: A list of dataframes for each month of the pandemic
"""
covid_df = _load_covid_data()
covid_df["Date_YMD"] = covid_df["Date_YMD"].apply(lambda d: datetime.strptime(d, "%Y-%m-%d"))
return [x for _, x in covid_df.groupby(covid_df.Date_YMD.dt.month)]
[2022-12-01, 07:20:39 UTC] {taskinstance.py:1772} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/astro/sql/operators/dataframe.py", line 170, in execute
function_output = self.python_callable(*self.op_args, **self.op_kwargs)
File "/usr/local/airflow/dags/example_dataframe_api.py", line 51, in find_worst_covid_month
covid_month = covid_month_data.Date_YMD.iloc[0].__format__("%Y-%m")
ValueError: Invalid format specifier
Version
- Astro: [e.g. 0.6.0]
- OS: [eg. Debian]
To Reproduce Steps to reproduce the behavior:
- Write the DAG '...'
- Create connection '....'
- Run using '....'
- See error
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Additional context Add any other context about the problem here.