astro-sdk
astro-sdk copied to clipboard
issue with deserialization of timestamp datatype
Describe the bug The below task fails with current version of code in main.Refer to https://astronomer.slack.com/archives/C03868KGF2Q/p1669809348854679
@aql.dataframe(columns_names_capitalization="original")
def load_and_group_covid_data():
"""
Loads data from a COVID data REST API and then groups values based on the months.
:return: A list of dataframes for each month of the pandemic
"""
covid_df = _load_covid_data()
covid_df["Date_YMD"] = covid_df["Date_YMD"].apply(lambda d: datetime.strptime(d, "%Y-%m-%d"))
return [x for _, x in covid_df.groupby(covid_df.Date_YMD.dt.month)]
[2022-12-01, 07:20:39 UTC] {taskinstance.py:1772} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/astro/sql/operators/dataframe.py", line 170, in execute
function_output = self.python_callable(*self.op_args, **self.op_kwargs)
File "/usr/local/airflow/dags/example_dataframe_api.py", line 51, in find_worst_covid_month
covid_month = covid_month_data.Date_YMD.iloc[0].__format__("%Y-%m")
ValueError: Invalid format specifier
Version
- Astro: [e.g. 0.6.0]
- OS: [eg. Debian]
To Reproduce Steps to reproduce the behavior:
- Write the DAG '...'
- Create connection '....'
- Run using '....'
- See error
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Additional context Add any other context about the problem here.
Need to test after #1590 is implemented
This needs to be tested