Wrong time format in functions.format_dataframe() output?

[gmorain]

Hi, not a frequent user of this package, but today I noticed after pip-upgrading all my requirements and checking the last version of the Jupyter Notebook example that my timestamps were broken after the df = functions.format_dataframe(df) call (all dates end up on 01/01/1970).

I manually ran the pd.to_datetime() call on last_updated_ts adding unit='s' to get a valid answer.

Could be something to enforce in functions.format_dataframe()?

(my config : HA 2023.4.2, HAOS 9.5 on RPi 3, data migrated to MariaDB hosted on same RPi 3)

Best, Gilles

[robmarkcole]

Thanks for the heads up, I will update the notebook example when I get time

[gmorain]

I guess it is more a matter of updating the detective package by enforcing the unit='s' in functions.format_dataframe() than the notebook, unless you want to pass it as a new argument when called from the notebook example?

For now, I am just redefining format_dataframe() at the beginning of my notebook so that pd.to_datetime() includes the unit='s' argument instead of relying on the one imported from functions:

def format_dataframe(df: pd.DataFrame) -> pd.DataFrame:
    """Convert states to numeric where possible and format the last_changed."""
    df["state"] = pd.to_numeric(df["state"], errors="coerce")
    df["last_updated_ts"] = pd.to_datetime(
        df["last_updated_ts"].values, errors="ignore", utc=True, unit='s'
    ).tz_localize(None)
    df = df.dropna()
    return df