gqlalchemy icon indicating copy to clipboard operation
gqlalchemy copied to clipboard

[BUG] CSVLocalFileSystemImporter can't load datetime from csv

Open SuperBo opened this issue 1 year ago • 17 comments

Memgraph version Which version did you use? Memgraph v2.13.0

Environment Some information about the environment you are using Memgraph on: operating system, how do you connect, with or without docker, which driver etc.

Docker

Describe the bug

Cant load datetime string from csv to memgraph.

To Reproduce Steps to reproduce the behavior:

Use following scripts

importer = loaders.CSVLocalFileSystemImporter(
    data_configuration=yaml.safe_load(DATA_CONFIG),
    path = "tmp/airflow/data",
    memgraph=memgraph
)
importer.translate(drop_database_on_start=True)

Expected behavior

No error

Logs If applicable, add logs of Memgraph, CLI output or screenshots to help explain your problem.

Loading data from table tmp/airflow/data/a.csv...
Traceback (most recent call last):
  File "/Users/superbo/Workspace/airflow-pipelines/test_memgraph_load.py", line 36, in <module>
    load_data()
  File "/Users/superbo/Workspace/airflow-pipelines/test_memgraph_load.py", line 32, in load_data
    importer.translate(drop_database_on_start=True)
  File "/Users/superbo/Workspace/airflow-pipelines/.venv/lib/python3.11/site-packages/gqlalchemy/transformations/importing/loaders.py", line 472, in translate
    self._load_nodes()
  File "/Users/superbo/Workspace/airflow-pipelines/.venv/lib/python3.11/site-packages/gqlalchemy/transformations/importing/loaders.py", line 480, in _load_nodes
    self._save_row_as_node(label=collection_name, row=row)
  File "/Users/superbo/Workspace/airflow-pipelines/.venv/lib/python3.11/site-packages/gqlalchemy/transformations/importing/loaders.py", line 591, in _save_row_as_node
    QueryBuilder(connection=self._memgraph)
  File "/Users/superbo/Workspace/airflow-pipelines/.venv/lib/python3.11/site-packages/gqlalchemy/query_builders/declarative_base.py", line 748, in node
    properties_str = to_cypher_properties(kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/superbo/Workspace/airflow-pipelines/.venv/lib/python3.11/site-packages/gqlalchemy/utilities.py", line 157, in to_cypher_properties
    value_str = to_cypher_value(value, config)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/superbo/Workspace/airflow-pipelines/.venv/lib/python3.11/site-packages/gqlalchemy/utilities.py", line 109, in to_cypher_value
    return f"{datetimeKwMapping[value_type]}('{_format_timedelta(value) if isinstance(value, timedelta) else value.isoformat()}')"
              ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: <class 'pandas._libs.tslibs.timestamps.Timestamp'>

Additional context Add any other context about the problem here.

SuperBo avatar Jan 21 '24 07:01 SuperBo