fugue icon indicating copy to clipboard operation
fugue copied to clipboard

`SAVE AND USE' should work for CSV.

Open goodwanghan opened this issue 2 years ago • 0 comments

Originally posted by @keiranmraine in https://github.com/fugue-project/fugue/issues/331#issuecomment-1160126031

File c:\Users\XXX\.venv\lib\site-packages\fugue\workflow\workflow.py:1518, in FugueWorkflow.run(self, *args, **kwargs)
   1516         if ctb is None:  # pragma: no cover
   1517             raise
-> 1518         raise ex.with_traceback(ctb)
   1519     self._computed = True
   1520 return DataFrames(
   1521     {
...
    231         pdf = _safe_load_csv(
    232             p.uri, **{"index_col": False, "header": None, "names": columns, **kw}
    233         )

ValueError: columns must be set if without header

This is because SAVE AND USE on csv files does not take the parameters sep and header.

Solution:

  1. It is recommended not to use CSV when you want to use SAVE AND USE because CSV has so many special parameters, they are not consistent cross frameworks, and they are not consistent between read and write, it is very hard to unify them, sometimes impossible.
  2. Just try to use the parameters when loading, if it doesn't work, we should raise an exception, and also warn people that it's not a good idea to save and use csv.

We will implement 2, but please keep 1 in mind.

goodwanghan avatar Jun 20 '22 17:06 goodwanghan