fugue
fugue copied to clipboard
`SAVE AND USE' should work for CSV.
Originally posted by @keiranmraine in https://github.com/fugue-project/fugue/issues/331#issuecomment-1160126031
File c:\Users\XXX\.venv\lib\site-packages\fugue\workflow\workflow.py:1518, in FugueWorkflow.run(self, *args, **kwargs)
1516 if ctb is None: # pragma: no cover
1517 raise
-> 1518 raise ex.with_traceback(ctb)
1519 self._computed = True
1520 return DataFrames(
1521 {
...
231 pdf = _safe_load_csv(
232 p.uri, **{"index_col": False, "header": None, "names": columns, **kw}
233 )
ValueError: columns must be set if without header
This is because SAVE AND USE
on csv files does not take the parameters sep
and header
.
Solution:
- It is recommended not to use CSV when you want to use
SAVE AND USE
because CSV has so many special parameters, they are not consistent cross frameworks, and they are not consistent between read and write, it is very hard to unify them, sometimes impossible. - Just try to use the parameters when loading, if it doesn't work, we should raise an exception, and also warn people that it's not a good idea to save and use csv.
We will implement 2, but please keep 1 in mind.