read_csv_glob fails reading from azure storage account

[c3-cjazra]

System information

• python 3.9
• modin 0.12
• ray 1.92

pandas and modin read_csv works using the adlfs module (https://github.com/fsspec/adlfs):

first setup os environ

os.environ["AZURE_STORAGE_ACCOUNT_NAME"] = "someaccount"
os.environ["AZURE_STORAGE_ACCOUNT_KEY"] = "somekey"

then do

modin_pd.read_csv('abfs://[email protected]/filename.csv')

works!

read_csv_glob throws the following error:

modin_experimental_pd.read_csv_glob('abfs://[email protected]/filename.csv')
File /python3.9/site-packages/modin/experimental/pandas/io.py:183, in _make_parser_func.<locals>.parser_func(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, encoding_errors, dialect, error_bad_lines, warn_bad_lines, on_bad_lines, skipfooter, doublequote, delim_whitespace, low_memory, memory_map, float_precision, storage_options)
    180     f_locals["sep"] = "\t"
    182 kwargs = {k: v for k, v in f_locals.items() if k in _pd_read_csv_signature}
--> 183 return _read(**kwargs)

File python3.9/site-packages/modin/experimental/pandas/io.py:208, in _read(**kwargs)
    205 Engine.subscribe(_update_engine)
    207 try:
--> 208     pd_obj = FactoryDispatcher.read_csv_glob(**kwargs)
    209 except AttributeError:
    210     raise AttributeError("read_csv_glob() is only implemented for pandas on Ray.")

File /python3.9/site-packages/modin/core/execution/dispatching/factories/dispatcher.py:185, in FactoryDispatcher.read_csv_glob(cls, **kwargs)
    182 @classmethod
    183 @_inherit_docstrings(factories.ExperimentalPandasOnRayFactory._read_csv_glob)
    184 def read_csv_glob(cls, **kwargs):
--> 185     return cls.__factory._read_csv_glob(**kwargs)

File /python3.9/site-packages/modin/core/execution/dispatching/factories/factories.py:513, in ExperimentalPandasOnRayFactory._read_csv_glob(cls, **kwargs)
    506 @classmethod
    507 @doc(
    508     _doc_io_method_raw_template,
   (...)
    511 )
    512 def _read_csv_glob(cls, **kwargs):
--> 513     return cls.io_cls.read_csv_glob(**kwargs)

File python3.9/site-packages/modin/core/io/text/csv_glob_dispatcher.py:62, in CSVGlobDispatcher._read(cls, filepath_or_buffer, **kwargs)
     60 if isinstance(filepath_or_buffer, str):
     61     if not cls.file_exists(filepath_or_buffer):
---> 62         return cls.single_worker_read(filepath_or_buffer, **kwargs)
     63     filepath_or_buffer = cls.get_path(filepath_or_buffer)
     64 elif not cls.pathlib_or_pypath(filepath_or_buffer):

File python3.9/site-packages/modin/core/storage_formats/pandas/parsers.py:269, in PandasParser.single_worker_read(cls, fname, **kwargs)
    267 ErrorMessage.default_to_pandas("Parameters provided")
    268 # Use default args for everything
--> 269 pandas_frame = cls.parse(fname, **kwargs)
    270 if isinstance(pandas_frame, pandas.io.parsers.TextFileReader):
    271     pd_read = pandas_frame.read

File python3.9/site-packages/modin/core/storage_formats/pandas/parsers.py:312, in PandasCSVGlobParser.parse(chunks, **kwargs)
    309 index_col = kwargs.get("index_col", None)
    311 pandas_dfs = []
--> 312 for fname, start, end in chunks:
    313     if start is not None and end is not None:
    314         # pop "compression" from kwargs because bio is uncompressed
    315         with OpenFile(fname, "rb", kwargs.pop("compression", "infer")) as bio:

ValueError: not enough values to unpack (expected 3, got 1)

following calls method also fails

modin_experimental_pd.read_csv_glob('abfs://[email protected]/*')
modin_experimental_pd.read_csv_glob('abfs://[email protected]/*.csv')

[anmyachev]

Hi @c3-cjazra. Thanks for posting! The bug is fixed on 0.14.0. It is possible for you update Modin version?

[c3-cjazra]

Hi @anmyachev, thanks for the quick followup! I just tried on 0.14.0, I see the error is solved, however the problem now is that it is only reading from one file instead of all files.

so there are 2 csv file at the container path but modin_experimental_pd.read_csv_glob('abfs://[email protected]/*.csv') only return the content of 1 file

[anmyachev]

Blocked by https://github.com/modin-project/modin/issues/3572

[c3-cjazra]

Thank you for the update @anmyachev, any sense of the scope of the fix?

[anmyachev]

@c3-cjazra the main problem here is that we haven't fully switched to using fsspec, so reading multiple files can only be for s3, because we are explicitly using s3fs. This definitely needs to be fixed, the main difficulty is to correctly write the handling of cases when reading is done anonymously or under certain permissions (anon=True or False).

@devin-petersohn what is the planned date for the next release?

[anmyachev]

Some problems when trying to enable abfs-like URLs: https://github.com/fsspec/adlfs/issues/319.

[Anando304]

Hi @anmyachev ,

This new update of 0.14.0 is only for Python 3.8+? Is there anyway around this issue if we are using an older version of Python (3.7). My company takes a while to update to newer releases so kind of stuck on an older version. :/

[anmyachev]

Hi @Anando304,

You're right. 0.14.0 Modin release works only for Python 3.8+. Moreover, 0.13.0 release already works only for Python 3.8+.

You have several options to try.

First option is most easy, just rewrite read_csv_glob to several read_csv calls + concat. There is no parallelization, but your error can be avoided until you update to new Modin version.

dfs = [None]* len(filenames)
for i, filename in enumerate(filenames):
    dfs[i] = pd.read_csv(filename)
df = pd.concat(dfs, axis=0, copy=False)

Second option: you need to move your data to s3 bucket, since read_csv_glob should work with that type of buckets. However, note that there were bugs in this function that have been fixed in newer versions of Modin. If this option is suitable for you, then first try to transfer a small amount of data to check if it works correctly with your set of parameters and not spend a lot of money.

[vnlitvinov]

Hi @Anando304! We've recently introduced a compatibility mode for Python <3.8 which would pin pandas to 1.1.5 (should be released as modin==0.16 soon). For now you can install from master branch.

@c3-cjazra we also made a fix to switch to fsspec in read_csv_glob (https://github.com/modin-project/modin/pull/4898) also available in the master branch.

[mvashishtha]

@vnlitvinov it sounds like reading from azure should work now, but anyway, we should add unit tests checking that.

[anmyachev]

but anyway, we should add unit tests checking that.

@mvashishtha I believe since we completely rely on fsspec now and the code for protocol s3 and abfs is the same (in Modin), then it is enough to test at least one protocol, which is what we are already doing.