Feature Caching - Writing Metadata File fails

[mh0797]

Describe the bug

The feature caching results in an error right after computing all the features.

Setup

The error occurs on my local machine as well as on a cluster instance (both linux) using the devkit-0.6. The environment was reinstalled after updating the devkit

Steps To Reproduce

Steps to reproduce the behavior:

1. Run command python nuplan/planning/script/run_training.py +training=training_vector_model py_func=cache cache.cache_path=/path/to/cache data_augmentation="[]"^

Stack Trace

Traceback (most recent call last):
  File "/home/aah1si/anaconda3/envs/nuplan/lib/python3.9/site-packages/fsspec/registry.py", line 234, in get_filesystem_class
    register_implementation(protocol, _import_class(bit["class"]))
  File "/home/aah1si/anaconda3/envs/nuplan/lib/python3.9/site-packages/fsspec/registry.py", line 257, in _import_class
    mod = importlib.import_module(mod)
  File "/home/aah1si/anaconda3/envs/nuplan/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 's3fs'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/aah1si/store/nuplan/nuplan-devkit/nuplan/planning/script/run_training.py", line 73, in main
    cache_data(cfg=cfg, worker=worker)
  File "/home/aah1si/store/nuplan/nuplan-devkit/nuplan/planning/training/experiments/caching.py", line 175, in cache_data
    save_cache_metadata(cached_metadata, Path(cfg.cache.cache_path), node_id)
  File "/home/aah1si/store/nuplan/nuplan-devkit/nuplan/planning/training/experiments/cache_metadata_entry.py", line 61, in save_cache_metadata
    :return: List of CacheMetadataEntry.
  File "/home/aah1si/anaconda3/envs/nuplan/lib/python3.9/site-packages/pandas/core/generic.py", line 3551, in to_csv
    return DataFrameRenderer(formatter).to_csv(
  File "/home/aah1si/anaconda3/envs/nuplan/lib/python3.9/site-packages/pandas/io/formats/format.py", line 1180, in to_csv
    csv_formatter.save()
  File "/home/aah1si/anaconda3/envs/nuplan/lib/python3.9/site-packages/pandas/io/formats/csvs.py", line 241, in save
    with get_handle(
  File "/home/aah1si/anaconda3/envs/nuplan/lib/python3.9/site-packages/pandas/io/common.py", line 667, in get_handle
    ioargs = _get_filepath_or_buffer(
  File "/home/aah1si/anaconda3/envs/nuplan/lib/python3.9/site-packages/pandas/io/common.py", line 382, in _get_filepath_or_buffer
    file_obj = fsspec.open(
  File "/home/aah1si/anaconda3/envs/nuplan/lib/python3.9/site-packages/fsspec/core.py", line 441, in open
    return open_files(
  File "/home/aah1si/anaconda3/envs/nuplan/lib/python3.9/site-packages/fsspec/core.py", line 273, in open_files
    fs, fs_token, paths = get_fs_token_paths(
  File "/home/aah1si/anaconda3/envs/nuplan/lib/python3.9/site-packages/fsspec/core.py", line 621, in get_fs_token_paths
    cls = get_filesystem_class(protocol)
  File "/home/aah1si/anaconda3/envs/nuplan/lib/python3.9/site-packages/fsspec/registry.py", line 236, in get_filesystem_class
    raise ImportError(bit["err"]) from e
ImportError: Install s3fs to access S3

Additional context

I am not using an s3 server but just a local machine. The processed features seem to be fine.

[Jeff09]

Thanks for bringing up. I have got the same error.

[mspryn-motional]

Thanks for pointing this out! This will be fixed in the next release of the Nuplan devkit. Just to confirm, the path_to_cache that you are passing to the training command does not begin with s3://, correct?

[JingyuQian]

Can confirm this is an issue. It's the writing metadata csv that's getting affected, the cached data itself can be used, though.

Suggest an edit in the save_cache_metadata function:

if str(cache_path).startswith("s3"):
    sanitised_cache_path = sanitise_s3_path(cache_path)
else:
    sanitised_cache_path = str(cache_path)

# don't forget to makedirs for the sanitised_cache_path in local

Also need to be aware that the path to save metadata shouldn't be inside the cache_path, otherwise some cache_path.iterdir() functions will treat the metadata path as a cache folder.

[patk-motional]

@JingyuQian @mh0797,

Has this issue been resolved for you since the release of v1.0?

[mh0797]

Hi @patk-motional, Yes, it has been! - Thank you