Symlinks are not found when loading configuration files

[ElenaKhaustova]

Description

When having parameters in conf/base/sub_folder/parameters_A.yml structure where sub_folder is symlinked, the pipeline gives the following error, however, it is able to read the parameters from the following structure: conf/base/parameters_A.yml

ValueError: Pipeline input(s) {'params:length', 'params:width'} not found in the DataCatalog

Context

The issue is that fsspec.filesystem.glob() which we use to find the paths recursively doesn’t find symlinks.

https://github.com/kedro-org/kedro/blob/adfc593bcd2f1b74676e7ab7c1a3b9c168b7257f/kedro/config/omegaconf_config.py#L295

Steps to Reproduce

For a default spaceflights-pandas project create a symlink folder and place parameters_data_science.yml in the linked folder.

Run the pipeline.

Expected Result

Symlinks are found when loading configuration files.

Your Environment

• Kedro version used (pip show kedro or kedro -V): kedro, version 0.19.6
• Python version used (python -V): Python 3.11.9
• Operating system and version: macOS Sonoma version 14.5