khiops
khiops copied to clipboard
Support file globbing for "any" filesystem (thus for local one too)
Description
Currently in khiops (versions 10 and 11), the file globbing is supported for remote file systems (S3, GCS) via the corresponding drivers.
Example of a path with a globbing pattern :
gs://mybucket/folder/*/*.csv
All the files are concatenated to produce one source of data.
Questions/Ideas
An internal customer finds strange that this globbing feature is not supported for example for the local system too.
For his CI/CD chain and his production system he would like to be able to configure as easily as below
- CI/CD (Automated Integration Test) :
local path :
/folder/*/*.csv
- Q/A, Production :
remote path :
gs:/bucket/folder/*/*.csv
If it does not lead to a major rewrite it would be nice to have this homogenous globbing feature for every filesystem.