enhancement (datafusion-cli): Add support for glob patterns in CREATE EXTERNAL TABLE commands
Partly closes #16303
The purpose of this PR is to enable using CREATE command with glob pattern and a URL scheme - i.e.,
CREATE EXTERNAL TABLE ee3
STORED AS CSV
LOCATION 's3://tests/data/file-*-1.csv';
CREATE EXTERNAL TABLE pp
STORED AS PARQUET
LOCATION 's3://tests/data-p/te*';
Its currently possible to create an external table using this syntax just for local files:
CREATE EXTERNAL TABLE aa
STORED AS CSV
LOCATION '/Users/aa/projects/tmdb/tmdb_*.csv';
Therefore, the purpose here is to enable support for glob support also for remote url scheme.
The implementation involves some sort of workaround - it intercepts create_plan(), and when the table involves a glob pattern and remote scheme then it creates it as a ListingTable. Part of the reason for this approach is the fact that DataFusion core modules use ListingTable::parse() method in its core modules, which only takes a glob pattern when it invovles local files (see /datafusion/core/src/datasource/listing_table_factory.rs for example).