Performance compared to querying virtual filesystem through `rclone mount`

[daviewales]

I have a 1.4 GB parquet file in Azure storage.

I can mount the storage location using rclone as follows:

rclone mount account_name:container_name ~/local_mount_point --vfs-cache-mode full --max-read-ahead 1024Ki --read-only

In this scenario, the following query runs in 103 seconds:

select * from '~/local_mount_point/directory/file.parquet' limit 10;

I can also use the duckdb_azure extension to connect to the file directly:

force install azure from 'http://nightly-extensions.duckdb.org';
LOAD azure;
set azure_account_name='account_name';
set azure_credential_chain='cli';
select * from 'azure://container/directory/file.parquet' limit 10;

In this case, the query completes in 240 seconds.

I'm guessing there may be performance enhancements possible to make the direct query as fast (or faster?) than the query through rclone mount.

(I'm aware this is a very early preview. Thanks for your amazing work!)

Note that rclone provides some local caching, so subsequent runs of the same query run much faster. For example, re-running the same query from above completes in only 7 seconds on the second run through rclone. Re-running the direct query took 208 seconds.

I'm not sure how much of the first load performance difference is due to rclone caching.