snowflake-connector-python icon indicating copy to clipboard operation
snowflake-connector-python copied to clipboard

Make file_transfer_agent and pkg_resources imports lazy

Open AleksaC opened this issue 2 years ago • 3 comments

Fixes #1253

I noticed that two modules take up a significant portion of import time but aren't necessary for many use cases. For instance pkg_resources doesn't have to be imported if pandas isn't used. Similarly file_transfer_agent isn't going to be used if only executing get requests.

Moving these imports from the top level to the function they are used in can improve import time from:

$ hyperfine 'python -c "import snowflake.connector"'
Benchmark 1: python -c "import snowflake.connector"
  Time (mean ± σ):     466.0 ms ±  56.2 ms    [User: 434.9 ms, System: 29.1 ms]
  Range (min … max):   427.6 ms … 616.1 ms    10 runs

to:

$ hyperfine 'python -c "import snowflake.connector"'
Benchmark 1: python -c "import snowflake.connector"
  Time (mean ± σ):     255.3 ms ±   4.9 ms    [User: 233.8 ms, System: 21.8 ms]
  Range (min … max):   248.9 ms … 264.9 ms    11 runs

This is of course the best case. If pandas is used pkg_resources will get imported and no time will be saved there and if file transfer is used the time to import file transfer module will be paid on the first usage instead of the import time.

AleksaC avatar Sep 17 '22 01:09 AleksaC

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

github-actions[bot] avatar Sep 17 '22 01:09 github-actions[bot]

I have read the CLA Document and I hereby sign the CLA

AleksaC avatar Sep 17 '22 01:09 AleksaC

recheck

AleksaC avatar Sep 17 '22 01:09 AleksaC

Closing this PR. Revived this with tests here: https://github.com/snowflakedb/snowflake-connector-python/pull/1512

sfc-gh-aalam avatar Apr 17 '23 21:04 sfc-gh-aalam