Refactor `tiktoken` import bare except
We shouldn't suppress the import error if tiktoken is not installed. Also do the same thing for sentencepiece.
Instead of importing them at the top of the file, we can import them inside the only functions that use them; we can consider a different import structure if we end up needing to access them elsewhere.
This lazy importing should also decrease the load time of these modules.
cc: @msaroufim
:link: Helpful Links
:test_tube: See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1024
- :page_facing_up: Preview Python docs built from this PR
Note: Links to docs will display an error until the docs builds have been completed.
:white_check_mark: No Failures
As of commit a1e9b2c316bb41ce56a26332ca32919f38668bd6 with merge base c187f876d2f15c02b350f5e1c50855b485132a55 ():
:green_heart: Looks good so far! There are no failures yet. :green_heart:
This comment was automatically generated by Dr. CI and updates every 15 minutes.