neural icon indicating copy to clipboard operation
neural copied to clipboard

Feat: Add token counting

Open Angelchev opened this issue 1 year ago • 0 comments

Closes: #41

Overview

  • [x] Add command to count tokens from a visual selection.
  • [x] Add python script to count tokens with OpenAI's tiktoken.
  • [x] Add utils.vim to launch python scripts more easily.
  • [x] Add automatic venv setup for Neovim with lazy.nvim package manager via build.lua.
    • [x] Create a python virtual environment automatically for Linux or Windows (Untested).
    • [x] Install dependencies from requirements.txt automatically.
  • [x] Add Neovim animated notification integration with nvim-notify.

Todo

  • [ ] Add python test for new script
  • [ ] Check for tiktoken dependency in Python script, and throw error if not present.
  • [ ] Investigate automatically setting up python venv and installing requirements for vim 8.
  • [ ] Update README.md to document optionally setting up python venv.
  • Update vimdocs:
    • [ ] New command and availability dependant on tiktoken
    • [ ] Integration of nvim-notify

Testing

  • Test automatic setup for Neovim with lazy.nvim.
    • [x] Linux
    • [ ] Mac OSX
    • [ ] Windows

Screenshots

Neovim Animated Notifications via nvim-notify

Screenshot_20240103_233935

Automatic venv setup for Neovim with lazy.nvim

Screenshot_20240103_234057

Angelchev avatar Jan 03 '24 23:01 Angelchev