neelnanda-io
[Proposal] Merge utils and utilities
Proposal
Right now there are two modules in the code that are organized to basically do the same thing. The utils.py file should be broken up and organized into individual pieces within utlities. The pieces themselves can be a single function per file, or groups of functions that have a logical reason to go together. The utils.py will then need to be updated to import the newly reorganized functions in order to maintain compatibility until at least the next major version of TransformerLens.
Once the code organization is done, the unit tests need to be reorganized a bit too. Any existing unit and integration tests should also be broken up, and moved into a subdirectory with a single test per file. e.g. if there is a util file named tokenization.py, the test should then be test_tokenization.py in a utils directory. This structure will make it really easy for us to immediately identify a file where a bug has been introduced, as well as make it easier to identify where tests are lacking.
Optional
Once the tests have been reorganized, if additional tests can be added to any functions we find lacking in testing, that would be super cool, but not required.
Motivation
Let's get this all into a single module, so that the code itself can be better organized, and easier to understand.
Additional context
Checklist
- [x] I have checked that there is no similar issue in the repo (required)
