peft
peft copied to clipboard
huggingface
TST: torch compile tests
Right now, we don't have specific tests for torch.compile. Instead, we have a "hack" that allows to run all tests with torch.compile if we set the environment variable PEFT_DEBUG_WITH_TORCH_COMPILE=1.
This is not very practical because it takes a lot of time to run all these tests with compilation enabled. Also, currently hundreds of tests are failing, which makes it impossible to understand more closely what does or does not work.
This PR removes the aforementioned "hack" and instead replaces it with a list of explicit torch.compile tests. Currently, these tests cover training/inference with a bunch of different tuner types, as well as more advanced features with LoRA (e.g. quantization, multiple adapters, etc.).
Some of these tests pass and some of them fail. This is documented now, so that users can quickly look up if their use case would be compatible with torch.compile. This is very useful to have, because sometimes torch.compile may appear to work but actually returns the wrong result. For users, it's not immediately obvious when this happens.
The test suite is not exhaustive, there are many combinations of features that could be added. However, it should be a good starting point and can be extended later.
The test suite does not cover whether torch.compile actually accelerates the code. This may not be the case even if it works correctly (e.g. because of graph breaks). Testing this would require bigger models and more data, which is prohibitively slow to test.
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
@stevhliu Thanks for the feedback, I added most of your suggestions. I'm not sure about the tables though. Using tables was also my first instinct, but what I dislike about them is 1) there are many empty entries and 2) there is no correspondence between items in the same row, which is typically expected in a table. WDYT?
The empty entries don't really bother me, but 2 is a good point. How about just reorganizing the sections but keeping the lists?
## Training and inference
- list adapters that work and don't work for training and inference
## Advanced PEFT features
- list features that work and don't work
@stevhliu I re-arranged the sections, is this what you had in mind? There are now subsections with identical names, not sure if this could be problematic.