torchtune icon indicating copy to clipboard operation
torchtune copied to clipboard

Fix eval recipe for consecutive generation and non-generation tasks

Open SalmanMohammadi opened this issue 5 months ago • 0 comments

Currently, if we specify multiple tasks for the eval recipe, and one of the tasks is a generation task which uses KV-cacheing, then the cache is still enabled for non-generation tasks. The recipe will then error out here.

We should try having some check when using cacheing in addition to caches_are_enabled, which also checks that we're in e.g. inference mode.

EDIT: This is generally an issue for our models and will also become apparent for LoRA PPO.

SalmanMohammadi avatar Sep 18 '24 19:09 SalmanMohammadi