awslabs
Support for llama and another custom module?
Hello again. I am using PrivacyEngine_Distributed_extending to do private training on a llama2 model. As part of my model, I am using a custom module, a "label attention" block, as defined in here. The label attention has three linear layers, but uses the @ operator, and tensor.mul operators, as part of the forward pass when using the third linear layer parameters.
When training without privacy, I get descent performance on my training data, but the private learning basically fails having very high training loss. I am wondering if I am doing everything correctly, or if I have to extend the library in order to do my application. I am suspect that maybe I have to change both fastDP/transformers_support.py and fastDP/supported_differentially_private_layers.py. Can you please clarify if my guess is right and I have to modify the library in some way, and what are the required modifications? Also, I would appreciate any pointers on what to change and how to do it.
