Fedavg algorithm in CLIP settings

[ShanCa1]

Sincerely thank you for providing the code. I'm having a little trouble reproducing the Fedavg algorithm in CLIP settings. I optimized both the visual encoder and text encoder with the prompt "a photo of a [class]", but I get the low score. I set the learning rate 5e-5 and use Adam optimizer. Could you pleases give me some suggestions? I noticed the paper "FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning" reported the Fedavg results in ViT32， so I wonder to know how to reproduce the results.