About the threshold value

[ryanyangjt]

I notice that you set the threshold value as 1.1 in the run.sh.

But the code used to check the flag with the threshold in the metafed.py will never be triggered if you set it as 1.1, right?

if val_acc > self.args.threshold: self.flagl[idx] = True As a result, the experiment result you provided here may not be represented for the metafed. They would be normal training results, right?

Please let me know if I have any misunderstandings.

Thanks!

[lw0517]

In fact, it is a special case of MetaFed. In some situations, threshold>1 can bring better performance and there still exist some differences between MetaFed and FedAvg. There also exist situations with threshold<1, e.g. Figure 6 in MetaFed.

[ryanyangjt]

Thanks for your reply. Could you please suggest a data split case that can train well with the threshold < 1 if it is convenient for you? When I try the case of threshold < 1, like 0.9, 0.8, 0.7, etc., I only can get very bad results (ACC < 70) based on the data split you provided here. Thank you!

[lw0517]

You can try some other datasets, e.g. PAMAP2. For this situation, the best results are fixed in the fixed environment. MetaFed is a framework. The threshold is only a hyperparameter and different values correspond to different strategies.

[d2461795341]

when threshold=1.1 maybe it likes an iid suituation, not a non-iid problem?

[lw0517]

when threshold=1.1 maybe it likes an iid suituation, not a non-iid problem?

In fact, it is still for a non-iid problem. Besides the common knowledge accumulation stage, there also exists a stage for personalization. In addition, MetaFed is universal and it can be utilized in an iid situation.