FedKT icon indicating copy to clipboard operation
FedKT copied to clipboard

Published 20 hours ago • verified publisher icon QinbinLi

Accuracy issues on implementing

Open ashwiniyer176 opened this issue 3 years ago • 5 comments

  1. The Accuracy constantly is 0.0 regardless of the changing hyperparameters
  2. When observing the vote counts, the total number of votes is always 10.
  3. What are the minimum and maximum limit for teacher models per party
  4. Is it necessary that the public dataset be the same as the private dataset?
  5. In case of lack of public dataset, how will model wor output.txt k?

ashwiniyer176 avatar Oct 15 '21 13:10 ashwiniyer176

Hi ,

  1. In your script, the parameters epoch, stu_epochs, n_teacher_each_partition, n_partition are too low. Can you try the provided script to see whether the accuracy is 0.0?
  2. The total number of votes is equal to the number of models.
  3. THere is no limit for the number of teacher models per party. However, the performance may vary a lot with a different number of models.
  4. The public dataset is from the same source as the private dataset (e.g., all are from MNIST) in our experiments. The performance may drop a lot if using a different public dataset.
  5. FedKT does not work if there is no public dataset currently.

QinbinLi avatar Oct 15 '21 15:10 QinbinLi

I changed them to the default values. But still no difference. Also, the experiment generates log files, but there is nothing in them. All I get are 0KB files with date and time of the experiment

ashwiniyer176 avatar Nov 03 '21 07:11 ashwiniyer176

I changed them to the default values. But still no difference. Also, the experiment generates log files, but there is nothing in them. All I get are 0KB files with date and time of the experiment

Hello, I also have this problem now, I would like to know now this problem you have solved？

Cmnotjx avatar Mar 22 '22 23:03 Cmnotjx

Hi @Cmnotjx and @ashwiniyer176 ,

What is the operating system you used? I can successfully run the scripts and generate non-empty logs in a Linux system. There may be compatibility issues on Windows.

QinbinLi avatar Mar 23 '22 03:03 QinbinLi

Hi @Cmnotjx and @ashwiniyer176 ,

What is the operating system you used? I can successfully run the scripts and generate non-empty logs in a Linux system. There may be compatibility issues on Windows.

Thank you very much for your reply, I am using a windows system, I will use a Linux system to try

Cmnotjx avatar Mar 23 '22 08:03 Cmnotjx