Summary
The paper argue that the efficiency and data privacy of Federated Learning are non-orthogonal from the perspective of model training, which means they are restricting each other. So that the paper strictly formulates the problem at first, and designs a cloud-edge-end hierarchical FL system with adaptive control algorithm embedding a two-level Differential Protection method to relieve both the resource and privacy concerns. The design follows the following ideas:
1.Offload part of the model training from resource-limited end devices to the proximate edges by splitting the model into two parts(shallow layers and deep layers) to improve efficiency.
2.Apply two-level differential privacy (DP) noise injection to protect privacy against the honest-but-curious cloud and the edge server, perturbing model updates to cloud and intermediate features to edges. Specifically, the noise function between end and edge is DP_1(O,sigma_e)=O N(0,S_f^2 sigma_e^2), in which the function simply plus the intermediate features O of the end’s shallow layers and noise function N. For the cloud the DP function is DP_2(w_i^j)=zeta times w_i^j / parallel w_i^j parallel N(0,zeta^2 sigma^2) , in which the model needs to be performed L2-normalization layer by layer.
3.Adaptively coordinate resource optimization and privacy protection to maximize accuracy under resource budget and privacy budgets. Main adaptive controls are:
- Dynamically schedule local and global aggregations based on resource consumption to minimize loss.
- Adjust device sampling rate based on remaining rounds and privacy risk to avoid early termination.
- Tune offloading decision and local noise intensity to minimize resource consumption since more noise make it harder to coverage.
A prototype of AHFL have been implemented and experiments are conducted on CIFAR-10 and MNIST, showing that AHFL reduces end's computation time by 8.58%, communication time by 59.35%, memory by 43.61% and improves accuracy by 6.34% over state-of-the-art approaches like ltAdap, FLGDP, FEEL.
Strengths
- Base on the assumption that security issues are worth concerning which is relatively less considered in some works.
- Offloading model training tasks by splitting the model into two parts while having a relatively comprehensive consideration on the privacy concerns between end device, edge server and cloud.
- Formulate the problem rigorously with math and UML diagram tools.
Weaknesses
- Balancing between iteration times and noise intensity may not be the optimal way because the model can be evaluated in many aspects and the iteration times only contributes to a part of it.
- Does not evaluate the effect of splitting method at the end device because different splitting may cause different computation and influence the offloading.
- The evaluation on CIFAR-10 and MNIST datasets are relatively simple which make it less convincing.
Comments
The paper mainly focus on privacy and efficiency in a federated learning system and design a hierarchical architecture with adaptive algorithms for balancing. Although the experiments are relatively not that convincing and some key details are not introduced, it’s still a innovative work. Besides, the figure 2 impress me a lot for using a UML lane diagram combining formula to precisely illustrate the workflow.
