Federated Learning: Empowering Decentralized Collaboration in Machine Learning
The standard approach for model training in machine learning has been characterized by centralized data storage and processing. Centralized training requires the aggregation of data from various sources into a single location, raising significant privacy concerns, particularly in industries handling sensitive information like healthcare or finance. Moreover, transfer and communication costs and delays associated with moving data from its sources to a central server are another drawback of the conventional training paradigm. These limitations underscore the need for a more decentralized and privacy-preserving approach to machine learning.
Federated learning proposes a decentralized training approach in which models are trained directly on the devices or servers where the data resides. This approach addresses privacy concerns by keeping data local and alleviates costs associated with transferring large volumes of data to a central server. Federated learning operates on the principle of collaborative model training, where local models are trained on distributed data sources, and only model updates are shared with a central server for aggregation.
Federated learning was initially proposed by Google researchers in the paper titled "Communication-Efficient Learning of Deep Networks from Decentralized Data" [1], published in 2016. In this framework, federated learning begins with the central initialization of a global model. Each device then conducts local training on its dataset, updating the local model parameters in each round. These locally trained model parameters are subsequently transmitted to the central server, where they are aggregated using predefined methods such as averaging. Following this, the server produces an updated version of the global model and broadcasts it to all clients. This process consolidates insights from diverse data sources into an updated global model. The iterative process, consisting of local training, update aggregation, and model refinement, persists over multiple rounds to enhance the global model's performance while ensuring data privacy and minimizing communication overhead.
Federated learning requires different system designs depending on the type of data partitioning and client type [2]. According to the type of data partitioning, federated learning is divided into three categories: vertical, horizontal, and transfer federated learning. In horizontal federated learning, the data sets involved in the learning process share the same feature space but have different sample spaces. For example, when the user base (sample space) of several banks located in different regions is different, but the features they hold about their users (income level, account activity, etc.) are the same, horizontal FL can be a suitable approach. In vertical federated learning, the data sets share the same sample space but have different feature spaces. For example, a bank and an e-commerce company serving the same region provide services to the same or largely similar users. However, the information held by these companies about users differs, so the feature spaces may vary. Federated Transfer Learning is typically viewed as the convergence of horizontal and vertical FL. In essence, Transfer Learning is a learning paradigm employed to enhance a model trained on a particular dataset by integrating learned insights from another dataset. In Federated Transfer Learning, the goal is to improve the model or learning process of one client (the target domain) by leveraging knowledge or models from other clients (the source domains) without explicitly sharing raw data.
Depending on the client type, federated learning is divided into cross-device and cross-silo approaches. Cross-device federated learning involves individual devices, such as smartphones, IoT devices, or edge devices, as the participating clients. Each device holds its local data, and federated learning is used to train a global model across all devices without the need for centralized data aggregation. In the cross-device approach, the number of clients can reach up to millions. In cross-silo federated learning, the participating clients are organizations, companies, or institutions with distinct data silos. Cross-silo federated learning enables organizations to leverage insights from their respective data silos while preserving data privacy.
The applications of federated learning are extending across diverse industries and use cases. In healthcare, this approach facilitates collaborative model training on distributed medical datasets, driving improvements in diagnosis and treatment while upholding patient privacy. Likewise, in finance, federated learning empowers institutions to analyze extensive financial data for risk assessment and fraud detection without exposing sensitive information. Furthermore, federated learning shows potential in telecommunications, where it can optimize network performance and elevate user experiences by leveraging data from edge devices while maintaining user privacy.
References
[1] McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." Artificial intelligence and statistics. PMLR, 2017
[2] Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019). Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2), 1-19.