When utilizing the popular backpropagation as the default learning method, training deep neural networks—which can include hundreds of layers—can be a laborious process that can last weeks. Since the backpropagation learning algorithm is sequential, it isn’t easy to parallelize these models, even though the process works fine on a single computing unit. Each layer’s gradient in backpropagation depends on the gradient computed at the layer below it. Because each node in a distributed system needs to wait for gradient information from its successor before continuing with its calculations, the long waiting times between nodes directly result from this sequential dependency. Further, there can be a lot of communication overhead if nodes constantly talk to each other to share weight and gradient data.
This becomes an even bigger issue when dealing with massive neural networks, where a lot of data needs to be sent. The ever-increasing size and complexity of neural networks have propelled distributed deep learning to new heights in recent years. Key solutions that have arisen include distributed training frameworks like GPipe, PipeDream, and Flower. These frameworks optimize for speed, usability, cost, and size, allowing for the training of huge models. Data, pipeline, and model parallelism are some of the advanced approaches used by these systems to efficiently manage and perform training of large-scale neural networks across numerous processing nodes.
The Forward-Forward (FF) technique, which Hinton developed, offers a fresh method for training neural networks, in addition to the studies above focused on distributed backpropagation implementations. In contrast to more conventional deep learning algorithms, the Forward-Forward algorithm performs all of its computations locally, layer by layer. In a distributed scenario, FF’s layer-wise training feature leads to a less reliant architecture, which reduces idle time, communication, and synchronization. This contrasts with backpropagation, primarily focused on solving problems without distribution.
A new study by Sabanci University presents training distributed neural networks with a Forward-Forward Algorithm called Pipeline Forward-Forward Algorithm (PFF). Because it does not impose the dependencies of backpropagation on the system, PFF achieves higher use of computational units with fewer bubbles and idle time. This fundamentally differs from the classic implementations with backpropagation and pipeline parallelism. Experiments with PFF reveal that, compared to the typical FF implementation, the PFF Algorithm achieves the same level of accuracy while being four times faster.
Compared to an existing distributed implementation of Forward-Forward (DFF), PFF achieves 5% more accuracy in 10% fewer epochs, demonstrating even bigger benefits. Because PFF only transmits the layer information (weights and biases), whereas DFF transmits the entire output data, the amount of data shared between layers in PFF is significantly lower than in DFF. When contrasted with DFF, this leads to lower communication overhead. Beyond the remarkable outcomes of PFF, the team hopes that their study opens a fresh chapter in the Distributed Neural Network training field.
The team also discusses several methods that exist for enhancing PFF.
The present implementation of PFF allows for parameter exchange between various layers after each chapter. The team highlights that trying this swap after each batch may be worthwhile if it helps fine-tune the weights and yields more accurate results. But there’s a chance it might raise the communication overhead.
Using PFF in Federated Learning: Since PFF doesn’t share data with other nodes during model training, it can be used to establish a Federated Learning system in which each node contributes its data.
Sockets were utilized to establish communication between various nodes in the experiments conducted in this work. Data transmission across a network adds extra communication overhead. The team suggests that a multi-GPU architecture, in which the PFF’s processing units are physically near together and share a resource, can significantly reduce the time needed to train a network.
The Forward-Forward Algorithm relies heavily on generating negative samples since it influences the network’s learning process. Therefore, greater system performance is assuredly achievable by discovering novel and improved negative sample production methods.
Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.
If you like our work, you will love our newsletter..
Don’t Forget to join our 40k+ ML SubReddit
Asjad is an intern consultant at Marktechpost. He is persuing B.Tech in mechanical engineering at the Indian Institute of Technology, Kharagpur. Asjad is a Machine learning and deep learning enthusiast who is always researching the applications of machine learning in healthcare.
