Friday, May 9, 2025
News PouroverAI
Visit PourOver.AI
No Result
View All Result
  • Home
  • AI Tech
  • Business
  • Blockchain
  • Data Science & ML
  • Cloud & Programming
  • Automation
  • Front-Tech
  • Marketing
  • Home
  • AI Tech
  • Business
  • Blockchain
  • Data Science & ML
  • Cloud & Programming
  • Automation
  • Front-Tech
  • Marketing
News PouroverAI
No Result
View All Result

GROKFAST: A Machine Learning Approach that Accelerates Grokking by Amplifying Slow Gradients

June 6, 2024
in AI Technology
Reading Time: 4 mins read
0 0
A A
0
Share on FacebookShare on Twitter


Grokking is a newly developed phenomenon where a model starts to generalize well long after it has overfitted to the training data. It was first seen in a two-layer Transformer trained on a simple dataset. In grokking, generalization occurs only after many more training iterations than overfitting. This requires high computational resources, making it less practical for most machine learning practitioners with limited resources. To fully understand this unusual behavior, there is a need for faster generalization in these overfitting systems. The main aim is to speed up the grokking phenomenon.

An existing method, grokking is a newly discovered phenomenon that shows that overparameterized neural networks can generalize and reason beyond just memorizing the dataset. Most research focuses on understanding this mechanism, linking grokking to the double descent phenomenon, where validation error first increases and then decreases as model parameters grow. Apart from this method, optimization techniques are used in which a model’s generalization patterns vary significantly with different optimization methods like mini-batch training, choice of optimizer, weight decay, noise injection, dropout, and learning rate all affect the model’s grokking pattern.

Researchers from the Seoul National University, Korea introduced GROKFAST, an algorithm that accelerated grokking by amplifying slow gradients. Researchers proved through experiments that GROKFAST algorithms have the potential to solve a wide variety of tasks that contain images, languages, and graphs. This makes the unique artifact of immediate generalization practically useful. Further, the parameter trajectories under gradient descent are split into two components: the fast-varying, overfitting-yielding component and the slow-varying, generalization-inducing component. This analysis helps the grokking method to become 50 times faster with only a few lines of code.

During the experiment, the idea of the algorithmic dataset used in the first report on grokking is shown, where the network is a two-layer decoder-only transformer trained to predict the answer of a modular binary multiplication operation. Comparing the time to reach an accuracy of 0.95, the validation accuracy keeps improving longer. It reaches its peak 97.3 times later than the training accuracy, which quickly reaches its maximum and starts overfitting. Further, hyperparameters are selected from a simple grid search, and found that the filter works best when λ(scalar factor) = 5 and w(window size) = 100. Also, there is a reduction of 13.57 times in the number of iterations to reach the validation accuracy of 0.95, which is a good result.

The proposed method is based on the idea that slow gradients (low-pass filtered gradient updates) help in generalization. The training dynamics of a model are interpreted under grokking as a state transition, where the model goes through three stages:

  • Initialized, where both the training and validation losses are not saturated
  • Overfitted, where the training loss is completely saturated but validation loss is not saturated
  • Generalized, where both losses are saturated

Moreover, research suggests that the weight decay hyperparameter significantly plays a critical role in the grokking phenomenon.

In conclusion, researchers from the Seoul National University, Korea have proposed an algorithm, GROKFAST, that accelerated grokking phenomenon by amplifying slow gradients. The analysis of how each model parameter changes into a random signal during training iteration helps to separate gradient updates into fast-varying and slow-varying components. Despite showing outstanding results, there is a limit in the utilization by GROKFAST that needs w times more memory to store all the previous gradients. Also replication of the model parameters also makes the training slower.

Check out the Paper and GitHub. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

Sajjad Ansari is a final year undergraduate from IIT Kharagpur. As a Tech enthusiast, he delves into the practical applications of AI with a focus on understanding the impact of AI technologies and their real-world implications. He aims to articulate complex AI concepts in a clear and accessible manner.

🐝 Join the Fastest Growing AI Research Newsletter Read by Researchers from Google + NVIDIA + Meta + Stanford + MIT + Microsoft and many others…

If you like our work, you will love our newsletter..

Don’t Forget to join our 43k+ ML SubReddit | Also, check out our AI Events Platform



Source link

Tags: acceleratesAmplifyingApproachGradientsGROKFASTGrokkingLearningMachineslow
Previous Post

Anthropic’s Claude AI Now Available in Canada

Next Post

sanjiv bhasin stocks: FMCG to outperform in near term; bet on Bata, Marico and Nestle: Sanjiv Bhasin

Related Posts

How insurance companies can use synthetic data to fight bias
AI Technology

How insurance companies can use synthetic data to fight bias

June 10, 2024
From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset
AI Technology

From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset

June 10, 2024
Decoding Decoder-Only Transformers: Insights from Google DeepMind’s Paper
AI Technology

Decoding Decoder-Only Transformers: Insights from Google DeepMind’s Paper

June 9, 2024
How Game Theory Can Make AI More Reliable
AI Technology

How Game Theory Can Make AI More Reliable

June 9, 2024
Buffer of Thoughts (BoT): A Novel Thought-Augmented Reasoning AI Approach for Enhancing Accuracy, Efficiency, and Robustness of LLMs
AI Technology

Buffer of Thoughts (BoT): A Novel Thought-Augmented Reasoning AI Approach for Enhancing Accuracy, Efficiency, and Robustness of LLMs

June 9, 2024
Deciphering Doubt: Navigating Uncertainty in LLM Responses
AI Technology

Deciphering Doubt: Navigating Uncertainty in LLM Responses

June 9, 2024
Next Post
sanjiv bhasin stocks: FMCG to outperform in near term; bet on Bata, Marico and Nestle: Sanjiv Bhasin

sanjiv bhasin stocks: FMCG to outperform in near term; bet on Bata, Marico and Nestle: Sanjiv Bhasin

From Simple Rules to Smart Exploration: Intelligent Go-Explore IGE Bridges the Gap with Foundation Models in Autonomous Systems

From Simple Rules to Smart Exploration: Intelligent Go-Explore IGE Bridges the Gap with Foundation Models in Autonomous Systems

Microsoft CA to PKIaaS | 7 Reasons to Replace Your Microsoft CA

Microsoft CA to PKIaaS | 7 Reasons to Replace Your Microsoft CA

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

  • Trending
  • Comments
  • Latest
Is C.AI Down? Here Is What To Do Now

Is C.AI Down? Here Is What To Do Now

January 10, 2024
Porfo: Revolutionizing the Crypto Wallet Landscape

Porfo: Revolutionizing the Crypto Wallet Landscape

October 9, 2023
A Complete Guide to BERT with Code | by Bradney Smith | May, 2024

A Complete Guide to BERT with Code | by Bradney Smith | May, 2024

May 19, 2024
A faster, better way to prevent an AI chatbot from giving toxic responses | MIT News

A faster, better way to prevent an AI chatbot from giving toxic responses | MIT News

April 10, 2024
Part 1: ABAP RESTful Application Programming Model (RAP) – Introduction

Part 1: ABAP RESTful Application Programming Model (RAP) – Introduction

November 20, 2023
Saginaw HMI Enclosures and Suspension Arm Systems from AutomationDirect – Library.Automationdirect.com

Saginaw HMI Enclosures and Suspension Arm Systems from AutomationDirect – Library.Automationdirect.com

December 6, 2023
Can You Guess What Percentage Of Their Wealth The Rich Keep In Cash?

Can You Guess What Percentage Of Their Wealth The Rich Keep In Cash?

June 10, 2024
AI Compared: Which Assistant Is the Best?

AI Compared: Which Assistant Is the Best?

June 10, 2024
How insurance companies can use synthetic data to fight bias

How insurance companies can use synthetic data to fight bias

June 10, 2024
5 SLA metrics you should be monitoring

5 SLA metrics you should be monitoring

June 10, 2024
From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset

From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset

June 10, 2024
UGRO Capital: Targeting to hit milestone of Rs 20,000 cr loan book in 8-10 quarters: Shachindra Nath

UGRO Capital: Targeting to hit milestone of Rs 20,000 cr loan book in 8-10 quarters: Shachindra Nath

June 10, 2024
Facebook Twitter LinkedIn Pinterest RSS
News PouroverAI

The latest news and updates about the AI Technology and Latest Tech Updates around the world... PouroverAI keeps you in the loop.

CATEGORIES

  • AI Technology
  • Automation
  • Blockchain
  • Business
  • Cloud & Programming
  • Data Science & ML
  • Digital Marketing
  • Front-Tech
  • Uncategorized

SITEMAP

  • Disclaimer
  • Privacy Policy
  • DMCA
  • Cookie Privacy Policy
  • Terms and Conditions
  • Contact us

Copyright © 2023 PouroverAI News.
PouroverAI News

No Result
View All Result
  • Home
  • AI Tech
  • Business
  • Blockchain
  • Data Science & ML
  • Cloud & Programming
  • Automation
  • Front-Tech
  • Marketing

Copyright © 2023 PouroverAI News.
PouroverAI News

Welcome Back!

Login to your account below

Forgotten Password? Sign Up

Create New Account!

Fill the forms bellow to register

All fields are required. Log In

Retrieve your password

Please enter your username or email address to reset your password.

Log In