Sunday, June 8, 2025
News PouroverAI
Visit PourOver.AI
No Result
View All Result
  • Home
  • AI Tech
  • Business
  • Blockchain
  • Data Science & ML
  • Cloud & Programming
  • Automation
  • Front-Tech
  • Marketing
  • Home
  • AI Tech
  • Business
  • Blockchain
  • Data Science & ML
  • Cloud & Programming
  • Automation
  • Front-Tech
  • Marketing
News PouroverAI
No Result
View All Result

Parameter-Efficient Sparsity Crafting (PESC): A Novel AI Approach to Transition Dense Models to Sparse Models Using a Mixture-of-Experts (Moe) Architecture

January 20, 2024
in AI Technology
Reading Time: 3 mins read
0 0
A A
0
Share on FacebookShare on Twitter


The emergence of large language models (LLMs) like GPT, Claude, Gemini, LLaMA, Mistral, etc., has greatly accelerated recent advances in natural language processing (NLP). Instruction tweaking is a well-known approach to training LLMs. This method allows LLMs to improve their pre-trained representations to follow human instructions using large-scale, well-formatted instruction data. However, these tasks are complex in and of themselves, making fine-tuning the model difficult. For general tasks, larger models may not be able to maximize losses from competing activities, leading to poor performance.

Increasing the model’s capacity can enhance instruction tuning’s efficacy for general tasks. Most LLMs, however, are dense pre-trained models built using transformer architecture, severely restricting scalability when tweaking the instructions. Instruction tweaking offers the chance to obtain outstanding performance on general tasks by turning dense models into MoE models. The MoE models’ expert layers are initially set up as duplicates of the original feedforward neural network (FFN) layers to make this change. Training such massive models is hindered by computational costs and GPU memory constraints caused by the need to update the expert weights in the MoE layer due to the large parameter scale of existing LLMs. 

New research by the Shanghai Artificial Intelligence Laboratory and The Chinese University of Hong Kong presents Parameter-Efficient Sparsity Crafting (PESC), a method for transforming dense models into sparse ones using the MoE blueprint. By integrating adapters into sparse models’ MoE layers, PESC makes it possible to differentiate experts without changing their weights individually. This method drastically cuts down on GPU memory needs and computational expenses. Because adapters are integrated, the model capacity can be expanded with minimal increase in parameters.

To differentiate across experts without changing the weights of each expert in the MoE layers, PESC inserts adapters into the MoE layers of sparse models. The researchers also update other sparse model weights using the QLoRA methodology, a popular PEFT method. 

The researchers simultaneously trained the sparse model with MoE layers on various skills, including coding, mathematics, and other general talents from many areas, to illustrate the model’s learning capabilities. For instruction tuning, this training integrated three separate datasets from different domains: SlimORCA, Magicoder, and MetaMathQA datasets. The final dataset included 520k instructions after filtering and sampling.

Furthermore, they have utilized the PESC method to create Camelidae sparse models. Camelidae-8Ï34B outperforms GPT-3.5 in general and reaches SOTA performance on all open-source sparse models.

Check out the Paper and Model. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our Telegram Channel

\"\"

Dhanshree Shenwai is a Computer Science Engineer and has a good experience in FinTech companies covering Financial, Cards & Payments and Banking domain with keen interest in applications of AI. She is enthusiastic about exploring new technologies and advancements in today’s evolving world making everyone\’s life easy.

🐝 Join the Fastest Growing AI Research Newsletter Read by Researchers from Google + NVIDIA + Meta + Stanford + MIT + Microsoft and many others…



Source link

Tags: ApproachArchitectureCraftingDenseMixtureofExpertsmodelsMoEParameterEfficientPESCsparsesparsitytransition
Previous Post

To Data or Not to Data.. The question is not anymore whether we… | by Erdogan Taskesen | Jan, 2024

Next Post

Nikki Haley’s last push in New Hampshire as Trump steps up verbal attacks

Related Posts

How insurance companies can use synthetic data to fight bias
AI Technology

How insurance companies can use synthetic data to fight bias

June 10, 2024
From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset
AI Technology

From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset

June 10, 2024
Decoding Decoder-Only Transformers: Insights from Google DeepMind’s Paper
AI Technology

Decoding Decoder-Only Transformers: Insights from Google DeepMind’s Paper

June 9, 2024
How Game Theory Can Make AI More Reliable
AI Technology

How Game Theory Can Make AI More Reliable

June 9, 2024
Buffer of Thoughts (BoT): A Novel Thought-Augmented Reasoning AI Approach for Enhancing Accuracy, Efficiency, and Robustness of LLMs
AI Technology

Buffer of Thoughts (BoT): A Novel Thought-Augmented Reasoning AI Approach for Enhancing Accuracy, Efficiency, and Robustness of LLMs

June 9, 2024
Deciphering Doubt: Navigating Uncertainty in LLM Responses
AI Technology

Deciphering Doubt: Navigating Uncertainty in LLM Responses

June 9, 2024
Next Post
Nikki Haley’s last push in New Hampshire as Trump steps up verbal attacks

Nikki Haley's last push in New Hampshire as Trump steps up verbal attacks

No signs of panic as Nikola gets 2nd Nasdaq delisting warning

No signs of panic as Nikola gets 2nd Nasdaq delisting warning

Video: Building a Greener Web (Smashing Meets Goes Green)

Video: Building a Greener Web (Smashing Meets Goes Green)

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

  • Trending
  • Comments
  • Latest
23 Plagiarism Facts and Statistics to Analyze Latest Trends

23 Plagiarism Facts and Statistics to Analyze Latest Trends

June 4, 2024
Accenture creates a regulatory document authoring solution using AWS generative AI services

Accenture creates a regulatory document authoring solution using AWS generative AI services

February 6, 2024
Managing PDFs in Node.js with pdf-lib

Managing PDFs in Node.js with pdf-lib

November 16, 2023
Graph neural networks in TensorFlow – Google Research Blog

Graph neural networks in TensorFlow – Google Research Blog

February 6, 2024
13 Best Books, Courses and Communities for Learning React — SitePoint

13 Best Books, Courses and Communities for Learning React — SitePoint

February 4, 2024
From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset

From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset

June 10, 2024
Can You Guess What Percentage Of Their Wealth The Rich Keep In Cash?

Can You Guess What Percentage Of Their Wealth The Rich Keep In Cash?

June 10, 2024
AI Compared: Which Assistant Is the Best?

AI Compared: Which Assistant Is the Best?

June 10, 2024
How insurance companies can use synthetic data to fight bias

How insurance companies can use synthetic data to fight bias

June 10, 2024
5 SLA metrics you should be monitoring

5 SLA metrics you should be monitoring

June 10, 2024
From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset

From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset

June 10, 2024
UGRO Capital: Targeting to hit milestone of Rs 20,000 cr loan book in 8-10 quarters: Shachindra Nath

UGRO Capital: Targeting to hit milestone of Rs 20,000 cr loan book in 8-10 quarters: Shachindra Nath

June 10, 2024
Facebook Twitter LinkedIn Pinterest RSS
News PouroverAI

The latest news and updates about the AI Technology and Latest Tech Updates around the world... PouroverAI keeps you in the loop.

CATEGORIES

  • AI Technology
  • Automation
  • Blockchain
  • Business
  • Cloud & Programming
  • Data Science & ML
  • Digital Marketing
  • Front-Tech
  • Uncategorized

SITEMAP

  • Disclaimer
  • Privacy Policy
  • DMCA
  • Cookie Privacy Policy
  • Terms and Conditions
  • Contact us

Copyright © 2023 PouroverAI News.
PouroverAI News

No Result
View All Result
  • Home
  • AI Tech
  • Business
  • Blockchain
  • Data Science & ML
  • Cloud & Programming
  • Automation
  • Front-Tech
  • Marketing

Copyright © 2023 PouroverAI News.
PouroverAI News

Welcome Back!

Login to your account below

Forgotten Password? Sign Up

Create New Account!

Fill the forms bellow to register

All fields are required. Log In

Retrieve your password

Please enter your username or email address to reset your password.

Log In