Researchers from ISTA Austria and Neural Magic Introduce QMoE: A Revolutionary Compression Framework for Efficient Execution of Trillion-Parameter Language Models

A neural network model designed to combine the output of multiple expert subnetworks to make predictions or decisions is called Mixture of Experts ( MoE ). This architecture is particularly useful when dealing with complex and diverse data, where different subsets or aspects of the data may require specialized models to handle effectively. MoE models are often more robust to outliers or noise in the data because they can learn to ignore the output of experts who perform poorly on certain inputs.

The computational cost of a MoE architecture can vary significantly depending on the model’s specific design, the complexity of the task it’s addressing, and the hardware used for training and inference. MoE architectures can be computationally more expensive than traditional neural networks, especially involving many experts and complex gating mechanisms. For example, the Switch Transformer-c2048 model has 1.6 trillion parameters, which require 3.2 TB of accelerator memory to run efficiently, which makes it challenging and expensive.

Researchers present a solution to this memory problem in a new framework called QMoE. It consists of a scalable algorithm that accurately compresses trillion parameter MoEs to less than 1 bit per parameter. QMoE can compress the 1.6 trillion parameters of the SwitchTransformer-c2048 model to less than 160 GB, which can be processed in less than a day on a single GPU. This is the first time accurate sub-1-bit compression of trillion parameters MoEs is feasible and can be achieved via affordable retraining-free compression techniques.

This is typically achieved by creating copies of certain model components, each responsible for processing only a subset of all input tokens. A router layer generally decides the corresponding input-to-component assignments. Quantization is the method that is currently used for reducing the model size and corresponding model weights to lower numerical precision. However, some MoEs are so large that reduction rates significantly higher than four times would be required to render them practical. Quantizing models to extremely low precision requires more sophisticated data-dependent methods.

Instead of training a neural network with full-precision (32-bit or 16-bit) weights and activations, data-dependent quantization methods train the model with quantized weights and activations. This helps the model learn to adapt to the limitations of lower-precision numerical representations. Popular frameworks and tools for data-dependent quantization include TensorFlow, PyTorch, and TensorRT, which provide built-in support for quantization-aware training and calibration.

Researchers have only considered the decoding operations and encoding matrices with reasonable efficiency. They plan to focus on the direct compression of the pretrained base model. In the future, their work will include finetuning a compressed model for specialized downstream tasks.

Check out the Paper and Github. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 32k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..

We are also on Telegram and WhatsApp.