Language Models (LMs) face challenges in self-supervised learning due to representation degeneration. LMs like BERT or GPT-2 LMs have low angular variability and outlier dimensions on a small scale, comprised of a neural network processing token sequences to generate contextual representations. A language modeling head, typically a linear layer with parameters W, produces next-token probability distributions. Current trends involve scaling up generative pretraining like GPT-2 despite concerns about energy and hardware limitations. Evaluation of the Pythia model suite revealed performance saturation in late pretraining phases when training small models on extensive corpora.
Pythia models, trained on 300B tokens from the Pile, exhibit performance drops in smaller variants during late Lambada dataset training. Scaling laws predict inefficiencies when training compact models on vast corpora, but recent efforts focus on reducing inference costs by training smaller language models on extensive datasets. The softmax bottleneck underscores limitations in models with insufficient hidden dimensions. Representation degeneration in pre-trained models leads to low-entropy singular value distributions, impacting language modeling. Some works connect scaling laws to data dimensionality, using Singular Value Decomposition (SVD) to analyze linear classifiers’ performance limitations.
The researchers from Inria Paris and Sorbonne Universite provide a thorough study to analyse the correlation between saturation and representation degeneration, particularly in the language modeling head of small models. They demonstrated that a linear language modeling head can pose a performance bottleneck for architectures with small hidden dimensions. This bottleneck arises from a mismatch between the hidden dimension of smaller models and the high rank of the target contextual probability distribution, affecting the performance through the softmax bottleneck phenomenon.
The researchers investigated performance saturation in Pythia models across various sizes, confirming saturation up to 410M parameters. Loss saturation shows an increase in in-domain loss during advanced training stages. A scaling law matches data points from models over 410M parameters, revealing optimal parameters (A = 119.09 and α = 0.246). The final checkpoints underperform the extrapolation by approximately 8% on average, while the best checkpoints fall short by about 4% due to incomplete learning rate cooldown.
The key contributions of this research are the following:
Characterizing performance saturation of small language models through evaluation and extrapolation of scaling laws.
Identifying concurrent degeneration of representations in smaller models, particularly rank saturation in LM prediction heads.
Empirically verifying the high rank of the target contextual distribution and the substantial impact of a low-rank linear head on the performance.
Theoretically quantifying the performance limitation induced by LM heads.
Anisotropy, a prevalent representation degeneration in small language models, exhibits reduced angular variability across layers. Measurement of Anisotropy using average cosine similarity indicates its pervasive presence. A correlation between anisotropy and performance saturation is observed in Pythia models. Singular value distributions of language modeling heads highlight spectral saturation patterns that co-occur with performance saturation. Theoretical analysis aims to establish a formal link between contextual distribution dimensionality and the performance bottleneck induced by low-rank heads.
In conclusion, This research investigates performance saturation in small language models, which stems from mapping challenges between low-dimensional output representations and high-rank contextual probability distributions via linear language modeling heads. The paper establishes a theoretical link between this performance gap and spectral properties of contextual probability distributions. Empirical results confirm the mapping’s relatively high rank. Experiments reveal significant performance drops with LM head hidden dimensions below 1000. Analysis correlates saturation with last-layer anisotropy and spectral saturation in small models’ LM heads, advancing understanding of the softmax bottleneck’s impact on language modeling.
Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.
If you like our work, you will love our newsletter.
Don’t Forget to join our 40k+ ML SubReddit
For Content Partnership, Please Fill Out This Form Here.
Asjad is an intern consultant at Marktechpost. He is persuing B.Tech in mechanical engineering at the Indian Institute of Technology, Kharagpur. Asjad is a Machine learning and deep learning enthusiast who is always researching the applications of machine learning in healthcare.
