Artificial neural networks (ANNs) exhibit a consistent pattern when trained on natural data, regardless of the initial conditions, dataset, or training objectives. Models trained on the same data domain tend to converge to similar learned patterns. For instance, in various image models, the initial layer weights often converge to Gabor filters and color-contrast detectors. These features suggest a global representation that transcends both biological and artificial systems, and they are also observed in the visual cortex. While these findings are well-established and practical in the field of literature interpretation by machines, they lack theoretical explanations.
The most commonly observed universal features in image models are localized versions of canonical 2D Fourier basis functions, such as Gabor filters or wavelets. When vision models are trained on tasks like efficient coding, classification, temporal coherence, and next-step prediction, these Fourier features emerge in the initial layers of the model. Additionally, non-localized Fourier features have been noted in networks trained for tasks where cyclic wraparound is permitted, such as modular arithmetic or invariance to cyclic translations.
Researchers from KTH, Redwood Center for Theoretical Neuroscience, and UC Santa Barbara have introduced a mathematical explanation for the prevalence of Fourier features in learning systems like neural networks. This prevalence stems from the downstream invariance of the learner, which becomes insensitive to certain transformations like planar translation or rotation. The team has derived theoretical guarantees regarding Fourier features in invariant learners that can be applied to various machine-learning models. This derivation is based on the concept that invariance is a fundamental bias that can be implicitly or explicitly injected into learning systems due to the symmetries present in natural data.
The standard discrete Fourier transform is a specific case of more general Fourier transforms on groups, where the basis of harmonics is replaced with different unitary group representations. Previous theoretical works have focused on sparse coding models, establishing conditions under which sparse linear combinations can recover the original bases generating the data through a network. This proposed theory covers different scenarios and neural network architectures, laying the groundwork for a learning theory of representations in artificial and biological neural systems.
The team presents two informal theorems in this paper. The first theorem suggests that if a parametric function of a certain kind is invariant to the input variable under the action of a finite group G, then each component of its weights W corresponds to a harmonic of G up to a linear transformation. The second theorem states that if a parametric function is nearly invariant to G within certain functional bounds and the weights are orthonormal, then the multiplicative table of G can be deduced from W. Furthermore, a model is implemented to align with the proposed theory and trained through various learning approaches to support invariance and extraction of the multiplicative table of G from its weights.
In conclusion, researchers have provided a mathematical explanation for the emergence of Fourier features in learning systems like neural networks. They have demonstrated that if a machine learning model is invariant to a finite group, its weights are closely related to the Fourier transform on that group, and the algebraic structure of an unknown group can be recovered from an invariant model. Future work will explore analogs of the proposed theory on real numbers, aligning more closely with current practices in the field.
For more details, refer to the Paper. Credit for this research goes to the project’s researchers. Follow us on Twitter, join our Telegram Channel, Discord Channel, and LinkedIn Group.
If you appreciate our work, you’ll love our newsletter. Don’t forget to join our 42k+ ML SubReddit.
Sajjad Ansari is a final year undergraduate from IIT Kharagpur. As a Tech enthusiast, he delves into the practical applications of AI with a focus on understanding the impact of AI technologies and their real-world implications. He aims to articulate complex AI concepts in a clear and accessible manner.
