Meet GlotLID: An Open-Source Language Identification (LID) Model that Supports 1665 Languages

In recent times, when communication across national boundaries is constantly growing, linguistic inclusion is critical. Natural language processing (NLP) technology should be accessible to a wide range of linguistic varieties rather than just a few chosen medium and high-resource languages. Access to corpora, i.e., linguistic data collections for low-resource languages, is crucial for achieving this. Promoting linguistic variety and guaranteeing that NLP technology may help people worldwide depend on this inclusion.

There have been tremendous advancements in the field of Language Identification (LID), especially for the roughly 300 high and medium-resource languages. Several studies have suggested LID systems that work well for various languages. But there are a number of issues with it, which are as follows.

No LID system currently exists that supports a wide variety of low-resource languages, which are essential for linguistic diversity and inclusivity.

The current LID models for low-resource languages don’t provide a thorough assessment and dependability. Ensuring that the system can accurately recognise languages in a variety of circumstances is crucial.

One of the main concerns with LID systems is their usability, i.e., user-friendliness and effectiveness. 

To overcome these challenges, a team of researchers has introduced GlotLID-M, a unique Language Identification model. With a remarkable identification capacity of 1665 languages, GlotLID-M provides a significant improvement in coverage over earlier research. It is a big step towards enabling a wider range of languages and cultures to use NLP technology. A number of difficulties have been addressed in the context of low-resource LID, which has been overcome by this new approach.

Inaccurate Corpus Metadata: Inaccurate or inadequate linguistic data is a common problem for low-resource languages, which has been accommodated by GlotLID-M while maintaining accurate identification.

Leakage from High-Resource Languages: GlotLID-M has addressed the problem of low-resource languages getting occasionally mistakenly associated with linguistic traits from high-resource languages.

Difficulty Distinguishing Closely Related Languages: Dialects and closely related variants can be found in low-resource languages. GlotLID-M has provided a more accurate identification by differentiating between them.

Macrolanguage vs. Varieties Handling: Dialects and other variations are frequently included in macrolanguages. Within a macro language, GlotLID-M has been made capable of effectively identifying these changes.

Handling Noisy Data: GlotLID-M works well with handling noisy data, as working with low-resource linguistic data can be difficult and noisy at times. 

The team has shared that upon evaluation, GlotLID-M has demonstrated better performance than four baseline LID models, which are CLD3, FT176, OpenLID, and NLLB, when accuracy-based F1 score and false positive rate were balanced. This proves that it can consistently recognise languages accurately, even in difficult situations. GlotLID-M has been created with usability and efficiency and can be easily incorporated into pipelines for creating datasets.

The team has shared their primary contributions as follows.

GlotLID-C has been created, which is an extensive dataset that encompasses 1665 languages and is notable for its inclusivity, with a focus on low-resource languages across diverse domains.

GlotLID-M, an open-source Language Identification model, has been trained on the GlotLID-C dataset. This model is capable of identifying languages among the 1665 languages in the dataset, making it a powerful tool for language recognition across a wide linguistic spectrum.

GlotLID-M has outperformed multiple baseline models, demonstrating its efficacy. Compared to low-resource languages, it achieves a notable improvement of over 12% absolute F1 score on the Universal Declaration of Human Rights (UDHR) corpus.

When it comes to balancing F1 scores and false positive rates (FPR), GlotLID-M also performs exceptionally well. The FLORES-200 dataset, which mostly comprises high- and medium-resource languages, performs better than baseline models.

Check out the Paper, Project, and Github. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 32k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..

We are also on Telegram and WhatsApp.