GlotLID

Description

GlotLID is a Fasttext language identification (LID) model that supports more than 2000 labels.

Stable: GlotLID v4 (model.bin, the same file as model_v4.bin) supports 2060 labels (1902 three-letter ISO codes with script, with the "zxx" and "und" series). GlotLID v4 small (model_v4_small.bin) has the same labels and is about 4 times faster. For the supported languages, performance and the changes from v3, see languages-v4.md; for all versions, versions.

Previous: GlotLID v3 (model_v3.bin) supports 2102 labels. For its supported languages and performance, see languages-v3.md.

• Demo: huggingface (language identification, and code-switching with MaskLID)
• Repository: github
• Paper: paper (EMNLP 2023)
• Point of Contact: amir@cis.lmu.de

How to use

Here is how to use this model to detect the language of a given text:

# pip install fasttext-turbo huggingface_hub
>>> import fasttext
>>> from huggingface_hub import hf_hub_download

# model.bin is the latest version always
>>> model_path = hf_hub_download(repo_id="cis-lmu/glotlid", filename="model.bin")
>>> model = fasttext.load_model(model_path)
>>> model.predict("Hello, world!")

We recommend fastText-turbo (pip install fasttext-turbo): it is a drop-in replacement for fastText (same import fasttext, same model files and API, prebuilt wheels) that predicts with GlotLID about 3 times faster than the original fastText in our tests (2.7x with v4, 3.4-4.3x with v3), with the same predictions. The original fasttext package works too.

For the faster small model, download model_v4_small.bin (or the quantized .ftz versions); for v3, model_v3.bin.

To limit the prediction to a smaller set of languages without training another model, use CustomLID (see the GitHub README).

If you are not a fan of huggingface_hub, then download the model directly:

>>> ! wget https://huggingface.co/cis-lmu/glotlid/resolve/main/model.bin

>>> import fasttext

>>> model = fasttext.load_model("/path/to/model.bin")
>>> model.predict("Hello, world!")

License

The model is distributed under the Apache License, Version 2.0 plus notices (see LICENSE file for full terms).

Version

We always maintain the previous version of GlotLID in our repository.

To access a specific version, simply append the version number to the filename.

• For v1: model_v1.bin (introduced in the GlotLID paper and used in all experiments).
• For v2: model_v2.bin (an edited version of v1, featuring more languages, and cleaned from noisy corpora based on the analysis of v1).
• For v3: model_v3.bin (an edited version of v2, featuring more languages, excluding macro languages, further cleaned from noisy corpora and incorrect metadata labels based on the analysis of v2, supporting "zxx" and "und" series labels)
• For v4: model_v4.bin (an edited version of v3: labels kept only where they work on independent benchmarks, so some v3 labels were merged or removed, further cleaned corpora, and trained on random short spans and normalized lines, which makes it markedly better on short text and calibrated). It supports 2060 labels (1902 three-letter ISO codes, with the "zxx" and "und" series). Trained with fastText-turbo; it loads in fastText and fastText-turbo like the previous versions. It is smaller (96-dimensional, 783 MB) and uses character n-grams of 2 to 5 with word n-grams of 1, so nothing is shared between neighbouring words: the scores of a text add up from the scores of its words, which makes the per-word scores that MaskLID uses for code-switching exact. model_v4.ftz is its quantized version (101 MB; the same top label for 99.8% of lines).
• For v4 small: model_v4_small.bin (the same 2060 labels in a 16-dimensional model with hierarchical softmax, 131 MB). It predicts about 4 times faster than v4 on one CPU thread (about 43,000 lines per second against 10,000), faster than CLD2 and close to fastText's lid.176, and is more accurate than both on their own languages; it is less accurate than v4, mostly between closely related languages. model_v4_small.ftz is its quantized version (35 MB; the same top label for 99.5% of lines).

model.bin always refers to the latest stable version (v4).

References

If you use this model, please cite the following paper:

@inproceedings{
  kargaran2023glotlid,
  title={{GlotLID}: Language Identification for Low-Resource Languages},
  author={Kargaran, Amir Hossein and Imani, Ayyoob and Yvon, Fran{\c{c}}ois and Sch{\"u}tze, Hinrich},
  booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},
  year={2023},
  url={https://openreview.net/forum?id=dl4e3EBz5j}
}