Update README.md

b24209a verified 12 months ago

7.37 kB

---
pipeline_tag: token-classification
tags:
- morphology
- udmurt
- low-resource
- morphology-tagging
- first-subtoken-only
license: apache-2.0
language:
- udm
base_model: cis-lmu/glot500-base
model_name: bert-morph-tagger-udmurt
datasets:
- tsakorpus-udmurt
- lingvodoc-udmurt
references:
  - https://aclanthology.org/2025.xllm-1.9/
model-index:
- name: bert-morph-tagger-udmurt
  results:
  - task:
      type: token-classification
      name: Morphological tagging (first subtoken)
    dataset:
      name: Udmurt Morphological Dataset (Tsakorpus + LingvoDoc)
      type: custom
      split: test
    metrics:
    - type: accuracy
      value: 0.9325
      name: Token Accuracy (all)
    - type: accuracy
      value: 0.857
      name: Token Accuracy (ambiguous)
---
# Udmurt Morphological Tagger (BERT-based, two-stage fine-tuning)

This model performs **morphological tagging for the Udmurt language**.  
It was trained using a **two-stage fine-tuning procedure** combining *automatically ambiguously annotated* and *manually disambiguated* data (see [paper](https://aclanthology.org/2025.xllm-1.9.pdf) for the details).

The model reached **93.25 % token accuracy** (85.7 % on ambiguous tokens) and is released together with training code.

---

## Model Overview

- **Task:** Morphological tagging (token classification)
- **Architecture:** AutoModelForTokenClassification
- **Base model:** [cis-lmu/glot500-base](https://huggingface.co/cis-lmu/glot500-base)
- **Languages:** Udmurt
- **Label format:** concatenation of POS + morphological features (e.g., `N,nom,sg`, `V,pst,3pl`), inherited from the training data
- **Special handling:** labels assigned **only to the first subtoken** of each word; other subtokens masked during training loss computation.

---

## Methodology

### Two-Stage Fine-Tuning Pipeline

1. **Pre-Fine-Tuning (PFT):**  
   Uses *automatically ambiguously labeled* data (AML) — texts annotated by a rule-based analyzer without contextual disambiguation.  
   Introduces a **modified multi-label cross-entropy (MLCE)** loss, allowing multiple pseudo-correct labels for ambiguous tokens.  

2. **Task Fine-Tuning (FT):**  
   Uses *manually disambiguated* labeled data (MDL) to teach precise selection within homonymous tag groups.

3. **Vocabulary Adaptation (VA):**  
   Applies **Vocabulary Initialization with Partial Inheritance** (Samenko et al., 2021) to better match Udmurt subword segmentation.  
   Optimal vocabulary size: 32 K WordPiece tokens (fertility ≈ 1.18). [Not used in this model, but was improving accuracy in other training setups with backbone models without Udmurt pre-training]

---

## Data

| Dataset | Type | Description | Volume |
|----------|------|-------------|---------|
| **Train-AML** | Automatically labeled | [Udmurt corpus](http://web-corpora.net/UdmurtCorpus/search/) (Arkhangelskiy 2019) annotated with [uniparser-grammar-udm](https://github.com/timarkh/uniparser-grammar-udm/); includes ambiguous labels | 558 K tokens / 64 K sentences |
| **Train-MDL / Valid-MDL / Test-MDL** | Manually disambiguated | The [Udmurt corpus of **LingvoDoc**](https://lingvodoc.ispras.ru/corpora_all?language=508%2C43) (Normanskaja et al. 2022), 80/10/10 split | 100 K tokens / 12 K sentences |

---

## Evaluation Results

| Backbone | Setup | Token Accuracy | Ambiguous Token Acc |
|-----------|--------|----------------|----------------------|
| mBERT | FT | 86.28 % | 77.04 % |
| mBERT | VA + PFT + FT | **91.38 %** | **81.54 %** |
| ruBERT | VA + PFT + FT | 91.24 % | 81.00 % |
| **Glot500-m** | **PFT + FT (this model)** | **93.25 %** | **85.70 %** |

---

## How to Use

Example usage:

```python
from transformers import pipeline
tagger = pipeline(
    task="token-classification",
    model="ulyanaisaeva/bert-morph-tagger-udmurt",
    aggregation_strategy="first"
)
text = "Example sentence."
preds = tagger(text)
```

Batch inference with clean predictions:

```python
from tqdm import tqdm
def morph_tag_batch(sentences, batch_size=16):
    all_results = []
    for i in tqdm(range(0, len(sentences), batch_size)):
        batch = sentences[i:i+batch_size]
        preds = tagger(batch)
        all_results.extend([
            [(p["word"], p["entity_group"]) for p in sent_preds]
            for sent_preds in preds
        ])
    return all_results
result = morph_tag_batch(sentences)
```

---

## Intended Use & Limitations

**Intended Use**
- Morphological tagging for Udmurt texts.  
- Research and educational applications in computational morphology.  
- Transferable methodology for other low-resource languages.

**Limitations**
- Performance validated on one language (Udmurt) only.  
- Accuracy may drop on noisy, code-switched, or non-standard orthography.

**Disclaimer**. For model selection and validation, a subset of the LingvoDoc Udmurt corpus was used. Although this dataset provides high-quality manual disambiguation, it was not the subject of independent verification within this study. Therefore, the released model may implicitly inherit potential inconsistencies or annotation artifacts present in that corpus. The goal of this research was to explore fine-tuning methodology, not to audit or benchmark the dataset itself. Users applying this model for linguistic analysis should be aware that occasional systematic biases might reflect corpus-specific labeling patterns rather than model deficiencies.

---

## Citation

If you use this model or training methodology, please cite:

```
@inproceedings{isaeva-etal-2025-combining,
    title = "Combining Automated and Manual Data for Effective Downstream Fine-Tuning of Transformers for Low-Resource Language Applications",
    author = "Isaeva, Ulyana  and
      Astafurov, Danil  and
      Martynov, Nikita",
    editor = "Fei, Hao  and
      Tu, Kewei  and
      Zhang, Yuhui  and
      Hu, Xiang  and
      Han, Wenjuan  and
      Jia, Zixia  and
      Zheng, Zilong  and
      Cao, Yixin  and
      Zhang, Meishan  and
      Lu, Wei  and
      Siddharth, N.  and
      {\O}vrelid, Lilja  and
      Xue, Nianwen  and
      Zhang, Yue",
    booktitle = "Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025)",
    month = aug,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.xllm-1.9/",
    doi = "10.18653/v1/2025.xllm-1.9",
    pages = "86--90",
    ISBN = "979-8-89176-286-2",
    abstract = "This paper addresses the constraints of down-stream applications of pre-trained language models (PLMs) for low-resource languages. These constraints are pre-train data deficiency preventing a low-resource language from being well represented in a PLM and inaccessibility of high-quality task-specific data annotation that limits task learning. We propose to use automatically labeled texts combined with manually annotated data in a two-stage task fine-tuning approach. The experiments revealed that utilizing such methodology combined with vocabulary adaptation may compensate for the absence of a targeted PLM or the deficiency of manually annotated data. The methodology is validated on the morphological tagging task for the Udmurt language. We publish our best model that achieved 93.25{\%} token accuracy on HuggingFace Hub along with the training code1."
}
```

	---
	pipeline_tag: token-classification
	tags:
	- morphology
	- udmurt
	- low-resource
	- morphology-tagging
	- first-subtoken-only
	license: apache-2.0
	language:
	- udm
	base_model: cis-lmu/glot500-base
	model_name: bert-morph-tagger-udmurt
	datasets:
	- tsakorpus-udmurt
	- lingvodoc-udmurt
	references:
	- https://aclanthology.org/2025.xllm-1.9/
	model-index:
	- name: bert-morph-tagger-udmurt
	results:
	- task:
	type: token-classification
	name: Morphological tagging (first subtoken)
	dataset:
	name: Udmurt Morphological Dataset (Tsakorpus + LingvoDoc)
	type: custom
	split: test
	metrics:
	- type: accuracy
	value: 0.9325
	name: Token Accuracy (all)
	- type: accuracy
	value: 0.857
	name: Token Accuracy (ambiguous)
	---

	# Udmurt Morphological Tagger (BERT-based, two-stage fine-tuning)

	This model performs morphological tagging for the Udmurt language.
	It was trained using a two-stage fine-tuning procedure combining automatically ambiguously annotated and manually disambiguated data (see [paper](https://aclanthology.org/2025.xllm-1.9.pdf) for the details).

	The model reached 93.25 % token accuracy (85.7 % on ambiguous tokens) and is released together with training code.

	---

	## Model Overview

	- Task: Morphological tagging (token classification)
	- Architecture: AutoModelForTokenClassification
	- Base model: [cis-lmu/glot500-base](https://huggingface.co/cis-lmu/glot500-base)
	- Languages: Udmurt
	- Label format: concatenation of POS + morphological features (e.g., `N,nom,sg`, `V,pst,3pl`), inherited from the training data
	- Special handling: labels assigned only to the first subtoken of each word; other subtokens masked during training loss computation.

	---

	## Methodology

	### Two-Stage Fine-Tuning Pipeline

	1. Pre-Fine-Tuning (PFT):
	Uses automatically ambiguously labeled data (AML) — texts annotated by a rule-based analyzer without contextual disambiguation.
	Introduces a modified multi-label cross-entropy (MLCE) loss, allowing multiple pseudo-correct labels for ambiguous tokens.

	2. Task Fine-Tuning (FT):
	Uses manually disambiguated labeled data (MDL) to teach precise selection within homonymous tag groups.

	3. Vocabulary Adaptation (VA):
	Applies Vocabulary Initialization with Partial Inheritance (Samenko et al., 2021) to better match Udmurt subword segmentation.
	Optimal vocabulary size: 32 K WordPiece tokens (fertility ≈ 1.18). [Not used in this model, but was improving accuracy in other training setups with backbone models without Udmurt pre-training]

	---

	## Data

	\| Dataset \| Type \| Description \| Volume \|
	\|----------\|------\|-------------\|---------\|
	\| Train-AML \| Automatically labeled \| [Udmurt corpus](http://web-corpora.net/UdmurtCorpus/search/) (Arkhangelskiy 2019) annotated with [uniparser-grammar-udm](https://github.com/timarkh/uniparser-grammar-udm/); includes ambiguous labels \| 558 K tokens / 64 K sentences \|
	\| Train-MDL / Valid-MDL / Test-MDL \| Manually disambiguated \| The [Udmurt corpus of LingvoDoc](https://lingvodoc.ispras.ru/corpora_all?language=508%2C43) (Normanskaja et al. 2022), 80/10/10 split \| 100 K tokens / 12 K sentences \|

	---

	## Evaluation Results

	\| Backbone \| Setup \| Token Accuracy \| Ambiguous Token Acc \|
	\|-----------\|--------\|----------------\|----------------------\|
	\| mBERT \| FT \| 86.28 % \| 77.04 % \|
	\| mBERT \| VA + PFT + FT \| 91.38 % \| 81.54 % \|
	\| ruBERT \| VA + PFT + FT \| 91.24 % \| 81.00 % \|
	\| Glot500-m \| PFT + FT (this model) \| 93.25 % \| 85.70 % \|

	---

	## How to Use

	Example usage:

	```python
	from transformers import pipeline

	tagger = pipeline(
	task="token-classification",
	model="ulyanaisaeva/bert-morph-tagger-udmurt",
	aggregation_strategy="first"
	)

	text = "Example sentence."
	preds = tagger(text)
	```

	Batch inference with clean predictions:

	```python
	from tqdm import tqdm

	def morph_tag_batch(sentences, batch_size=16):
	all_results = []
	for i in tqdm(range(0, len(sentences), batch_size)):
	batch = sentences[i:i+batch_size]
	preds = tagger(batch)
	all_results.extend([
	[(p["word"], p["entity_group"]) for p in sent_preds]
	for sent_preds in preds
	])
	return all_results

	result = morph_tag_batch(sentences)
	```

	---

	## Intended Use & Limitations

	Intended Use
	- Morphological tagging for Udmurt texts.
	- Research and educational applications in computational morphology.
	- Transferable methodology for other low-resource languages.

	Limitations
	- Performance validated on one language (Udmurt) only.
	- Accuracy may drop on noisy, code-switched, or non-standard orthography.

	Disclaimer. For model selection and validation, a subset of the LingvoDoc Udmurt corpus was used. Although this dataset provides high-quality manual disambiguation, it was not the subject of independent verification within this study. Therefore, the released model may implicitly inherit potential inconsistencies or annotation artifacts present in that corpus. The goal of this research was to explore fine-tuning methodology, not to audit or benchmark the dataset itself. Users applying this model for linguistic analysis should be aware that occasional systematic biases might reflect corpus-specific labeling patterns rather than model deficiencies.

	---

	## Citation

	If you use this model or training methodology, please cite:

	```
	@inproceedings{isaeva-etal-2025-combining,
	title = "Combining Automated and Manual Data for Effective Downstream Fine-Tuning of Transformers for Low-Resource Language Applications",
	author = "Isaeva, Ulyana and
	Astafurov, Danil and
	Martynov, Nikita",
	editor = "Fei, Hao and
	Tu, Kewei and
	Zhang, Yuhui and
	Hu, Xiang and
	Han, Wenjuan and
	Jia, Zixia and
	Zheng, Zilong and
	Cao, Yixin and
	Zhang, Meishan and
	Lu, Wei and
	Siddharth, N. and
	{\O}vrelid, Lilja and
	Xue, Nianwen and
	Zhang, Yue",
	booktitle = "Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025)",
	month = aug,
	year = "2025",
	address = "Vienna, Austria",
	publisher = "Association for Computational Linguistics",
	url = "https://aclanthology.org/2025.xllm-1.9/",
	doi = "10.18653/v1/2025.xllm-1.9",
	pages = "86--90",
	ISBN = "979-8-89176-286-2",
	abstract = "This paper addresses the constraints of down-stream applications of pre-trained language models (PLMs) for low-resource languages. These constraints are pre-train data deficiency preventing a low-resource language from being well represented in a PLM and inaccessibility of high-quality task-specific data annotation that limits task learning. We propose to use automatically labeled texts combined with manually annotated data in a two-stage task fine-tuning approach. The experiments revealed that utilizing such methodology combined with vocabulary adaptation may compensate for the absence of a targeted PLM or the deficiency of manually annotated data. The methodology is validated on the morphological tagging task for the Udmurt language. We publish our best model that achieved 93.25{\%} token accuracy on HuggingFace Hub along with the training code1."
	}
	```