Multilingual Language Models Representations and Fine-Tuning

This study evaluates multilingual representation spaces using XGLM-564M and GPT-2 on the FLORES-200 dataset, focusing on English, Spanish, German, Arabic, Tamil, and Quechua. We analyze hidden representations using PCA with scikit-learn and t-SNE with openTSNE to visualize and interpret these high-dimensional spaces. We then fine-tune XGLM-564M on the Monolingual-Quechua-IIC corpus, comparing four approaches: full fine-tuning, BitFit [Zaken et al., 2022], LoRA [Hu et al., 2021], and IA³ [Liu et al., 2022]. Our analysis examines both performance improvements on Quechua and cross-lingual transfer effects.

By: Camilo Martínez, Honglu Ma

View on GitHub

We evaluated the models: XGLM-564M, a multilingual autoregressive language model (with 564 million parameters) trained on a balanced corpus of a diverse set of 30 languages totaling 500 billion sub-tokens, and GPT-2, a transformers model pretrained on a very large corpus of English data in a self-supervised fashion. As evaluation dataset, we used the famous FLORES-200 dataset, available on HuggingFace, specifically on six languages: English, Spanish, German, Arabic, Tamil, and Quechua.

We analyzed the multilingual embeddings (both sentence-level and token-level) from both pre-trained language models using dimensionality reduction techniques: PCA with scikit-learn and t-SNE with openTSNE.

Finally, we finetuned the XGLM-564M model on a specific language: Quechua with a dataset the model hadn't seen before, Monolingual-Quechua-IIC, a monolingual corpus of Southern Quechua, consisting of nearly 450K segments [Zevallos et al., 2022]. We used different fine-tuning methods: full fine-tuning, BitFit [Zaken et al., 2022], LoRA [Hu et al., 2021], and IA³ [Liu et al., 2022] and analyzed their performance and evaluation loss on the six languages mentioned above to see how much the performance on the Quechua language improved, and whether it decreased for the rest.

Experiments and Analyses

We first compared the performance of the XGLM-564M model with GPT-2, in terms of Mean Language Modeling Loss.

Loss of XGLM-564M compared to GPT-2 — **Figure 1:** Loss of the XGLM-564M model compared to GPT-2, on the original languages. Both struggle with `quy_Latn` (Quechua).

Afterwards, we visualized the hidden representations of both the XGLM-564M model using PCA and t-SNE for both sentence-level and token-level embeddings across all layers of the model. Below are visualizations from our experiments, which clearly show the progression of how the model learns to better separate the languages as we move to deeper layers:

t-SNE Visualization of Sentences in Layer 24 — **Figure 2:** t-SNE Visualization of Sentences for Layer 0.

PCA Visualization of Sentences in Layer 24 — **Figure 2:** t-SNE Visualization of Sentences for Layer 0.

t-SNE Visualization of Tokens in Layer 24 — **Figure 4:** t-SNE Visualization of Tokens for Layer 0.

PCA Visualization of Tokens in Layer 24 — **Figure 4:** t-SNE Visualization of Tokens for Layer 0.

Finally, the results for the finetuning of the XGLM-564M model on the Monolingual-Quechua-IIC dataset were the following:

Loss of XGLM-564M vs Fine-tuned Versions — **Figure 6:** Loss of the XGLM-564M model compared to its finetuned versions (FFT meaning Full Fine-Tuning).

**Figure 7:** Performance metrics comparison of finetuning methods in training and validation.

Conclusion

This project provides insights into multilingual representation spaces (in sentence and token-level) for the XGLM-564M model on an underrepresented language, e.g., Quechua, and demonstrates the effectiveness of several fine-tuning techniques. While full fine-tuning offers the best performance, methods such as LoRA, BitFit and IA³ offer practical alternatives under computational constraints, such as our case.

For more details, please refer to the project's GitHub repository.