Enhancing Zero-shot Cross-Lingual Transfer with Multi-Source Training

Abstract

The training process for multilingual language models is highly influenced by the availability of language-specific training data for the supported languages. However, for languages with limited resources, it is challenging to obtain enough training data for model training, resulting in poor performance in multilingual language models. To mitigate this issue, the concept of cross- lingual transfer, which assists the learning of a target language through information from a source language with abundant data, has been introduced. Cross-lingual transfer methods are implemented through various ideas, but most of them share the common feature of reducing the gap between languages. In this thesis, we propose a cross-lingual transfer method called Multi-Source Training (MST), which aims to increase the diversity of source languages to reduce the gap between languages and improve classification performance. Experimental results demonstrate that the MST method, which enhances the diversity of source languages for cross-lingual transfer, shapes representations for each language similarly and improves classification performance, indicating that MST method successfully reduces gap between languages. Additionally, indiscriminately combining source languages can lead to a decrease in performance. To address this, this thesis provides a meaningful benchmark for identifying efficient combinations to enhance cross-lingual transfer performance, making the MST method a more effective approach.