A Pre-trained Language Model for Chinese Pinyin-to-character Task base on BERT

Abstract

In academia, the research on pre-trained language models (PLMs) has become a very hot topic. In recent years, many articles that can change the entire NLP filed have been proposed. PLMs technology represented by BERT model has gradually developed into an indispensable mainstream technology in the NLP field. Chinese Pinyin-to-character task is an important task in the field of natural language processing. The language model built on this task can provide a robust language backend for Chinese automatic speech recognition and improve the performance of the overall Chinese speech recognition task. It also has important applications in smart input methods, brain-computer interfaces, and other Chinese NLP tasks. We want to make use of the powerful language representation extraction capability of the pre-trained language model in large-scale corpus. Hence， In this paper, we optimized the BERT model and proposed a novel BERT-P2C model which embedding the Chinese word vectors and character vectors as joint input to get both fine-grained and coarse-grained representations of Chinese text. And then integrated the output of optimized BERT into Transformer encoder-decoder model in fine-tuning stage. At the same time, we also tried and made a breakthrough in how to use BERT to fine-tune downstream tasks. The results show that BERT-P2C model has a significant improvement over other non-pretrained models and the original BERT. And reached state-of-the-art (SOTA) result in Chinese Pinyin-to-character task.