A Family of Pretrained Transformer Language Models for Russian

Авторы: Zmitrovich Dmitry, Abramov Aleksandr, Kalmykov Andrey, Kadulin Vitaly, Tikhonova Maria, Taktasheva Ekaterina, Astafurov Danil, Baushenko Mark, Snegirev Artem, Shavrina Tatiana, Markov Sergei S., Mikhailov Vladislav, Fenogenova Alena
Сборник: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Год издания: 2024
Место издания: ELRA and ICCL Torino, Italia
Первая страница: 507
Последняя страница: 524
Аннотация: Transformer language models (LMs) are fundamental to NLP research methodologies and applications in various languages. However, developing such models specifically for the Russian language has received little attention. This paper introduces a collection of 13 Russian Transformer LMs, which spans encoder (ruBERT, ruRoBERTa, ruELECTRA), decoder (ruGPT-3), and encoder-decoder (ruT5, FRED-T5) architectures. We provide a report on the model architecture design and pretraining, and the results of evaluating their generalization abilities on Russian language understanding and generation datasets and benchmarks. By pretraining and releasing these specialized Transformer LMs, we aim to broaden the scope of the NLP research directions and enable the development of industrial solutions for the Russian language.
Добавил в систему: Шаврина Татьяна Олеговна

	ИСТИНА	Войти в систему Регистрация
	Интеллектуальная Система Тематического Исследования НАукометрических данных
	Главная Поиск Статистика О проекте Помощь

ИСТИНА