TAPE: Assessing Few-shot Russian Language Understanding

Taktasheva, E.; Shavrina, T.; Fenogenova, A.; Shevelev, D.; Katricheva, N.; Tikhonova, M.; Akhmetgareeva, A.; Zinkevich, O.; Bashmakova, A.; Iordanskaia, S.; Spiridonova, A.; Kurenshchikova, V.; Artemova, E.; Mikhailov, V.

Авторы: Taktasheva Ekaterina, Shavrina Tatiana, Fenogenova Alena, Shevelev Denis, Katricheva Nadezhda, Tikhonova Maria, Akhmetgareeva Albina, Zinkevich Oleg, Bashmakova Anastasiia, Iordanskaia Svetlana, Spiridonova Alena, Kurenshchikova Valentina, Artemova Ekaterina, Mikhailov Vladislav
Сборник: Findings of the Association for Computational Linguistics: EMNLP 2022
Год издания: 2022
Место издания: Association for Computational Linguistics Abu Dhabi, United Arab Emirates
Первая страница: 2472
Последняя страница: 2497
DOI: 10.18653/v1/2022.findings-emnlp.183
Аннотация: Recent advances in zero-shot and few-shot learning have shown promise for a scope of research and practical purposes. However, this fast-growing area lacks standardized evaluation suites for non-English languages, hindering progress outside the Anglo-centric paradigm. To address this line of research, we propose TAPE (Text Attack and Perturbation Evaluation), a novel benchmark that includes six more complex NLU tasks for Russian, covering multi-hop reasoning, ethical concepts, logic and commonsense knowledge. The TAPE’s design focuses on systematic zero-shot and few-shot NLU evaluation: (i) linguistic-oriented adversarial attacks and perturbations for analyzing robustness, and (ii) subpopulations for nuanced interpretation. The detailed analysis of testing the autoregressive baselines indicates that simple spelling-based perturbations affect the performance the most, while paraphrasing the input has a more negligible effect. At the same time, the results demonstrate a significant gap between the neural and human baselines for most tasks. We publicly release TAPE (https://tape-benchmark.com) to foster research on robust LMs that can generalize to new tasks when little to no supervision is available.
Добавил в систему: Шаврина Татьяна Олеговна

	ИСТИНА	Войти в систему Регистрация
	Интеллектуальная Система Тематического Исследования НАукометрических данных
	Главная Поиск Статистика О проекте Помощь

ИСТИНА

Интеллектуальная Система Тематического Исследования НАукометрических данных

TAPE: Assessing Few-shot Russian Language Understandingстатья