Don t Use English Dev On the Zero Shot Cross Lingual Evaluation of Contextual Embeddings

Multilingual contextual embeddings have demonstrated state-of-the-art performance in zero-shot cross-lingual transfer learning . English dev accuracy is often uncorrelated (or even anti-correlated) with target language accuracy . We recommend providing oracle scores alongside zero shot results to make results more consistent by avoiding arbitrarily bad checkpoints . These reproducibility issues are also present for other tasks with different pre-trained embeddeddings (e.g., MLQA with XLM-R), such as MLQ a with XLQA and MLN-R . Reportability of these tasks makes it difficult to obtain reproducible results on the MLDoc and XNLI tasks, and we recommend providing Oracle scores with oracle results alongside zero shoot results . We also recommend providing .

Links: PDF - Abstract

Code :


Keywords : results - tasks - shot - oracle - recommend -

Leave a Reply

Your email address will not be published. Required fields are marked *