[TMP-081] Multi-layer evaluation sets for speech translation of web-based meetings

Contact person: Mārcis Pinnis (marcis.pinnis@tilde.lv)

Internal Partners:

1. Tilde
2. Charles University

Within the scope of the project, we have created evaluation and development data sets for speech translation for meetings (for English->Latvian, Latvian->English, and Lithuanian- >English) (http://hdl.handle.net/20.500.12574/74),. We specifically contributed speech translation evaluation and development data sets for English->Latvian (4 hours and 40 minutes), Latvian->English (4 hours and 52 minutes), and Lithuanian->English (3 hours and 31 minutes). Additionally, we created an automatic minuting test set for the AutoMin 2023 shared task on automatic creation of meeting summaries (“minutes”) for English and Czech (https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-4692). We specifically created for Task A: 10 English + 10 Czech meeting transcripts with

manually created reference minutes and for Task D: manually created alignments between AutoMin 2021 system outputs and transcripts.We also contributed to the human evaluation of machine translation systems for the Seventh Conference on Machine Translation (WMT) for the English->Czech translation direction.The microproject produced data that will be beneficial for future developments of speech translation and automatic minuting systems. The data will allow us to better understand capabilities of these systems and also identify potential areas of improvement. The results will contribute towards developing robust and trustworthy AI systems. Furthermore, the project contributed towards mobilization of the research landscape by involving a European research institution (Charles University) and an industry partner (Tilde).

Results Summary

We show that the proposed approach provides high-quality semantic segmentation from the robot’s perspective, with accuracy comparable to the original one. In addition, we exploited the gained information and improved the recognition performance of the deep network for the lower viewpoints and showed that the small robot alone is capable of generating high-quality semantic maps for the human partner. The computations are close to real time, so the approach enables interactive applications.

Tangible Outcomes

1. evaluation and development data sets for speech translation for meetings (for English->Latvian, Latvian->English, and Lithuanian->English) (http://hdl.handle.net/20.500.12574/74 )
2. ELITIR minuting cortpus: an automatic minuting test set for the AutoMin 2023 shared task on automatic creation of meeting summaries (“minutes”) for English and Czech (https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-4692 )