[TMP-124] Trustworthy Voting Advice Applications

Contact person: Elisabeth Stockinger (elisabeth.stockinger@gess.ethz.ch )

Internal Partners:

1. ETHZ, Elisabeth Stockinger, elisabeth.stockinger@gess.ethz.ch
2. UMU, Virginia Dignum, virginia@cs.umu.se
3. TU Delft, Jonne Maas, J.J.C.Maas@tudelft.nl

External Partners:

1. University of Amsterdam Christopher Talvitie, christalvitie@gmail.com

Voting Advice Applications (VAAs) are increasingly popular throughout Europe. While commonly portrayed as impartial tools to measure issue agreement, their developers must take several design decisions at each step of the design process. Such decisions may include the selection of issues to incorporate into a questionnaire, the placement of candidates or parties on a political spectrum, or the algorithm measuring the distance between user and candidate. These decisions have to be made with great care, as they can cause substantial differences in the resulting list of recommendations.

As there is no known ground truth by which to measure different VAA designs, it is imperative that their design follows guidelines and best practices of pro-ethical design. Similarly, as VAAs aim to directly inform voter decisions in a democratic election, users must be able to trust the fulfillment of these guidelines based on the information available to them.

Results Summary

Firstly, we conduct an ethics assessment of several VAAs used in European countries, representing different design strategies. This assessment focuses on trustworthiness in the eyes of the electorate, focusing on user-centric documentation. By using the Ethics Guidelines for Trustworthy AI, we refer to a framework that is acknowledged by the democratic institutions of the countries hosting the VAAs and the respective elections, contributing to the democratic validity of a normative analysis of tools embedded in electoral processes.

Secondly, we identify the abstract criteria that a trustworthy VAA must fulfil according to the EGTAI, and accordingly evaluate a representative set of VAAs within Europe (StemWijzer, Kieskompas What2Vote, Smartvote, Wahl-O-Mat, Aftonbladets valkompass, HS Vaalikone and SVT Nyheters valkompass). None of the VAAs under investigation scored highly on the adapted EGTAI assessment list. For several requirements, many sub-requirements are not fulfilled by any VAA in this study. In particular, scores on societal and environmental well-being (R6) or accountability (R7) are low without significant differences between VAAs.

Thirdly, we present a list of recommendations based on these issues to contribute to future VAA development efforts. Across VAAs, we identify the need for improvement in (i) transparency regarding the subjectivity of recommendations, (ii) diversity of stakeholder participation, (iii) user-centric documentation of algorithm, and (iv) disclosure of the underlying values and assumptions.

Tangible Outcomes

1. Stockinger, E., Maas, J., Talvitie, C. et al. Trustworthiness of voting advice applications in Europe. Ethics Inf Technol 26, 55 (2024). https://doi.org/10.1007/s10676-024-09790-6 
2. Dataset showing the evaluated VAAs and the frameworks used to evaluate them https://static-content.springer.com/esm/art%3A10.1007%2Fs10676-024-09790-6/MediaObjects/10676_2024_9790_MOESM1_ESM.pdf 
3. The code used for the analysis: https://github.com/ethz-coss/vaa-egtai-compliance 
4. A video explaining Robust and Value-Based Political Guidance to the general public https://www.youtube.com/watch?v=5riTfDuRTlk&ab_channel=ComputationalSocialScienceETH
5. The project was presented at:

1. 1. the Digital Democracy Workshop (2023, http://digdemlab.io/event/wk2023 )
   2. the Workshop on Co-Creating the Future: Participatory Cities and Digital Governance (2023, http://www.participatorycities.net )

1. 3. the 1st Twin Workshop on Ethics of Smart Cities and Smart Societies (2023, http://coss.ethz.ch/research/CoCi/TwinWorkshop )
   4. the HumanE AI Conference (2022, http://www.humane-ai.eu/event/humane-ai-conference ).