[TMP-019] An AR-Based Support System for Pancreatic Surgery

Contact person: Hamraz javaheri (hamraz.javaheri@dfki.de) 

Internal Partners:

1. DFKI  

External Partners:

1. Hospital Saarbrücken “Der Winterberg”  

In this project, we successfully implemented and clinically evaluated an AR assistance system for pancreatic surgery, enhancing surgical navigation and achieving more precise perioperative outcomes. However, the system’s reliance on preoperative data posed challenges, particularly due to anatomical deformations occurring in the later stages of surgery. In future research, we aim to address this by integrating real-time data sources to further improve the system’s accuracy and adaptability during surgery.

Results Summary

Throughout our project, we developed and clinically evaluated ARAS, an augmented reality (AR) assistance system designed for pancreatic surgery. The system was clinically evaluated by field surgeons during pancreatic tumor resections involving 20 patients. In a matched-pair analysis with 60 patients who underwent surgery without ARAS, the ARAS group demonstrated a significantly shorter operation time compared to the control group. Although not statistically significant, the ARAS group also exhibited clinically noticeable lower rates of excessive intraoperative bleeding and reduced need for intraoperative red blood cell (RBC) transfusions. Furthermore, ARAS enabled more precise tumor resections with tumor-free margins, and patients in this group had better postoperative outcomes, including significantly shorter hospital stays. In this project, we published 2 journal papers (1 is accepted and will be published soon), 1 conference paper, 1 demo paper (Best Demo Paper Award), and 2 more conference papers are currently under submission. The success of our project got also several international and local news and media attention including Deutsche Welle news channel (Example links provided)

Tangible Outcomes

1. Beyond the visible: preliminary evaluation of the first wearable augmented reality assistance system for pancreatic surgery, Journal of International Journal of Computer Assisted Radiology and Surgery (https://doi.org/10.1007/s11548-024-03131-0 )
2. Enhancing Perioperative Outcomes of Pancreatic Surgery with Wearable Augmented Reality Assistance System: A Matched-Pair Analysis, Journal of Annals of Surgery Open ( https://doi.org/10.1097/AS9.0000000000000516)
3. Design and Clinical Evaluation of ARAS: An Augmented Reality Assistance System for Pancreatic Surgery (IEEE ISMAR 2024 (https://www.researchgate.net/publication/385116946_Design_and_Clinical_Evaluation_of_ARAS_An_Augmented_Reality_Assistance_System_for_Open_Pancreatic_Surgery_Omid_Ghamarnejad) 
4. ARAS: LLM-Supported Augmented Reality Assistance System for Pancreatic Surgery, ISWC/UbiComp 2024 (https://doi.org/10.1145/3675094.3677543) 
5. Media coverage for the project:
   1. https://www.dw.com/en/artificial-intelligence-saving-lives-in-the-operating-room/video-68125878
   2. https://www.dw.com/de/k%C3%BCnstliche-intelligenz-im-op-saal-rettet-leben/video-68125903
   3. https://www.saarbruecker-zeitung.de/app/consent/?ref=https%3A%2F%2Fwww.saarbruecker-zeitung.de%2Fsaarland%2Fsaarbruecken%2Fsaarbruecken%2Fsaarbruecken-winterberg-klinik-international-im-tv-zu-sehen_aid-106311259
   4. https://www.saarbruecker-zeitung.de/app/consent/?ref=https%3A%2F%2Fwww.saarbruecker-zeitung.de%2Fsaarland%2Fsaarbruecken-mittels-ki-erfolgreiche-operation-an-82-jaehriger-v29_aid-104053203
   5. https://m.focus.de/gesundheit/gesundleben/da-gibt-es-keinen-raum-fuer-fehler-kuenstliche-intelligenz-im-op-saal-rettet-leben_id_259629806.html