Deep Learning and Explainable AI Methods for Surgical Skills Assessment

In this thesis we introduce and analyze a new annotated and multi-modal dataset of a surgical knot-tying task. We also design and develop three deep learning models, including a multi-modal model leveraging both image and time-series kinematic data that demonstrates performance comparable to expert human raters. Further, we try to open the ”black-box” of the AI models, and investigate the important features in the dataset the AI uses to make its predictions. Using a saliency-map based approach, we find that the image-based features are similar to the features used by human evaluators. As objective assessment of technical skill continues to be a growing, but resource-heavy, element of surgical education, this study is an important step towards automated surgical skill assessment, ultimately leading to reduced burden on training faculty and institutes.