Quantifying similarities between MediaPipe and a known standard for tracking 2D hand trajectories

Abstract

Marker-less motion tracking methods have promise for use in a range of domains, including clinical settings where traditional marker-based systems for human pose estimation is not feasible. MediaPipe is an artificial intelligence-based system that offers a markerless, lightweight approach to motion capture, and encompasses MediaPipe Hands, for recognition of hand landmarks. However, the accuracy of MediaPipe for tracking fine upper limb movements involving the hand has not been explored. Here we aimed to evaluate 2-dimensional accuracy of MediaPipe against a known standard. Participants (N = 10) performed trials in blocks of a touchscreen-based shape-tracing task. Each trial was simultaneously captured by a video camera. Trajectories for each trial were extracted from the touchscreen and compared to those predicted by MediaPipe. Specifically, following re-sampling, normalization, and Procrustes transformations, root mean squared error (RMSE; primary outcome measure) was calculated for coordinates generated by MediaPipe vs. the touchscreen computer. Resultant mean RMSE was 0.28 +/-0.064 normalized px. Equivalence testing revealed that accuracy differed between MediaPipe and the touchscreen, but that the true difference was between 0-0.30 normalized px (t(114) = -3.02, p = 0.002). Overall, we quantify similarities between MediaPipe and a known standard for tracking fine upper limb movements, informing applications of MediaPipe in a domains such as clinical and research settings. Future work should address accuracy in 3-dimensions to further validate the use of MediaPipe in such domains.