16th European Youth Olympic Festival


Next-Generation AI-Driven Winter Sport Analytics

Research project between the University of Udine and EYOF 2023


Goal

The goal of the partnership between the Machine Learning and Perception Lab of the University of Udine and EYOF 2023 is to make a step forward into the development of artificial intelligence (AI) solutions for the winter sports industry. The potential of AI technologies is being received in other sports (e.g. football or basketball), but this has not been the case for winter sports yet. New software tools driven by AI could help trainers in providing precise measurements of relevant performance factors in order to increase the technical level of athletes, as well as they can make broadcasting contents more meaningful and appealing if the goal is to capture new practioners. Furthermore, the development of such AI systems becomes even more important if we consider that current young generations are constantly in contact with such tools (for example when they use digital and social platforms) and so they are used to rely on their capabalities. Hence, in this view, it is crucial to develop intelligent software so that young winter sport athletes can leverage AI to improve their sport activity and experience as they do for other activities.

Towards this end, the research project with EYOF 2023 focused on the application of such technology to winter sports. More specifically, of the branch of AI known as computer vision. In very simple words, this is the scientific field that aims to make machines understand the world from images and videos, as humans do with their eyes. This kind of automation is suitable in sport analytics because it allows the measurement of performance parameters without the need of the athlete wearing sensors, as the excercise can be captured just with a videocamera external to the point of action. Additionally, computer vision provides the advantage of producing an output that can be visualized and hence comprehended easily. More in detail, the research effort was spent on two major aspects: the development of new computer vision algorithms for performance analysis and understanding; and the design and implementation of an hardware platform to support the development and application of AI and computer vision to the discipline of alpine skiing. More information is given in the following.

Automatic Trajectory Reconstruction and Analysis in Videos

As a first step towards the development of AI-driven analytical solutions, the project focused on the problem of trajectory reconstruction and analysis in videos capturing the performance of athletes in skiing disciplines. An optimal trajectory is one of the key factors to achieve higher performance in different of such sports. For example, in alpine skiing the turns performed by an athlete form a trajectory of points on the slope surface that, if optimized with respect to the position of the gates, can lead to saved time during the descent. In ski jumping, a trajectory is formed by the points traversed during the flight phase, and different flight paraboles influence the distance of the jumps. The optimization of these curves can lead to an increased chance of better results. It is hence important to analyze trajectories in order to determine the specific points of the excercise that correlate to the final scores. To tackle this problem, our team designed and developed an algorithm that is capable of reconstructing the trajectories skied by alpine skiers, or flown by freestyle skiers, ski jumpers, or snowboarders, without (or in the worst case minimal) manual effort. Through modern machine learning techniques, the algorithm was trained to recognize an athlete and to track his/her position while he/she moves in the environment. In the case of a moving videocamera, the algorithm is also able to understand how much it moved and so it can compensate the trajectory such that it respects all the relations with the elements in the surrounding environment. This algorithm works online, i.e. it needs just the latest video frame available to compute the trajectory. This feature is important for real-world applications as the processing time can be reduced. Indeed, the computation of our algorithm can begin as soon as the exercise begins and does not need to wait the conclusion of the performance before starting the processing. Some examples of the trajectories reconstructed by our solution are shown in the following.

Example of the visual reconstruction of the trajectory of an alpine skier produced by our algorithm for a moving videocamera. In this case, the algorithm is applied on videos captured by a broadcasting videocamera.


Example of the visual reconstruction of the trajectory of an EYOF 2023 alpine skier produced by our algorithm for a moving videocamera. In this case, the algorithm is applied on videos captured by a drone for broadcasting purposes.


Example of the visual reconstruction of the flight trajectory of an EYOF 2023 ski jumping athlete produced by our algorithm. This video has been recorded with a standard smartphone.


Example of the visual reconstruction of the flight trajectory of an EYOF 2023 freestyle ski athlete produced by our algorithm. This video has been recorded by a broadcasting videocamera.


Based on such basic ability of generating trajectories and understanding the motions of athletes and cameras, we implemented higher-level performance comparison algorithms as shown below.

Example of the visual reconstruction and comparison of the flight trajectories of two EYOF 2023 ski jumping athletes. The algorithm is given the two videos and it automatically computes the athlete's trajectories and blend the two videos such that the evolution of the performances can be visually compared. The two videos have been recorded with a standard smartphone.


Example of the visual reconstruction and comparison of the trajectories of two EYOF 2023 snowboard alpine athletes. The algorithm is given the two videos and it automatically computes the athlete's trajectories and blend the two videos such that the evolution of the performances can be visually compared. The two videos have been recorded with by broadcasting videocameras.


Example of the visual reconstruction and comparison of the flight trajectories of two Beijing 2022 snowboard big air athletes. The algorithm works in the same way as for the previous example, without any particular tuning for the different winter discipline. In this case, the two videos have been recorded by broadcasting videocameras.

Alpine Skiing Track 2.0

The research did not focus just on the development of AI algorithms, but it took also in consideration the design and implementation of an infrastructure capable of supporting the development and exploitation of such AI technologies. More specifically, in the resort of Tarvisio (located in the north-eastern part of the Friuli-Venezia Giulia region), the alpine skiing "C" slope, usually reserved to competitions and training during the winter, has been augmented with a networking infstracture. Thanks to the support of Promoturismo FVG, the managers of the winter resort, Internet cables have been put along the sides of the slope and different points of connection have been installed. Such a configuration enables the connection of multiple videocameras as well as other types of sensors, and even computers. In this way, the performance of skiers can be observed by multiple views and measured with different synchronized devices, overall leading to an improved performance data generation system. This cutting-edge infrastructure has served to the our lab's researchers, and in the future will hopefully serve to other researchers and engineers, as a "living lab" to develop AI solutions for winter sports, as well as a technological platform for gaining precise information during training. To the best of our knowledge, this ski slope is a first of a kind in the world, and we believe it will be the prototype of the next generation of training venues, end even of alpine skiing slopes for recreational purposes. The resources that follow showw where the slope is located and some of its features.

This rendering shows where the smart ski slope is located as well some of its characteristics and those of the videocamera network infrastructure.


Picture of the actual installation on the slope's sides of some of the videocameras and of the boxes containing the network connections.


The following videos instead demonstrate some applications of the aforementioned computer vision algorithm to the videos of the cameras placed on the sides of the slope. The generated trajectories are analyzed and compared in different ways. Refer to the caption of the videos to know more.

Example of the visual reconstruction and comparison of the trajectories skied by different athletes on the networking-augmented slope. Our algorithm computes the trajectory for each of the descending athletes and labels it with the green or red color depending whether the time taken to perform it was the best or the worst, respectively. This result is obtained in real-time.


Example of the reconstruction of the trajectories skied by different athletes on the networking-augmented slope with information regarding the distance from the turn gates. After the athlete has completed the section, for each of the gate inside the track's section, our algorithm computes the point of the trajectory where the distance is the shortest. Such an output can be used to understand more precisely the characteristics of the trajectory. These results are obtained in real-time.


Example of the distribution of the reconstructed trajectories skied by multiple athletes on the networking-augmented slope. After the athlete has completed the section, our algorithm labels all the trajectories with different colors based on their associated section time. In the end (in the case of this video, after circa 20 minutes of observation), it can be understood which parts of the track or section are faster and which are slower. These results are obtained in real-time. (The video is speeded up for visualization purposes).


Recording of the 2nd run of top 30 competitors the EYOF 2023 Giant Slalom Boys held on January 26th 2023, where the trajectory reconstruction and analysis algorithm has been run. The examples shown in the previous videos (trajectory comparisons, distance from gates, distribution of trajectories) here have been obtained in real-time during the competition.

Outreach

Beside research, attention has been put in the disclosure of the topics and the developments of the project. Different audiences have been targeted, such as the general public, students attending sport-centered high-schools, and AI researchers and engineers. In the following, we provide more information on this kind of activity as well as links to public contents.

News and dedicated shows for the more general public about the topics of the collaboration with EYOF 2023 have been featured in the regional section of RAI, the main Italian broadcasting television, and in the regional television Telefriuli.

News appeared in the regional section of the RAI television announcing our development of a new class of performance analysis tools based on AI for the disciplines of skiing.


Interview to the Machine Learning and Perception Lab's director, Prof. Christian Micheloni, appeared in the main Friuli-Venezia Giulia television Telefriuli, talking about the main goals and developments of the project.


The researchers in our lab also took part into an event organized by the EYOF 2023 Organizing Committee aimed to explain to high-school students the olympic values behind the event and, more in general, that should be followed in any sport competition. Our team explained what AI and computer vision are and discussed, by showing real application examples, the importance that such technologies have and will have in the future for the technical advancement of sports.

Recording of the event dedicated to high-school students where the Machine Learning and Perception Lab explained what AI and computer vision are and how they have been used to implement new performance analysis solution in the MLP-EYOF 2023 project. (Our presentation starts at 37:00, please visit YouTube to watch the video).


The scientific results of the project were also presented to Cohere for AI, an initiative led by the Cohere company to disseminate worldwide the latest results obtained by researchers and engineers in AI.

Recording of the talk given by Dr. Matteo Dunnhofer to Cohere for AI.


The partnership between the lab and EYOF 2023 gave also birth to an event dedicated to researchers in AI and computer vision. In January 2022 and 2023, the 1st and the 2nd Workshop on Computer Vision for Winter Sports have been organized in conjunction with the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), the premier conference in the world where the most important applications of computer vision technologies are showcased. The workshops, organized with the help of the University of Trento, Italy, had two goals: on one hand to promote the employment of computer vision and AI solutions in the winter sports industry, by presenting the latest research solutions for winter sports-related problems; and at the same time, to stimulate the interest of the computer vision and AI audience with new and interesting problems arising in the domain of winter sports that could lead to the development of new algorithmic principles. The two events hosted speeches from prominent researchers and engineers that had experience in the usage of computer vision in winter sport domains, and short talks by authors of scientific papers that contributed to the advancement of AI solutions for such sport disciplines. Overall, the workshops connected people from Europe, North America, and Asia.

Logo of the 2nd Workshop on Computer Vision for Winter Sports held in conjuction with WACV 2023. Click on the image to visit the event's website.


Finally, the more technical outcomes of the project, such as methodologies, algorithms, and experimental results, have been delivered through the following scientific articles:

  • "Automatic Reconstruction of the Trajectories Performed by Skiers", Matteo Dunnhofer and Christian Micheloni, 9th International Congress on Science and Skiing, 2023
  • "Visualizing Skiers' Trajectories in Monocular Videos", Matteo Dunnhofer, Luca Sordi, and Christian Micheloni, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 9th International Workshop on Computer Vision in Sports (CVsports), 2023
  • "Tracking Skiers from the Top to the Bottom", Matteo Dunnhofer, Luca Sordi, Niki Martinel, and Christian Micheloni, IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2024
  • "Visual tracking in camera-switching outdoor sport videos: Benchmark and baselines for skiing", Matteo Dunnhofer and Christian Micheloni, Computer Vision and Image Understanding, 2024

Copyright © Machine Learning and Perception Lab - University of Udine - 2022 - 2023

Website template was inspired by TAO Dataset.