Acoustic Guitar Chords Recognition

An innovative automated system for recognizing guitar chords in acoustic guitar videos. Our approach combines YOLO [Redmon et al., 2016] for fretboard detection and DINOv2 [Oquab et al.] with a ViT backbone [Dosovitskiy et al., 2020] for chord classification. We investigate hand pose estimation using MediaPipe and extend the work of [Kristian et al., 2024] by integrating modern deep learning techniques and proposing an audio generation component.

By: Camilo Martínez, Dhimitrios Duka, Honglu Ma

View on GitHub

This system automates chord recognition from acoustic guitar videos by detecting and classifying chords based on video input. It leverages YOLO [Redmon et al., 2016] and Faster R-CNN [Ren et al., 2016] for fretboard detection, allowing the system to identify the position of the hand and fingers on the guitar neck. For chord classification, it utilizes Vision Transformers [Dosovitskiy et al., 2020] and DINOv2 [Oquab et al.], which process visual cues to distinguish between different chords. Additionally, hand pose estimation with MediaPipe was explored as a potential method to perform chord recognition. Finally, we extend the work from [Kristian et al., 2024] by exploring the potential of using state-of-the-art deep learning models and techniques with an additional proposal for an audio generation module.

This system was created Dhimitrios Duka, Honglu Ma and myself, as the final project for the High-Level Computer Vision course lectured by Prof. Dr. Bernt Schiele at Saarland University during the Sommer Semester 2024.

Fretboard Detection

The table below shows the performance metrics of the different models tested on the finetuning dataset (Guitar necks detector), and the figure below shows the Recall vs. mAP@50 for the models tested and finetuned on the fretboard class, while showcasing the number of parameters. Naturally, the models finetuned with a Frozen Backbone (FB) performed slightly worse than the models finetuned without a Frozen Backbone; this was expected since the latter had the advantage of being able to learn the new task from scratch, using all layers, while the former only trained a smaller classifier head. Since we wanted to retain the ability to recognize the other 80 valuable classes from the COCO dataset, we chose a model from the (FB) list, the YOLOv9 (FB) model, as the best model for our task. This model obtained the highest precision and, after re-evaluating on the COCO dataset + fretboard class, it delivered better results in terms of confusion matrix and Precision-Recall curve.

Qualitative Results

Some qualitative results are shown below, comparing the original YOLOv9 (c) prediction with the finetuned model and the model with a frozen backbone + classifier layer.

Guitar Chord Classification

Hand Pose Estimation + Classifier

To evaluate our approach against those in our reference paper by [Kristian et al., 2024], we implemented the InceptionResNetv2 model as described by the authors. After training the model using the hyperparameters provided by [Kristian et al., 2024] on our dataset, we obtained the results shown in the table below, which provided us with a baseline to compare our models against. Surprisingly, this approach performed well, achieving good accuracy during validation and testing on two datasets. However, the model struggled to generalize to the third dataset, which was created by us. This outcome was anticipated, as the samples in our dataset were out of the training distribution, and the model lacked the complexity needed to generalize to such data.

Classifier only approach

To address this limitation of the previous approach, we decided to explore more complex models, such as Vision Transformers and DINOv2, which is also available on Hugging Face. The results of our experiments are summarized below:

ViT models show varying performance across different datasets. The base models perform exceptionally well, with high accuracy on all datasets. However, the larger models do not exhibit the same performance. We argue that this is happening because the available data is not sufficient to train the large version of the models effectively. Additionally, we can also observe that the patch 16 versions of the ViT models perform better than the patch 32 versions. This is likely due to the fact that the patch 16 versions have a higher resolution, which is important for accurately distinguishing between different hand positions.

Moreover, both DINOv2 variants demonstrated strong and consistent performance across all datasets. The DINOv2-L model, in particular, achieved the highest accuracy on the Guitar_Chords_Ours dataset, slightly outperforming the small variant. The superior performance of DINOv2 can be attributed to its self-supervised learning approach. Unlike models pre-trained on ImageNet, which does not contain a specific class for hands, DINOv2's self-supervised learning enables it to learn more generic and transferable representations, leading to better generalization in our task. This enhanced generalization is further supported by attention visualizations of the model when applied to images from Guitar_Chords_Ours dataset, where the model correctly focuses on the hand performing the fretting, as evidenced by the following figures.

Overall, our proposed models outperformed the InceptionResNetv2 model, achieving higher accuracy across all datasets. This demonstrates the potential of using more advanced models for chord classification tasks.

For more details, please refer to the project's GitHub repository.