Elaborating Advanced Machine Learning Techniques in the Music Class

In music education, there are several cases where the instructor needs to set preparatory tasks and use verbal communication, both of which, nonetheless, interrupt the music continuity. These “interruptions” are considered as learning barriers. Having researched teaching communication habits on several music instruction cases, we have come up with the idea of designing a set of software blocks that, laid down together as a digital aid to the class, can generously assist music teaching by providing communication facilitators in a wide range of commonly used music teaching exercise tasks. In this direction, a range of algorithms and software blocks have been implemented at the Ionian University using the Max/MSPTM dedicated software platform, comprising the FIG set of tools. A specific subset of these software tools has included Machine Learning (ML) logic in order to promote a wiser instructor-student communication that advances class musicality and potentially facilitates deeper consolidation of musical structures.

Introduction

There is a prominent necessity for non-verbal communication techniques in music teaching. That necessity stems from the very nature of the teaching subject. As teaching music includes the teaching of gestures, phthongs, and meters, one needs to employ body language, as for these terms, students need to be tight with sound and movement acuteness. In this paper, we demonstrate the development process and basic uses of a proprietary piece of software that we built to thoroughly assist the aforementioned type of communication. This is achieved by breaking down a common set of novice music teaching materials and by designing a software helper that guides the instructor through a step-by-step process of performing the teaching tasks, in a manner that is purely musical, easy to reach, and straightforward. As the algorithms used have been originally developed for general purposes, identifying the terms under which they could be directly useful to the music class is a critical issue for our research planning.

Towards a More Musical Music Lesson Based on Kinesthetic Activities Using ML Algorithms

To date, Artificial Intelligence (AI) and ML-based music generators have not thoroughly convinced musicophiles in terms of appreciation [1]. This premise is amplified by the fact that humanity has a long way ahead in morally accepting the creative endeavors of machines [2], [3]. Therefore, articulating intent when using ML&AI in music practice seems to be an acceptable practice [4] and this explains the fact that ML&AI music generators are being promoted as creation assistants [5], [6]. Likewise, we suggest that ML algorithms have the potential to be used effectively as teaching assistants.

Music Software as Teaching Assistants

The teaching and learning of musical concepts utilizing information and communication technologies (ICT) is required at all levels of music education. Moreover, teachers are asked to use ICT and teaching techniques that take maximum advantage of the visual, auditory, and kinesthetic abilities of the students.

As music software and ICT literacy advance teachers tend to utilize ready-to-use software in the classroom. Such software includes acoustics and audio, MIDI concepts and basics, advanced production and DAW’s, music notation, aural awareness, and teaching and learning [7], [8]. Furthermore, recent resources offer teachers the possibility to develop software by themselves according to their needs [9], [10].

Machine Learning Algorithms in Music Practice

The study of human-computer co-creative music practice using ML and AI algorithms in both improvisation and composition is a lively scientific field. An overview of current systems can be found in [11]–[14]. However, to the author’s best knowledge, the study of the human-computer teaching practice using ML algorithms in mainstream and special music education is in its infancy. Recent research by Parke-Wolfe et al. [15] suggests that the field is characterized by potentiality in the creation of ML music teaching assistants.

In our research of appropriate ML algorithms, we found that the ones implemented by FluCoMa^TM [16] and by Weka^TM [17] have been successfully applied to interactive music software design on Max/MSP^TM [18]. Further work based on the FluCoMa^TM algorithms has been done by Rodrigo Constanzo, namely the SP-Tools^TM, which is a set of ML tools that are optimized for real-time performance [19]. Respectively, further work based on the Weka^TM Algorithms led to the creation of the Wekinator^TM, a free and open-source piece of software for artists and musicians that uses a collection of ML algorithms for data mining tasks. Wekinator^TM is used to create interactive systems in which the computer responds, in real time, to human actions, for example, physical and musical gestures [20].

Our research seeks to integrate various algorithms by FluCoMa^TM and Weka^TM into the digital tools to be developed.

Establishing the Need for Musicality in the Classroom

One can establish the musicality of the music class in a deterministic or quantitative manner, by adding up the amount of time spent on musical sounds in the classroom air during a certain time period. From another perspective, musicality is achieved through a flourishing set of musical notes that are played during, for example, an instrument improvisation, a clear indication of a rise in students’ creativity and technical control level. Several studies, including Forsythe’s [21] have shown that off-task levels are mainly based on preparatory tasks and during verbal communication between students and teachers. Driven by our constant quest for even more musical lessons we are in search of developing tools that maximize the musicality and the kinesthetic activities and minimize verbal communication and preparatory tasks in the music class.

Solution Design and Software Architecture

Considering the difficulties of lesson preparation, the peculiarities of the spatial and technological infrastructures of each educational unit, and the wide gradation of high and low-level abilities of students, our research proposes new user experiences (UX) and digital tools that do not require programming knowledge. As ML technology comes to the rescue, such tools can be customized according to the respective spatial infrastructure and do not require low-level skills from the students’ side. Our software potentially enhances the communication techniques between instructor and student by facilitating access to information, minimizing preparation, providing immediate feedback, facilitating the transfer, and increasing the musicality of the music lesson. The originality lies in the innovations of inputting data into the computer and in applying ML for data mining. In addition to the common ways of inputting information (mouse, computer keys, MIDI devices), audio and video inputs are used for signal routing to music information retrieval systems to recognize and categorize musical and physical gestures. For example, chord building and harmony are taught with a set of digital tools which, firstly, allow for simpler performance of chords, thus separating the cognitive functions of chord analysis and progression from the physical act required to perform the chords. Secondly, these digital tools have the potential to increase the musicality of the lesson through new ways of inputting data and through ML e.g., using pitch or other musical/physical gestures to activate certain functions of the digital tool in use. The proposed human-machine interaction mode will be particularly useful in situations where there is a lack of infrastructure (mainly musical instruments) which is usually accompanied by a lack of skills as well as for teachers and performers of monophonic instruments who are not familiar with polyphonic ones, the use of which is necessary for the doctrine of harmony.

The FIG software is an innovative music education suite designed by the authors and built having in mind an expandable structure of Max/MSP^TM patches. It is presented for the first time in this paper and consists of prime modules, one for each task category, named Chord-Builder, Melody-Maker, and Ear-Trainer, respectively.

Αs shown in Fig. 1, the order of the patches of our Chord-Builder follows a step-by-step music teaching methodology of chord building and thus are organized as follows: intervals, triads, extensions, ML game. The last patch is a prototype patch that uses ML algorithms by FluCoMa^TM, as integrated in SP-Tools^TM [19]. Apart from standard Max/MSP^TM objects, all patches use a library of external objects by Agostini and Ghisi [22]. Patches three and four use external objects developed by Manzo [9].

Fig. 1. Prime task structure of FIG’s ChordBuilder corresponding to software patches.

Using FIG Chord-Builder in the Class

The first patch (see screenshot in Fig. 2) assists the teacher in delivering the concept of intervals. The user chooses the key and interval to be performed. The way of inputting data is done by either using a MIDI keyboard, mouse on a stage, or a microphone. The audio stream from the microphone is analyzed in real-time by the Fiddle object [23] which detects the pitch. The user interface (UI) offers two ways of displaying the intervals, staves and keyboard.

Fig. 2. Intervals patch on FIG’s ChordBuilder.

The second patch (Fig. 3) assists the teacher in explaining and demonstrating triads. The user chooses key, triad quality and triad root. The way of inputting data is the same as in the first patch. For reasons of consistency, the UI offers similar functionality routing and same rules of display look-and-feel throughout FIG.

Fig. 3. Triads management on ChordBuilder.

Τhe third patch (Fig. 4) assists the teacher in demonstrating chord extensions. The user sets the key and the scale and chooses the desired chord degree for which he can add and hear the available extensions. The different input methods assist the music teacher in experimenting with more composite lesson delivering techniques while interacting with the class.

Fig. 4. Chord extensions on FIG’s ChordBuilder.

The fourth patch uses supervised ML. Low level audio features are extracted with the use of Mel Frequency Coefficients (MFCC’s). Its usage is shown in Fig. 5, while Figs. 6 and 7 show its internal functionality. The user selects classes, provides features, and trains the FluCoMa^TM fluid knn classifier.

Fig. 5. Presentation mode of the machine learning patch on ChordBuilder.

Fig. 6. Edit mode on ChordBuilder’s ML Game: feature extraction, training, and classification.

Fig. 7. Edit mode on ChordBuilder’s ML Game: triggering, auditioning, and displaying of chords.

Any sound can be used for feature extraction. When training is completed the system is ready to accept features to be classified. Then, the user sets the key and the scale/mode from which the corresponding scale degree chords will be formed. Each chord can be assigned to a specific sound/class i.e., sound/class 1 corresponds to the chord formed on scale degree I etc. By creating 7 classes the user can have all 7 chords of an eptatonic scale assigned to 7 different sounds. The classification triggers the corresponding chord which is sounded through a general MIDI bank.

The blue labelled patch cord at the bottom left of Fig. 6 carries the classified data.

This cord continues on to Fig. 7 to the red labelled from symbol object which turns the symbol into an integer.

The triggering of the selected chord is then managed by the modal triad object [9]. The chords are displayed on both an onscreen keyboard and on a musical stave. This patch can be used in the classroom for either assessment, performance or composition.

Formative Evaluation Procedure

On completion of the current software development phase, it is of prime concern as to the level of acceptability FIG receives from the music education community. At the time of writing, our software develops in a generic way. We focus on the creation of an all-purpose software teaching assistant for harmony, specifically chord building, and melody. Additionally, we work on the development of a real-time feedback ear training app. Harmony and melody, as fundamental elements of music, constitute an integral part of nearly every music lesson and pillars upon which the formation of music curricula at all levels of music education is based. We believe that this generic design can act as a testing and evaluation vehicle aiming to get as many teachers as possible to use our software no matter what level or module they currently teach. Along the path of making this software an off-the-shelf preference in standard music education settings, we have been seeking useful feedback throughout the development process. Our target group of formative evaluators lies within our social sphere of music instructors. They have been trying out FIG in the lab and their classes, offering fruitful feedback. The mainstream of observations and commentary lies in the process of training the patch. Other categories of comments include screen layout, task structure, font size, and system requirements. In the course of time, we have considered all that useful feedback and have incorporated it into our design thinking.

Discussion

Towards the level of acceptability of a software instructor in music education, there are no illusions of top levels of success. In a pilot study of Greek secondary education music teachers and private music teachers we have just completed, it is evident that a non-trivial portion of subjects is not willing to experiment with digital media in the classroom, not to mention ML/AI tools. This leads us to think that the challenge of providing a well-suited UI and UX is critical to our pursuit of offering FIG as a commodity/reference music tool to state music education. Certain tailor-made usability/UX attributes must be employed for us to ensure the functional, learnable, and satisfactory merits of our system. Cultural aspects that affect the degree of acceptability are also to be carefully considered, on the way.

In terms of algorithmic testing, the knn classifier by FluCoMa^TM performs significantly fast onset detection which makes it a perfect candidate for the percussive audio signal. Comparably, Wekinator^TM has considerably slower onset detection but due to Open Sound Control [24] compatibility demonstrates promising flexibility in cross application multimedia use. The response of the pitch detector used in all patches of FIG’s ChordBuilder has been tested successfully with string instruments (violin, guitar, double bass) and with reed instruments (tenor, alto, and soprano saxophone).

Finally, the course material to be included in FIG is another consideration for our strategic research plan. It would be ideal to offer the full set of the state secondary education white book in our full functionality range. However, it is important that the initial set of functionalities is thoroughly tested in the real class environment with the developed generic ChordBuilder.

Conclusion

In this paper, an innovative software platform for music teaching is introduced. The FIG is an aspiring digital music teaching assistant that is based on machine learning algorithmic logic. We have demonstrated that FIG has been successful in putting into practice the standard machine learning logic into a music teaching assistant tool. It is our belief that adopting machine learning techniques can enhance the music class in a fruitful and productive manner, as it can potentially increase musicality, student engagement, and kinesthetic activity in the class.