I - Score Following
An important part of the project consists on comparing a performance with a score : The matching of differences between them will lead us to errors detection.
Such systems have been implemented for other purpose, mainly for real-time and interactive music. Known as score-following systems, their goal is to synchronize live instrumental music with computer activated samples, what is in fact very close to our goal : to activate samples, we need to know exactly when and how the expected note was played. These data could be compared to the expected ones in order to detect potential errors.
Score following for jMax - IRCAM, centre Pompidou (Paris, France)
jMax is a visual programming environment for interactive real-time music and multimedia, developped at IRCAM.
For our purpose, the most important feature of jMax among all the different packages which have been so far implemented is the Score Following Package.
Based on Hidden Markov Models, this package is being implemented for 2 years in the real-time system team at IRCAM. The score is modeled as a markov chain, with 2 states for each note or chord : One state for the right note (normal state, or n-state), one state for anything but the right note (ghost state, or g-state), and transitions between states, with their own probabilities.
Example : a score and its model
This states succession is very interesting for our purpose : if we know which state has been reached, we have the possibility to know where an error was done... Furthermore, it is possible to extend this model : type of used datas, other parameters, etc.
jMax and its Score Following could be used for the Virtual Music Teacher in different ways, mainly :
- Using the Score Following package as input for our system
- Implementing another package with some of the Score Following functions
- Grouping the two packages in a "super-package"
A discussion with jMax's authors at IRCAM will allow to choose between those possibilities.
Sources and further informations :
More informations about jMax and Score Following :
jMax :
http://www.ircam.fr/equipes/temps-reel/web-jmax/en/
Score-Following mailing list :
http://listes.ircam.fr/wws/info/score-recognition
Personal documentation :
http://gmwork.free.fr/ScoreFollowing/
Papers about HMM based Score Following :
N. Orio and F. Déchelle.
In Proceedings of the ICMC, Havana, Cuba, 2001.
N. Orio and D. Schwarz.
Alignment of Monophonic and Polypophonic Music to a Score.
In Proceedings of the ICMC, Havana, Cuba, 2001.
II - Speech and language Recognition
As we can consider that music is a kind of language, the language recognition approach could be use for music recognition :
- get data (speech signal / musical signal)
- extract reliable features (dynamic, spectrum, ceptrum, ...)
- model (diagrams, trees, neural networks...)
- compare (expected text in a dictionnary / expected sequence in a score)
Furthermore, as we can have a speaker adaptation in speech recognition systems, we can have an instrument adaptation in music recognition system, in order to improve the acuracy of the matched performance.
The speech recognition approach has been already used for the HMM score Following described above.
Sources and further informations :
L. Mayfield Tomokiyo and Alex Waibel
Adaptation Methods for Non-native Speech
In Proceedings of Multilinguality in Spoken Language Processing, Aalborg, September, 2001.
L. Mayfield Tomokiyo, L. Wang, M. Eskenazi
An Empirical Study of the Effectiveness of Speech-Recognition-Based Pronunciation Tutoring.
In Proceedings of ICSLP, Beijing, October 2000.
X. Huang, F. Alleva, H.W. Hon, M.Y. Hwang, R. Rosenfeld
The Sphinx II speech recognition system : an overview
January 1992