AI knows the score — and it could help instrumentalists make beautiful music

The multidisciplinary project takes the lead on innovating STEM (science, technology, engineering and math) and art through AI. The project is funded by a National Science Foundation grant.

Yeon-Ji “Kristen” Yun, a performing cellist and the project’s principal investigator, said music and AI already have been used together in areas like recording and composition.

“But it’s not been used much in music performance and education,” said Yun, a clinical associate professor of music at the Patti and Rusty Rueff School of Design, Art, and Performance in Purdue’s College of Liberal Arts. “So as a musician myself, I wondered if AI technology could actually provide a benefit to musicians and possible advantages to performance while keeping their creative ability.”

This project will develop and integrate techniques from computer vision, natural language processing and audio analysis to create two AI-enabled tools for string music performers. The first tool, the Evaluator, aims to improve individual practice and performance by analyzing audio and video of a musician, then comparing it to digitized music scores and a database of video performances. It will offer potential improvements by detecting deviations in the audio and posture adjustments in the video.

The second tool, the Companion, is intended to play the part of absent instruments in an ensemble by using audio analysis of performances to match tempo and style of the musicians.

The initial focus of the research is on stringed instrument performances, but the intent is for it to be used with any instrument.

“These tools can be for string players with any level,” Yun said. “They can be for professionals or for amateur musicians, young or old.”

Yun is joined in the research by Purdue co-principal investigators Yung-Hsiang Lu, professor of electrical and computer engineering; Yingjie “Victor” Chen, professor of computer graphics technology in Purdue Polytechnic Institute; Cheryl Zhenyu Qian, chair of art and design at the Rueff School; and Mohammad Rahman, the Daniels School Chair in Management at the Mitchell E. Daniels, Jr. School of Business.

Yun said the Companion tool is much more in-depth than simply reading the music and reproducing the song.

“Musicians have freedom to create their own interpretations,” she said. “If they place a note slightly longer or slightly slower, we need to separate the duration from frequency while still following the ensemble’s tempo.”

The team is looking at the research from the perspective of when AI technology can provide measurable benefits to musicians’ practice and performance as well as what factors affect future musicians’ acceptance of AI technology in their work.

Yun will collaborate on user studies for the project with Ka-Wai Yu, associate professor in music at Utah Tech University.

About Purdue University

Purdue University is a public research institution with excellence at scale. Ranked among top 10 public universities and with two colleges in the top 4 in the United States, Purdue discovers and disseminates knowledge with a quality and at a scale second to none. More than 105,000 students study at Purdue across modalities and locations, with 50,000 in person on the West Lafayette campus. Committed to affordability and accessibility, Purdue’s main campus has frozen tuition 12 years in a row. See how Purdue never stops in the persistent pursuit of the next giant leap, including its first comprehensive urban campus in Indianapolis, the new Mitchell E. Daniels, Jr. School of Business, and Purdue Computes at https://www.purdue.edu/president/strategic-initiatives.