P1-4: Learning to Sing with Your Own Voice: Designing and Evaluating Generative AI-Assisted Vocal Coaching
Tianyu Gao, Shuqi Dai, Roger B. Dannenberg, Zhiyong Huang, Jin Song Dong, Jocelyn Dueck
Subjects: Creativity ; Cognitive and user-centered MIR ; Music training and education ; User-centered evaluation ; Creative practice involving MIR or generative technology ; Applications ; Open Review ; Personalization ; Human-AI co-creativity ; Creativity and learning
Presented In-person
Traditional vocal coaching centers on vocal demonstration, yet voice mismatch between coach and student has long constrained its pedagogical effectiveness. Recent advances in generative AI (GenAI) make it feasible to generate personalized demonstrations in the student's own voice, potentially narrowing this gap. However, it remains unexplored how such demonstrations can be integrated into real-world coaching practice to augment human coaching. In this paper, we present the first controlled real-world study of GenAI-assisted vocal coaching, a new form of coaching in which AI demos in the student's own voice serve as the only source of vocal demonstration while the human coach provides verbal and interpretive guidance. We evaluate the feasibility of this approach with 22 student singers, comparing single-session GenAI-assisted coaching with traditional human coaching under blind expert assessment. The results show that the GenAI-assisted group achieved immediate technical and perceptual gains comparable to expert human coaching. Student and coach feedback further revealed a shift in coaching dynamics, with students gaining greater autonomy, despite an imperfect connection to the cloned voice. These findings provide the first empirical evidence that GenAI-assisted coaching can augment, rather than replace, human coaching in vocal pedagogy, and we derive design implications for future MIR systems supporting GenAI-assisted vocal coaching.
Q18. Write ONE line (in your own words) with the main take-home message from the paper.
Using generated audio based on a learner's voice might be helpful in vocal coaching.
Q23. Main review and comments for the authors (to be completed before the discussion phase). Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.
This is an interesting paper that asks whether using a generated voice based on a learner's voice can be useful for voice coaching. The rationale for the study is that voice mismatch is a problem, supported by a citation showing that pitch matching is easier when imitating one's own voice.
The study was well-executed, with participants split between two scenarios: vocal coach only versus vocal coach using the generated audio. Five blinded reviewers analysed pre and post coaching recordings to decide on what errors were made, and various interpretive measures, such as expressiveness. Post was usually considered better than pre.
There are several issues I can see with the paper, most of which can be addressed in the writing.
I think it is important to explain what vocal coaching is, since it is different to singing lessons that work on vocal technique and production. It is more about learning a piece of music, or being coached to perform it better (more accurate, better expression, etc.), and this is what was done in this paper.
The main thing that was found was that voice coaching helped people learn a song and that hearing a recording of the song helped "ear-reliant" singers (non-readers) learn it well. It is not clear from this study, as presented, whether it matters that it was based on the singers' own voice. Looking at the supplementary data (thank you for including that) there is some, albeit not statistically significant, evidence that supports benefit where the voice type between singer and coach is different. I understand from a couple of pronouns in the text that the coach was female, therefore the timbre difference is presumably greatest with male participants - assuming the coach was an alto, mezzo or soprano. The largest difference in improvement was between human-male and virtual-male. Experience suggests that there is also a difference in general for tenor-bass versus soprano-alto in terms of pitch skills, so it would be worth doing a similar study with a tenor/bass vocal coach to see if the results are symmetrical.
I might also suggest that it would be useful to record voice type instead of gender for this type of study - especially where gender is different to that assigned from birth. It would better reflect voice type difference too. You could potentially group as SA (soprano+alto) and TB (tenor+bass) if finer gradings are not possible.
On looking at the supplementary data, participants were either assigned "Memory" (presumably from Cats?) or the aria "Vaga Luna". I note that the generated versions were potentially in different genres and styles. Does this mean that, for example, Vaga Luna might be rendered as pop? I notice from the supplementary information that the songs are allocated fairly evenly across singing skills. Does that mean that a minimally skilled singer assigned Vaga Luna received a recording in a simple pop or folk style? I think it would be worth providing more detail on the assigned pieces in the paper.
Looking at the data, it is clear that Vaga Luna was not known by the participants, so their pre recording would have relied on their reading ability (pre error average 122.1). In contrast, Memory was much better known, with much lower error rate in pre-recording (42.1). The groups were well-balanced in terms of singing skill, song and gender, but it is unclear whether reading skill was a factor.
Thoughts regarding "phonemes". It became clear through the participant comments that some songs were in Italian and others possibly in English. Some of the benefits cited by participants is regarding pronunciation of Italian. I can see that there might be benefit in general for language pronunciation, which could be interesting to explore elsewhere, but is beyond the scope of ISMIR.
The dataset is interesting. I wonder if it could be made available in some other form, such as by anonymising the vocal timbre, or if there is more detail available regarding the evaluations (eg. specific notes that were errors). I didn't see it stated anywhere, but it is important to note whether the coach and experts in the study are co-authors of the paper.
Section 2.1 last sentence -> "in a student's own voice".
Q25. Meta-review and final comments for authors (to be completed after the discussion phase)
All reviewers found the paper interesting but there was a consensus that the claims are overstated, particularly related to pedagogy, and even regarding the benefit of the generated voice. Many details were omitted in the write-up that made it difficult to know exactly what was done. Despite these limitations, it was agreed that the paper could be accepted, provided key changes were made.
- Be more explicit about details of the methodology. Several reviewers have stated that certain details are unclear, or only apparent when digging into the supplementary material. In particular, there is confusion about whether participants are learning a song or learning vocal technique. The experiment is much more about learning the notes of a song, or for those who already knew it or who could sight read, it was about improving âdictionâ. Was evaluating for diction mainly picking up pronunciation errors in Italian?
- Tone down the claims. The study compares two quite different learning sessions, and doesnât really isolate the impact of timbre. It may, however, show a benefit in helping where the voice type between coach and learner is different.
Reviewers provide further suggestions in their individual reviews. The paper referred to by Reviewer 1 is: Suvi HÀÀrÀ, Isabelle Oktay, Rafael Ramirez. 2026. MuSA Voice: Evaluating Deep Learning-Based Vocal Technique Feedback for Self-Regulated, Reflective Singing Practice, in Proceedings of International Conference on Computer Supported Education (CSEDU).
Q18. Write ONE line (in your own words) with the main take-home message from the paper.
A study on the potential of using voice cloning to support singing coaching.
Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.
In my opinion, this is a good contribution. Therefore, I have only minor suggestions for improvement:
-
In 3.2, it is indicated that the group working with the AI-demos was singing along it, while the group working with the coach was imitating the teacher demonstrations. This difference is not necessarily a problem, since both methods are not comparable; however, the fact that the learning procedure was not the same should be explicitly pointed out and clarified to which extent this allows still taking the conditions as comparable and why singing along was chosen (instead of imitation) for the AI-demo approach.
-
In the description of Stages 2 and 4 (displayed in Figure 1), it would be useful to explicitly clarify whether the time to practice individually before stage 4 was the same for all participants. In addition, how long it was? Were the recordings performed in stage 2 and 4 also made in the lab?
-
In 3.3, an interesting point which was not discussed, would be whether there is any observed relationship emerging between the AI method (SVC or SVS) and the selected sample in terms of genre, voice type, or other criteria.
-
Between lines 449-473 the text reads a bit repetitive (space could be safe here if necessary to integrate reviewers suggestions).
-
In lines 513-514, the feedback from the participant could be connected with a recent publication from the special session in computer music education presented at CSEDU this year, if the authors get access to it: https://www.insticc.org/node/TechnicalProgram/csedu/2026/presentationDetails/150755
-
Between lines 531-534 it is briefly mentioned the fact that voice constitutes biometric data. I would suggest examining a bit more transparently what exactly the cloning implies in the current publication, for instance, how the data were handled/storage/shared, etc. This should be added in a dedicated ethical statement at the end of the paper. In particular, this is important because teachers face real dangers when using AI if the tools do not comply with the institutions data protection regulations; thus, more transparent information would support those interested in evaluate whether the presented approach can be realistically implemented in a practical scenario and which risks are associated.
-
line 399: differently" Despite -> differently". Despite
-
Quotation marks shoudl be revised, some are different to those use elsewhere in the paper (see lines 397-399); others are in the wrong direction (see 410).
-
line 5.1: as an Pedagogical -> as a Pedagogical
Q18. Write ONE line (in your own words) with the main take-home message from the paper.
The paper investigates whether AI-generated singing demos in the studentâs own voice can support vocal coaching.
Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.
This paper presents a controlled study comparing GenAI-assisted vocal coaching with traditional human vocal coaching. In the GenAI-assisted condition, students received verbal guidance from a coach while using AI-generated demonstrations in their own cloned voice as the only vocal model. In the human-coaching condition, the same coach provided traditional vocal demonstrations. The study involved 22 student singers, one experienced vocal coach, and five external professional evaluators who assessed pre- and post-session recordings. The topic is relevant to the ISMIR community. The paper is generally well written, clearly organized, and easy to follow. The application of existing SVS/SVC systems to personalized vocal coaching is interesting and potentially valuable, especially as an exploratory direction for future MIR research on music learning and generative AI systems. I also appreciate the attempt to evaluate the system in a real coaching context rather than only through technical audio-generation metrics.
However, I have substantial concerns about the strength of the pedagogical claims made in the paper. The main issue is that the actual intervention consisted of only a single coaching session, with only 30 minutes of coaching time. This is a very short exposure period for a paper that repeatedly frames its contribution in terms of pedagogical effectiveness, learning gains, student autonomy, and the ability of GenAI to augment human coaching. A 30-minute coaching session can certainly lead to immediate improvement on a specific piece, but this does not necessarily show that the improvement was caused by the voice-matched AI demonstration. It may also reflect familiarization with the piece, repeated exposure, immediate correction, ordinary practice effects, or the general benefit of receiving focused coaching. This concern is reinforced by the lack of delayed retention, transfer task, or longer-term follow-up. The study measures immediate pre/post improvement on the same musical material, but it does not show whether students retained what they learned, transferred it to another piece, or developed more stable vocal or musical skills. More broadly, music education and training literature generally evaluates pedagogical interventions over multiple sessions or weeks rather than a single short-term session. Therefore, I think the current results support a short-term feasibility or proof-of-concept claim, but not the stronger pedagogical conclusions currently made in the paper. A second concern is that the study does not cleanly isolate timbre matching as the causal pedagogical variable. The two conditions differ in more ways than the timbre of the demonstration. The GenAI-assisted group used a fixed, replayable, pre-generated audio demonstration, while the human-coaching group received live vocal modeling from the teacher. These are different pedagogical situations, not simply matched vs. mismatched vocal timbre. A live vocal demonstration includes not only the sound of the voice but also embodied, visual, interpersonal, and expressive dimensions of teaching. A teacher demonstrates breathing, phrasing, posture, facial expression, timing, and musical intention, all of which may affect engagement and learning (see for example work on embodied vocal pedagogy and multimodal teaching practices). In this sense, the AI condition may reduce one type of mismatch while introducing other pedagogical gaps. This limitation should be acknowledged much more clearly. A related issue is that the musical material is underdescribed. The paper does not provide enough information about the assigned piece: how it was selected, whether all participants sang the same piece, its style and difficulty, whether it was new to participants, and whether some students had prior familiarity with it. This information is essential for interpreting the pre/post improvement. If the piece was unfamiliar, then a large part of the improvement could simply reflect short-term familiarization with the melody, rhythm, diction, or accompaniment. The paper would also benefit from a more substantial limitations section. Important limitations include the short intervention period, the small sample size, the use of only one coach, the absence of retention or transfer measures, the lack of detailed information about the musical material, the fact that the two conditions differ in more than timbre alone, and the potential loss of embodied and interpersonal aspects of live vocal demonstration in the AI condition. The discomfort some students reported with the cloned voice and the ethical implications of using cloned singing voices should also be discussed more explicitly. Given these considerations, I do not think the current evidence supports the strength of the paperâs conclusions. The work would be stronger if framed as a preliminary feasibility study showing that personalized AI-generated vocal demonstrations can be integrated into a short coaching session and may support immediate piece-specific improvement. In its current form, however, the paper overstates its pedagogical contribution, and this is the main reason for my weak reject recommendation.
Q18. Write ONE line (in your own words) with the main take-home message from the paper.
AI-assisted vocal coaching can be effective for significantly improving one's pitch, rhythm, and diction score on a singing test of it in just 30m compared to the baseline without AI-assisted vocal coaching.
Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.
Some more explanation on the importance of diction would help a lot. I saw numerous breadcrumbs of its importance, such as the eval comments on how to grade solfege vs actual words, and there is perhaps a weakness in this. Also I'd question how much of diction matters, and without audio examples of right and wrong, it's hard to imagine what the test is. Perhaps if diction is intended and very important for this style of singing performance, that should be explained a bit more. (like for example, was it Italian opera, and non-native speakers had to learn how to pronounce the words in addition to singing them?)
It also seemed like the error rate was quite high for what looked like many singers with 5+ years of experience. 0.80 starting error rate with 5+ years of experience when the eval is based around just pitch, rhythm, and diction, not sure I understand that, and some examples would help.
I saw in the notebooks, there is a great plot of subjective expressive and pleasing judgements. If it's possible to work that into the paper and do an Ai-assisted vs no-Ai split, that would be pretty interesting to see and be a good supplement (any result is interesting and would strengthen the paper, not looking for one way or the other).
Thanks for including the code and visuals, love that kind of a submission!
I saw the note about audio not being able to be shared which is unfortunate, definitely would help understand it better, but I think explaining a bit more of the material would help. A typical written down version would help: musical notation of a phrase they were tested on and what the correct pitch, rhythm, and diction looks like written down.
Dear Meta-Reviewers and Reviewers, We sincerely thank the reviewers for your constructive feedback. Based on your feedback, we revised our manuscript to clarify the study design, ground the pedagogical context, refine our claims, and incorporate required policy declarations.
- Tone-Down the Study Scope: We updated the Abstract, Introduction, and Conclusion to explicitly frame this work as a short-term feasibility evaluation, clarifying that findings demonstrate immediate, single-session piece-specific gains rather than long-term motor skill acquisition.
- Methodological Clarifications: Section 3.1 specifies that participants learned one of two classical-style pieces ("Vaga luna" or "Memory" from Cats) matched to their voice type, with no prior familiarity. Section 3.4 clarifies that diction errors were judged by classical-vocal-trained evaluators using standard classical pronunciation conventions. Section 3.2 details that singing along enabled real-time alignment while phrase replay supported targeted imitation, and Stage 4 included 5 minutes of independent score review.
- Limitations & Literature: Section 5.3 explicitly addresses the 30-minute session duration, practice effects, sample size, single-coach setup, subjective ratings, and human embodied cues (e.g., posture, facial expression). Section 2.2 adds vocal pedagogy references, and Section 2.3 incorporates recent deep learning-based vocal feedback literature (HÀÀrÀ et al., CSEDU 2026).
- Ethics & Declarations: Section 7 details IRB approval, anonymous audio coding, and biometric privacy controls, noting that all voice cloning was executed locally without cloud uploads. Section 8 complies with ISMIR 2026 AI guidelines for system models, programming, flowcharts, and editing. We also streamlined Section 5.1 and corrected typographical and quotation mark errors. We believe these revisions make the manuscript better align with ISMIR standards.
Sincerely, The Authors