Abstract:

Real-time score following from sheet images remains challenging because the model must process streaming audio while resolving highly repetitive visual patterns under strict latency constraints. Recent image-based methods have attempted to use multi-resolution prediction by simultaneously predicting the positions of the active system, bar, and note. However, their predictions across these different levels of notation are independent, which makes the predictions unstable and introduces unnecessary extra search space for bar- and note-level predictions. Most existing methods also lack mechanisms to recover from score discontinuities, such as repeats, da capo (D.C.), or coda jumps. This paper proposes CODA, to the best of our knowledge, the first real-time score following system that addresses both gaps. CODA explicitly exploits the cascaded structure of music scores: it first selects the active system, then the active bar within it, and finally the active note within the selected bar. This enforces prediction consistency across resolutions. A silence-driven break mode enables recovery from arbitrary score discontinuities without requiring knowledge of the repeat structure. Evaluated on the Multimodal Sheet Music Dataset (MSMD) piano benchmarks, CODA achieves state-of-the-art tracking accuracy and discontinuity-recovery performance under real-time throughput. Code is available at https://anonymous.4open.science/r/CODA80.

Meta Review:

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

The cascaded score-to-audio alignment is robust to temporal discontinuity in music performance.

Q23. Main review and comments for the authors (to be completed before the discussion phase). Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This paper presents a real-time score following method using a cascaded score-image-to-audio alignment based on the structure of music score. The proposed methods uses three-levels of alignments in system, bar, and notes and conducts a two-phase curriculum learning which prevents the model from cascaded failure from the upper level. The model is particularly robust in discontinuous jumps during the performance, which is frequent in practice scenarios. The experiment results shows outperforming alignment accuracy over compared models and the ablation study clarity the effectiveness of the model components.

This paper is very well written, and I do not find any significant issues that need to be addressed.

The three-level cascaded model is well-grounded based on practical score-and-audio alignment scenarios. The potential weakness of the model is remedied well by the two-phase curriculum training.

This paper has a very solid experimental setup that rigorously evaluates the proposed model with directly comparable models. The ablation study clearly shows the contribution of the model components. The dedicated jump recovery evaluation convincingly shows the advantage of the cascaded model.

A minor issue is that the title does not indicate that the score is an image modality from the title. Considering the majority of score following is based on audio-to-audio or midi-to-midi alignment, it would be good to specify that the score is represented as an image in the title.

Overall, this paper is well written and easy to read. The proposed model represents a significant step in score-image-to-audio alignment. It has a great potential to real-world score following in music education contexts. I strongly recommend acceptance of this paper.

Q25. Meta-review and final comments for authors (to be completed after the discussion phase)

The reviewers unanimously praise the paper for its high-quality writing, strong musical grounding, and thorough evaluation.

The proposed two-phase curriculum learning strategy is highly commended for effectively mitigating the risk of cascaded failures from upper-level prediction errors. Furthermore, the model demonstrates state-of-the-art accuracy on the MSMD dataset and introduces a valuable repeat/jump-aware benchmark to the community, proving particularly robust against performance discontinuities.

There are a few minor points for clarification. They could be addressed in the camera-ready version.

Given the solid experimental validation, compelling ablation studies, and high potential for real-world application in music education, this paper is strongly recommended for acceptance. However, none of the reviews recommend it for a paper award.

Review 1:

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

The paper introduces a real-time score following system that employs a cascaded score following mechanism and silence-based jump handling, achieving state-of-the-art tracking accuracy and jump recovery.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

The paper demonstrates state-of-the-art results on image-based score following via the proposed CODA model. It is novel and valuable that the authors leveraged the cascaded structure of sheet music images to narrow the search space and enforce tracking consistency, and also a silence-based jump recovery system. The design choices (like the two stage training pipeline) and the ablations show a carefully designed and evaluated system.

Relevant baselines are included in the thorough evaluation setup. The paper additionally contributes a repeat/jump-aware benchmark which will be valuable for the community.

The proposed training paradigm depends on annotations of system/bar/note locations in the image, which of course limits generalizability to any real-world score image and/or introduces dependence on staff/bar detection systems. But this assumption is common in score-following literature, and not a concern for the scope of this paper.

Review 2:

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

A DNN for online sheet music image-to-audio system is proposed that predicts system, bar, and note position in a hierarchical manner.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This paper introduces CODA, a DNN for the online alignment of music audio and sheet music images. As a novelty, the system does a cascaded prediction (system, bar within system, note within bar), and claims to be discontinuity (jump)-aware.

Disclaimer: While I am familiar with music alignment, I am not yet familiar with audio-to-image alignment with transformer-based architectures. As such, I am reviewing this paper from the point of view of a generally interested ISMIR attendee without prior work in that field.

The paper is fluently written and gives a good introduction to the audio-to-score image alignment problem, as well as the additional difficulties that arise from the system/bar/note hierarchy and jumps in the sheet music image due to repeats, D.C., etc. The related work section seems to be comprehensive and cites a variety of work from the groups that are well-known for their work in audio-score alignment.

The method section is a bit hard to follow for the non-expert reader, but it seems to provide the necessary details. Figure 1 is a good overview, but could be made more accessible by trying to include all variable names that are used in the text (e.g., where are H_t, x_t, F, f_r... in Fig. 1?).

One thing I am not certain about is the jump detection and recovery strategy. As written in Sec. 3.6 and illustrated in Fig. 3, the authors seem to assume a silence break before a discontinuity; this is something I would not assume to be commonly occurring in music recordings, and the authors should add a deeper explanation/intuition for their motivation.

The experimental section is comprehensive and includes many baselines, by evaluating on the common MSMD dataset. The performance of the proposed method seems to be impressive, and the small yet understandable ablation studies offer insights w.r.t. the influence of the model's components.

Overall, this paper seems to be a valuable contribution to the score following community and is definitely a good ISMIR paper.

Review 3:

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

The authors propose a novel score-image based score-following system that is also capable of handling discontinuities in performance such as skips, repeats, etc.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This paper proposes a score-image based score following system that handles discontinuities in the performance. These discontinuities can range from musically defined repeats in the score, to performance error. The authors design the system to take advantage of the cascaded score structure, thereby ensuring consistency of prediction across note, bar and system levels.

The paper details a very thorough and clear explanation of the task at hand, by covering all the related work that was done in the task-domain as well. The proposed system architecture is very cleared defined, along with the motivation behind each part of the system. The usage of FiLM conditioning as well as cross-attention, and the final beam search methodology are not only interesting but are also based on strong musical reasoning, and are therefore well justified.

Some of the additional highlights that stood out were:

  1. The incorporation of the two-phase curriculum training to handle the downstream effects of incorrect system predictions, even though the model's system prediction accuracy is already quite good.
  2. The idea to suppress transition penalties and widen attention when pauses/silences are detected.

What was unfortunate was that there wasn't much room for discussion within the paper. I would have liked to know the authors' opinion and reasoning on some of the finer details of the parameters chosen, such as the silence onset of 3 frames/150 ms, energy onset and offset thresholds, and especially some insights on why the jump recovery was higher on the repeat subset than the random subset.

Congratulations to the authors on this work!

Post-Review Updates:
  1. Score-image modality in the title The meta-reviewer noted that the original title did not indicate that the score is represented as an image. We revised the title to explicitly identify the image-based score modality:

CODA: Cascaded Online Discontinuity-Aware Alignment for Real-Time Image-Based Score Following

  1. Method notation in Figure 1 Reviewer 2 suggested making the architecture overview more accessible by matching its labels to the notation used in the method section. We added (x_t), (z_t), (H_t), (F), and (f_r) to the corresponding audio, hidden-buffer, visual-feature, and ROI-feature paths in Figure 1.

  2. Motivation for silence-driven jump recovery Reviewer 2 requested more intuition for using silence as a jump-recovery cue. We now report Nakamura et al.'s observation that 59 of 63 repeats/skips in their clarinet practice recordings included breaks longer than 500 ms. We also clarify that silence is a practical cue for relaxing temporal constraints rather than a universal requirement for every discontinuity.

  3. Parameter choices and jump-recovery interpretation Reviewer 3 requested more reasoning for the silence parameters and the performance difference between the two jump subsets. We now explain that the energy-threshold gap prevents rapid state changes, the 3-frame onset filters brief energy dips, and the 8-frame grace window provides evidence after audio resumption. We also explain that annotated repeats resume at musically defined bar boundaries, while random jumps may resume at arbitrary onsets and form a harder stress test.