Abstract:

Beat and downbeat tracking have seen multiple improvements through recent deep learning approaches, but most target an offline setting, precluding their use in real-time scenarios. We adapt a recent offline beat tracking model, Beat This! [1], for online, low-latency operation and propose Beat This! Live, a causal variant designed for real-time beat and downbeat tracking. We present the architectural modifications required to obtain a causal system suitable for streaming inference and measure their impact on accuracy. By training our model with temporally shifted labels, it produces predictions slightly ahead of the current frame, which not only reduces latency, but enables much more effective peak-picking. Evaluation on the GTZAN dataset shows the F1-score improves from 80.6 to 86.7 for beats and 56.5 to 74.1 for downbeats compared to BeatNet+ [2], the current state of the art in online beat tracking.

Meta Review:

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Training the model to anticipate beats/downbeats by shifting annotations at training time is a simple yet powerful way of improving online deep learning beat tracking systems.

Q23. Main review and comments for the authors (to be completed before the discussion phase). Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This work proposes a causal, low-latency adaptation of the recent Beat This! architecture for online beat and downbeat tracking. The authors systematically convert the offline state-of-the-art model into a streaming-capable system and performance extensive experiments to analyze each design choice and demonstrate substantial improvements over prior online beat and downbeat tracking systems.

The paper is exceptionally clear, pedagogical, and well structured. The progression from causality constraints, to streaming inference, to latency-aware post-processing is easy to follow and supported by carefully designed experiments. The central idea is elegant and intuitive: the model is trained to predict beats slightly ahead of time. Finally, the experimental methodology is exemplary. I appreciated the authors dissecting the contribution of each architectural modification and double checking the causality of the model as well as carefully investigating the trade-off of their design choices (eg. Figure 2 and Tables 1, 2, 3). Really thorough.

I really enjoyed reading this paper, and I think it is a strong example of how to do thorough, sound MIR research. Overall, I believe this is a strong accept.

Q25. Meta-review and final comments for authors (to be completed after the discussion phase)

There was agreement among reviewers that this was a good paper and should be accepted at ISMIR. See main reviews for recommendations for the camera ready.

Review 1:

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Causal version of Beat This! provides low to negative latency.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This paper presents an online beat and downbeat tracking method based on Beat This! that achieves state of the art. Authors describe in detail the required architectural changes (causal convolutions, causal self-attention/sliding window attention, causal thresholding/shifted causal local max) and the methodology is very clear. I have only minor comments and suggestions.

  • Lines 127f: "with a dedicated task head each for beats and downbeats" -> "with dedicated task heads for beat and downbeat" (unsure what is each refering to here)

  • Line 188: "To combat this inherent" -> "To combat this inherent latency"

  • Lines 213f: This is more of a provocation - does the addition of a cooldown period after emisions contradicts the main idea of Beat This of avoiding imposing explicit musical constraints? (I guess it doesn't...)

  • Tables and Figures: Positioning could be improved. I caught myself going back and forth in pages several times, or looking for figure/table.

  • Lines 319f: "and this effect is more pronounced for downbeats, which are generally harder to predict" (add comma between "downbeats" and "which")

  • Table 3: Model names are very clear but also tiring to follow... I don't know how it could be improved. I imagined giving names for each segment of models and indicate explicity the better model. Example: call both 128-SW-256-3-1 and 128-SW-256-5-3 "Beat This! Live Zero", but say that the former is the better model (maybe an asterisk). Also, the table could use a column for the sliding window size.

  • Line 389: Used "$-7$" whereas in Table 3 caption "-7" (the minus sign looks like an n/m-dash in the first case)

Review 2:
Review 3:

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Beat This can be made causal and low-latency by training with backward-shifted labels, and the small resulting lookahead enables a peak-picking method (Shifted Causal Local Max) that reaches SOTA online beat and downbeat F1 on GTZAN at zero or even negative latency.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

The paper proposes Beat This! Live, an online causal version of the Beat This beat/downbeat tracker. It modifies convolutions and attention to remove future information, uses sliding-window attention for streaming, and introduces Shifted Causal Local Max postprocessing based on labels shifted earlier in time. Experiments on GTZAN report clear improvements over recent online beat/downbeat trackers, especially for downbeats.

Pros:

  1. The problem is important and well motivated. Offline beat trackers often perform well but are not directly usable in real-time systems, so adapting a strong open offline model is a useful contribution.
  2. The paper gives a clear causal conversion of the architecture. The changes to temporal convolutions, temporal attention, and sliding-window attention are well explained.
  3. The gradient analysis in Fig. 3 is a good sanity check. It makes the causality claim more convincing than only describing architectural changes. The subtle 4-frame leakage from the conv receptife field is a nice catch that shows rigor.
  4. The shifted-label idea is simple but effective. It gives the model time to anticipate beat/downbeat positions and enables better causal postprocessing.
  5. Shifted Causal Local Max is a strong practical contribution. It approximates the useful part of offline local-maximum postprocessing without using unavailable future audio.
  6. The ablations are well organised. The paper separately studies causality, streaming context, label shift, postprocessing lookahead, window size, and model size.
  7. The results are strong. Table 4 reports a clear improvement over previous SOTA methods, with especially large gains for online downbeat tracking.
  8. The paper is honest about the remaining engineering issue. Section 5 states that a real-time implementation still needs caching and low-overhead frame processing.
  9. Last but not the least: Good reproducibility. Same data, splits and spectrograms as Beat This, 3-seed means, code and weights released.

Suggestions for improvement:

  1. The paper should make the distinction between causal model, online evaluation, and actual real-time implementation more explicit. The current system is shown to be causal and fast in aggregate RTF, but the conclusion says that actual frame-by-frame deployment still requires extra engineering. This does not invalidate the result, but it should be stated earlier and more sharply. In other wods, "live" framing is slightly ahead of the evidence.
  2. Importantly, separate how much of the gain come from the stronger base model (Beat This) versus the online-specific parts (shift + SCLM). Much of the gap over previous online models may be due to a better base. Some material exists (e.g. Tab.3 does isolate shift versus no-shift on the validation set, but the method's effect against a weaker base model is not.
  3. Latency accounting need a clearer single explanation. Although these are spread across Sec 3.3 and 4.4, e.g. a small figure tying spectrogram lookahead + shift + reported latency together would help.
  4. The comparison in Table 4 is convincing but could discuss fairness more. RTF depends on hardware and implementation, as the paper notes, but latency and evaluation protocol (e.g. reproduced or copied) differences across others should be made explicit.
  5. The evaluation is mainly centred on GTZAN for the final SOTA comparison. Since Beat This! was trained on many datasets, the paper would be stronger with at least one additional held-out dataset or a short discussion of why GTZAN is the most suitable common benchmark for online comparison. In any case, a single dataset, evenmore one with known issues doesn't support generalization to SOTA.
  6. The novelty claim around predicting ahead should be phrased carefully because Meier et al. already use beat lookahead in a real-time system. The paper's novelty is better stated as shifted-label neural joint beat/downbeat tracking with causal local-max postprocessing, applied to Beat This!, rather than the general idea of beat anticipation.

Minor:

  • check reference formatting consistency as there are mixed venue styles in the bibliography: e.g. ICMC vs International Society for Music Information Retrieval (ISMIR).
  • Meier et al. " A real-time beat tracking system..." is from TISMIR, not ISMIR.

Score justification (Weak accept):

The paper is technically sound, reproducible, and the method is real (label shifting plus SCLM, with a large online downbeat gain). This is good work, I have no doubt. The minor corrections above (e.g. isolate the gain over a weaker base, one extra test set beyond GTZAN) would put it at strong accept.

Review 4:

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

The non-causal Beat This! beat & downbeat tracker can be adapted to operate causally, and doing so outperforms the state of the art.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

Summary: this submissions details the adaptation of the Beat This! beat and downbeat tracking system to operate in a causal fashion. The core of the methodology centers on the use of causal-only convolutions and self-attention together with a training regime that anticipates the ground truth labels of annotated beat and downbeat locations slightly early. In this way, the proposed approach can produce "full" peaks in the activation function (i.e. rising and falling) enabling more robust detection, with the added advantage that this essentially eliminates latency. A comparison against existing reference approaches, demonstrates the proposed approach considerably advances the state of the art.

General comments: In some sense there is no arguing with the numbers. The proposed approach goes well beyond incremental improvements in accuracy, especially for downbeat prediction, and firmly establishes a new state of the art. Leveraging this idea of shifting beat positions early is a nice one, and, without claiming to be a model of beat cognition, may align with the need to start the motion of tapping before the beat occurs such that the tap is synchronized with the musical stimulus.

That said, the submission doesn't make for the easiest reading. First, there is the disconnect between the title which implies this is operating in real time, versus what the submission describes which is a causal implementation with pre-computed input features. Even accounting for artistic license, the "predicting the future" is a bit on the cute side.

Second, the submission does little to nothing to motivate the need for a causal / real-time implementation of Beat This! There is of course the theoretical goal of "Is this possible?" but the text does little to explain "why this should be addressed." This is laid bare in the first paragraph of Section 1 which has no references at all. The submission could also note that the lack of any DBN post-processing for Beat This! means it is particularly well-suited to this kind of adaptation to predictive beat & downbeat tracking.

Third, the submission is quite heavy on (internal) evaluation, and structurally it's a bit confusing how sections 3 & 4 essentially mirror one another (i.e. the practical difference between causality and streaming), with the main comparison shoe-horned into a short section 4.5. To this end, the submission arguably spends too much time and space in self-reflection, when many of the internal results in Section 4 could be more concretely presented as an ablation study after the main comparison against the state of the art. Point in case, the results in Table 2 could be replaced with a sliding window evaluation (e.g. calculating F-measure every 5s with a 1s increment) over GTZAN and plotting the average over these 1s increments. Here the importance of some prior context would be clearly visible.

Perhaps more critically, there is little to learn about the nature of the improvements obtained over the state of the art, or more concretely, quantifying / qualifying the gap between the proposed approach and the reference non-causal Beat This! Real-time beat tracking presents significant unsolved challenges in terms of tracking changes in timing and tempo and yet the ability of the proposed approach to perform well in these conditions (which would be an important result - meaning the system could really "predict the future") is hidden within the average F-measure in Table 3. Given there's a full column before hitting the page limit, this space could be very valuably used by presenting the non-causal and causal activation functions together with predicted beat & downbeat locations for some successfully challenging excerpt in GTZAN.

In summary, the performance of the proposed method is certainly strong, but some reshaping of the structure of the submission would improve its readability and strengthen the contribution.

In addition to these general comments, here are a few more specific points:

line 80 "on what he called psychoacoustics" - Scheirer's contribution is a foundational paper in this area, which builds on decades of research into psychoacoustics. The wording of the text here almost appears dismissive.

lines 121-123, if only "close to" the text should state clearly what the state of the art is.

line 188 "To combat this inherent," - missing a word "difficulty", "challenge" ?

lines 224-225, it's a little unusual to present opinions on the method as its being described.

Somewhere in Sections 3 or 4, it would be useful to understand the hyperparameters used for training the model - this information seems to be entirely absent. The text should clarify if batch sizes, learning rates, early stopping etc, or whatever the precise methodology was in [1] has been directly applied here.

lines 279-280, it's not clear if the text introduces the notion or RTF or if there should be a reference here.

Section 6. Some minor tweaks for the references (beyond missing page numbers)

[6] darwin -> Darwin [11] ibt -> IBT [18] missing venue... ISMIR? [27] Raffel appears twice in the list of authors (first and last),

Post-Review Updates:

Typos, missing commas, ambiguous wordings have been fixed. There were inconsistencies in the formatting of references, which have also been rectified. As for more substantial changes:

  • We indicate which RTF scores have been computed, and which were copied from a previous publication. Since RTF is hardware dependent, and since BeatNet+ is open-source, we can present a more valid comparison by recomputing RTF on the same hardware.
  • As a step towards understanding where the performance gains over previous work come from, we add a per-genre comparison with BeatNet+.