Humans recognize a musical passage even when it is shifted in time or transposed in pitch, indicating a notion of equivariance in the representation space. Our analysis, however, shows that standard music transformers map such time-shifted or pitch-transposed inputs onto uncorrelated representations: these models become progressively less equivariant as they scale in size or train longer. This suggests that in standard music transformers, additional model capacity is allocated to memorizing absolute patterns rather than capturing shared musical structures. In this paper, we propose the Equivariant Music Transformer (EMT), which enforces equivariance through self-distillation by jointly optimizing a next-token-prediction and an auxiliary equivariance regularization loss. We find that the additional equivariance loss acts as a beneficial regularizer, simultaneously improving next-token prediction and producing equivariant latent representations. Through both objective and subjective evaluations, EMT demonstrates superior equivariance and generative capability compared to data augmentation, feature engineering, and state-of-the-art (SOTA) baselines. More broadly, our findings reveal that standard language modeling methods alone do not capture music’s translational symmetries, and dedicated inductive biases are required to produce better music representations. The code, weights and demos are available online.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Equivariance in musical data is relevant not only to data augmentation strategies, but also to the design of loss functions.

Q23. Main review and comments for the authors (to be completed before the discussion phase). Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This paper proposes a symbolic music modeling loss which takes time- and pitch-equivariance into account. Unlike previous works that typically use data augmentation to enforce equivariance, this work shows that using an equivariance loss is better than data augmentation. Overall, the idea is novel in the domain of symbolic music modeling. It is somewhat surprising that such a simple idea appears so late, as it is actually not novel in other domains (e.g., from a CV viewpoint, translation invariance is an old topic). Very interesting. The proposed loss function exhibits potential in symbolic music processing.

Aside from the contributions I can see, my critical comments are as follows.

One limitation is that the paper seems to treat pitch shifting and time shifting as the same type of operation. However, these two operations may have fundamentally different effects on music representations and may also lead to different musical implications. Investigating time-shift and pitch-shift separately would likely be more appropriate, especially if the goal is to understand the limitations of shifting values.

Another possible direction for future work is to transfer the model to multiple downstream tasks. Currently, the experiments mainly evaluate the model on next-token prediction ability.

Other comments:

• For the first cross-entropy loss function, it may be clearer to write it as (\sum x_i \log(p_i)) (less confusion).
• Table 1: I suppose you could still report the (L_{NTP}) values for the Anticipation Music Transformer, right?
• In the experiment of next token generation, what do you mean by the “unshifted” prompt? In some sense, all prompts can be viewed as unshifted.
• From your generation examples, I observed that the generated music may modulate every five seconds. Does this behavior align with the assumptions of the original model? More generally, how should we interpret the model’s robustness under local key modulation? Or, let's consider a case where the input pitches are shifted every second (is this still valid pitch shifting)? I expect more discussions here.

Q25. Meta-review and final comments for authors (to be completed after the discussion phase)

All the reviewers highly valued the idea of this paper, and are unanimous in accepting it. At the same time, reviewers also raised quite a few constructive comments. We strongly encourage the authors to revise the paper according to the comments.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Adding an auxiliary equivariance-regularization loss (a stop-gradient KL term applied to musically shifted inputs) alongside next-token prediction makes a symbolic-music Transformer's latent space more equivariant to pitch/time shifts while modestly improving generation, and the authors argue that such equivariance does not emerge from scaling standard Transformers alone.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

Summary. The paper proposes the Equivariant Music Transformer (EMT). It is an autoregressive symbolic-music Transformer. It trains with next-token prediction plus an auxiliary equivariance-regularization loss. The auxiliary loss applies a random musical transposition or time-shift to the input. It passes the shifted input through the shared-weight model. It then uses a stop-gradient KL term so the inverse-shifted output distribution matches the original. The authors evaluate on LakhMIDI. They run internal ablations and compare against external baselines (AMT, MIDI-LLM). They report validation cross-entropy, equivariance metrics, and a listening test. They argue that equivariance does not emerge from scaling.

Strengths.

The motivation is clear and musically meaningful. Pitch and time-shift equivariance reflect how listeners hear relative musical structure. Current symbolic-music models mostly rely on indirect input-level biases. The method is clean. The stop-gradient anchor is a sensible way to stop the two branches from collapsing to a trivial solution. The ablations are informative. They separate feature-level and model-level equivariance and augmentation at comparable sizes. The evaluation covers both prediction and structural robustness, including a targeted listening test. Reproducibility is good. The architecture, training, transforms, and evaluation are all reported, with public data and baselines. Weaknesses and suggestions.

The required AI/ethics statement is missing. The paper goes straight from the conclusion to the references, and the listening test recruits human participants, so a statement is needed here. Please add it. All evaluation uses LakhMIDI. Showing the approach on a second symbolic-music dataset would strengthen the generality of the findings. Reasoning for the overall rating. This is a clean, well-motivated contribution, and there is no major weakness. The one required fix is the missing AI/ethics statement; evaluating on a second dataset would also help. I recommend acceptance once the statement is added.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

This paper presents a valuable contribution to the symbolic music generation research field through proposing an additional loss function to enforce equivariance, which is well-supported through a detailed ablation study, superior objective and subjective scores despite fewer model parameters than SOTA models, and through the provided audio results.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This paper presents a very well-founded contribution to the music generation research field which is also very well written. The approach seems simple yet effective -- add an explicit loss function to enforce equivariance in music generation. The presented ablation study seems to be detailed enough to build confidence in the working of the proposed method, and the provided results (both objective as well as audio results) seem very promising. How the equivariance starts to go down as the non-equivariance-enforced models train for longer was particularly quite enlightening. The presented hypothesis on this matter "...without explicit equivariance supervision, transformers increasingly allocate capacity to memorizing absolute patterns rather than recognizing the shared underlying structure" seems to be well supported given that this model despite fewer parameters than other SOTA models is able to perform equivalently (or even better). I believe this approach has the potential to be adopted in other music generation systems. For all the reasons above, I strongly recommend this paper for publication.

Please find below some feedback that in my opinion may further enhance the paper:

Major: The only major feedback I had was that under related works, on lines 161-164, although the past works tackling equivariance have been cited, given that data augmentation and feature-level equivariance have been used in the ablation studies, a bit more elaboration on these approaches would be greatly beneficial.

Other minor comments: Lines 67-68: Please elaborate on what scaling is being referred to here. Lines 99-100: In line with my major feedback, it would be useful to elaborate a bit on what feature-level equivariance entails. Lines 203, 274: FEM, RoPE -> Please use full forms when referencing for the first time. Lines 238-240: Would be good to put actual numbers here for the number of parameters. Lines 258-260: Regarding the Sample ID mask, would be helpful to elaborate a bit on this, or provide a reference. Line 280: Unclear what the augmentation parameter here corresponds to. Was the equivariance loss also used in the data augmentation experiments? Please clarify. Line 295: Typo - this should be 24 pitch shifts and 20 time shifts. Line 306: It would be good to elaborate a bit on what data augmentations are specifically used here. Probably the previously suggested elaboration on the data augmentation method might clarify this. Line 314-215: I see that equivariance level is mentioned here but described in a later section. Perhaps put a reference to the section where it’s described to enhance readability. Lines 329-331: Are the randomly sampled shifts randomized at every validation epoch? If yes, please specify, if not, curious as to why such a small percentage of these transformations are used for calculating the objective scores. Also in that case, it would be useful to use a fixed percentage of OOD transformations instead of making this fully randomized with no control on how many OOD transformations are getting sampled. Lines 372-373: I'm confused - Does model 3 use the auxillary loss? I thought, as previously described it’s Model 2 + Data Augmentation, while Model 2 has only the NTP loss. Please clarify. Figure 3: These plots are too hard to read. They have to be bigger, or at minimum the colors for each plot could be chosen to be more distinct. Minor: technically all models are “yours” and what's being proposed (the equivariance loss) is applied in both (4) and (5) (and perhaps in (3) as well?). Perhaps change the notation for (5) from (ours) to something like (proposed). Lines 402-403: Going back to my major feedback, I would really like to see a bit more information on what data augmentation means here. Even a single line description would be useful. I understand that these past works have been cited, but a short description, either here or earlier in the paper, would be greatly useful for readers to get a better and faster understanding. Same goes for the feature-level equivariance. Line 414: I would replace the word "lowest" with something like "worst", given that lowest equivariance loss actually corresponds to best equivariance. I got very confused reading it the first time assuming this was referring to the loss value :) Lines 441-442: Please also provide the number of parameters that the S, M and L variants have if you can. Section 5.2: Wonder if there's any SOTA model that uses feature-level equivariance and if you're able to provide the equivariance loss values for those pre-trained models as part of your comparisons.

Concluding thoughts: This is a great paper! The hypothesis makes total sense and the ablation results seem very convincing. I even listened to the audio examples provided and they seemed convincing to me. Some food for thought: I wonder what this would mean for scenarios where the composition is not equivariant to pitch shifts – where the composition is intentionally different depending on the absolute pitch. For e.g. I can imagine a guitarist wanting to explore a different improvisation when playing in an upper octave as opposed to when playing in a lower octave. Wonder how this approach would tackle such scenarios. Also I wonder if this loss term would make the generation more restrictive and in the process end up hurting the “wow” element in the generated music. Just something to think about and potentially look further into :)

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Making a Transformer equivariant to pitch- and time- shift leads to better music generation capabilities.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This paper proposes a simple but smart regularization strategy to make music transformers more equivariant to time- and pitch-shifting. The method is nice, rigorously evaluated, and shows that domain knowledge can be more beneficial than scaling. The manuscript is quite clear overall, despite a few minor issues detailed below.

Method

Regarding the method itself, I only have very minor comments. Even if circular shift (torch.roll) is convenient, it may lead to boundary issues. I therefore think that cropping would be more appropriate. For instance, if there is a shift of $k$ semitones, the equivariance loss could be applied on $p_i[k:]$ versus $p_i^T[:-k]$, rather than relying on a circular shift.

I also agree with the authors that it feels right to put a stop-gradient operator in L_equiv​, but I am not fully convinced by the explanation given in lines 192-195. Even without the stop-gradient, I think the NTP loss should prevent collapse.

Since memory constraints are mentioned twice in the paper, I also have a small suggestion. Backpropagating the sum of two losses is equivalent to backpropagating both of them one after the other. Instead of backpropagating the sum of L_NTP​ and L_equiv​, one could try the following:

1. do the forward pass on the original sequence
2. clone the output sequence and detach the clone
3. compute L_NTP​ and backpropagate it
4. transform the input sequence and do the second forward pass
5. compute L_equiv​ and backpropagate it
6. call optimizer.step() This is equivalent to the current training procedure, but I think it may use less memory because the model does not have to store all intermediate activations for both sequences at the same time. This idea is inspired by Berton’s trick (https://twitter.com/gabriberton/status/1796531585958985941). I might be wrong, but I think it is definitely worth trying.

Evaluation

The evaluation is clear, technically sound, and convincing, particularly the objective evaluation. In my opinion, two small ablation studies are still missing.

First, it would be useful to test whether the stop-gradient operator is really useful. The paper emphasizes this point quite strongly, (“Crucially”, line 188), but do not provide a reference or empirical validation. Either a small ablation (ideally) or insisting less on this point (at least) would improve the paper.

Second, in line 281 it's written that lambda values were “selected empirically”. It would be nice to include a small ablation study on lambda. In particular, contrary e.g. to GANs, here the equivariance loss should not compete with the NTP loss, so your model should be robust to the choice of lambda. The ablation would enable to verify this.

I am conscious that additional experiments would take a bit more space, and that the paper already reaches the 6-page limit. To gain space for such experiments, I think the introduction could be condensed. It is well written, but contains several redundancies. The second paragraph of section 3.1 (lines 211–227) could also be shortened. Similarly, the first paragraph of section 4.2 (lines 267–275) could be reduced by simply mentioning that the model can be trained on two H100 GPUs using mixed precision. Figure 1 is also very large and could be reduced if necessary.

The subjective evaluation is interesting, but 12 participants is very low. Gathering people for user studies is always annoying, but it would be amazing if the authors could get more participants for the camera-ready version. I strongly encourage them to harass their colleagues and friends to reduce the confidence intervals in table 2.

Also, unless I missed it, it is not indicated which checkpoint of the Anticipatory model was used for the listening test. It should be clearly written, ideally on table 2.

Finally, the paper should absolutely mention in the results section that a supplementary website with audio examples is available, not only in the abstract. More generally, I do not think it is useful to include a footnote with a URL in the abstract.

Style and presentation

The manuscript is generally clear, but could easily be improved by addressing a few minor issues:

• Figure 1 is very far from its analysis, and it would be better to place it on the previous page.
• The authors should use equation environments for equations, so that they can be referenced properly, and add punctuation at the end of equations.
• In Equation 1, the formulation should be a conditional probability, i.e. p_i(x_i) should be written as p(xi | x_0,…,x_{i−1}).
• Figures 3 and 4 should be larger, especially Figure 3, which is unreadable without zooming a lot.
• The tables are also difficult to read. For Table 1, the first four columns could be merged into one to gain horizontal space. For Table 2, the size should be increased.
• The author for reference [20] is missing.

In terms of structure, I would prefer having first the setup and results for the objective evaluation, followed by the setup and results for the subjective evaluation. Right now, the setup and results of the MOS test are very far from each other, which forces the reader to scroll up and down a lot to properly analyze it.

Conclusion

Overall, this is a strong paper. The proposed regularization is simple, smart, and well motivated, and the evaluation is convincing. The comparison with models of different scales also provides useful insights, potentially beyond the scope of this paper.

Despite the few remarks above (among which most can easily be addressed), I would, therefore, strongly accept this paper for ISMIR 2026.

Additional explanations on feature-level equivariance and data augmentation. Re-organized the abstract to improve the flow.