This study analyzes the intermediate representations of music foundation models (MFMs) and reports the geometric structures used to represent pitch information. By inputting isolated musical notes into trained MFMs and analyzing their principal components, we reveal that the representations form a helical structure reflecting the octave periodicity of pitch. Furthermore, we show that the clarity and geometry of this helical structure vary not only across models but also with the acoustic properties of the input signals. Our analysis provides a novel approach for clarifying the internal mechanisms of MFMs.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

The paper presents a finding that transformers tend to encode pitch in a helical structure similarly to the structures described in musical psychophysics.

Q23. Main review and comments for the authors (to be completed before the discussion phase). Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

The paper provides stimulating research exploring the structure of latent representations acquired by transformers. Its exploration about the helical pitch structure after Shepard is useful, informative and links the paper to discussions in music cognition and psychophysics.

One main critical question is whether the helical representation is already active earlier encodings: acoustical features or even the tokenizer?

Strengths:

• Clear und valuable research question
• The mathematical methods are solid and well-described.
• The paper is clearly written and easy to follow

Weaknesses:

• There are few citations to relevant work
• The conclusions are claiming to much: they should be toned down as only singly pitch input is tested.
• Only two specific models are tested and this should be stated more clearly -- the generalization to foundation models in general is not clear.
• method 3.1 could be described in greater detail
• The method for extracting input and intermediate representations is not described clearly
• N_key is never properly defined

potential error in Line 167: why p=1?

Further comments and suggestions:

• For a paper like this it would be useful if it drew more connection to perception studies about human pitch perception following up on Shepard's foundational results.

• Give more explanation why harmonicity improves and non-harmonices hinders the formation of the helical structure.

• Results about the layer depth of the models would benefit from more discussion.

• The paper only assesses one geometry of musical ptich. In music theory, a range of different geometries have been described such as the (infinite) line of fifths vs. the circle of fifths, tonnetz structures, voice-leading geometries. It would be interesting if foundation models would also be analyzed with regards to such structures.

Q25. Meta-review and final comments for authors (to be completed after the discussion phase)

This is an interesting paper with a potentially useful insight. The discussion with the reviewers converged to one main point: in the current methodology, it is unclear whether the helix-like representation of pitch already arises at the feature level or tokenizer level, which would need additional controls. The paper does provide evidence that the helix representation occurs in the transformer layers. However, in order to be publishable the author must adapt the paper: either add additional controls (at the feature level and tokenizer level), or acknowledge these issues and tone down the claims and discussion of the paper so that they reflect clearly what has been shown and what is not shown.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Paper investigates if pitch is captured in intermediate layers of MFM and if there is any helical structure learnt.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

The manuscript proposes to investigate if large music foundation models embed pitch in their intermediate representations. Two models - Jukebox and MusicGen have been considered. Two datasets are considered - instrument note dataset and synthetic dataset. Helicality is defined (as inv of MSE of 3dim representation wrt helical features) per layer, and is proposed to be used as an indicator of presence of pitch. It is observed from the experimental setup that helicality increases in deeper layer representation. Regression co-efficients indicate stat. sign. of the regression fit. All the above indicating that pitch information is present in deeper layers of MFM.

Manuscript is well-written, has clarity and is thorough in analysis. Some questions/concerns:

1. What explains lesser layer max Helicality in Figure 8 for certain instruments? This is relevant because there is no pattern for the lesser layer max Helicality. Since it may not be influenced by timbre, what does it imply? a. Lesser representation of the instrument in training data of MFM? b. Regression not good enough for certain instruments across layers? c. Limitation of using 5C3 PCA combination of intermediate layer representation?

2. Figure 10 seems specific for Jukebox. MusicGen has 48 layers. It maybe a good thing to separate out the two instead of clubbing it into one figure.

3. Using the helicality score in terms of downstream task such as pitch identification/tracking or in any generation task would be insightful.

4. An analysis on how this approach would scale on any new model will be insightful to the readers.

Minor comments:

1. Consistency in referring to Figure as either Fig/Figure will be useful.
2. How is Figure 9 generated?

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

The paper shows that pitch representations in two generative music foundation models can form helix-like geometric structures, and that this structure is strongly influenced by the harmonic content of the input sound.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This paper studies whether the intermediate representations of music foundation models encode pitch as a geometric structure similar to the Shepard pitch helix. The authors input isolated notes from different instruments into Jukebox and MusicGen, extract intermediate representations from each Transformer decoder layer, apply PCA to pitch-conditioned representations, and fit a parametric 3D helix. They use the Helicality metric to measure how close the representations are to a pitch helix. The paper further uses artificial signals with controlled harmonic components and spectral tilt to analyze which acoustic factors help or weaken the formation of the helical structure. Overall, this is a small and careful interpretability paper. The research question is musically meaningful, the method is clear, and the paper provides some interesting observations.

Strengths:

1. The research question is interesting and musically meaningful. Compared with common linear probing studies, this paper does not simply ask whether a model can predict pitch. Instead, it asks whether pitch is organized inside the model as a geometric structure related to octave equivalence. This connects music cognition, pitch perception, and foundation model interpretability, and is a valuable perspective.

2. The method is clear and relatively reusable. The authors extract intermediate representations for different pitches from isolated notes, project them into a 3D space with PCA, and quantify the pitch-related geometry using helix fitting and the Helicality metric. This analysis framework could be useful for future work that studies other musical concepts or other model representations.

3. The artificial signal experiment is a highlight of the paper. The authors do not only observe Helicality across different instrument timbres. They also synthesize harmonic-controlled signals and analyze how octave-equivalent and non-octave-equivalent harmonics affect helix formation. This makes the paper go beyond visualization and tries to explain the relation between the helical structure and the acoustic properties of the input.

4. The results are informative. The paper shows that pitch-related helical geometry can be observed in the Transformer layers of Jukebox and MusicGen. This structure is stronger in deeper layers and is strongly affected by timbre and harmonic composition. These observations help us understand how generative music models may organize pitch-related information.

Weaknesses:

1. The main issue is the lack of important baselines. Because of this, it is hard to know whether the observed helix is an internal structure learned by the Transformer, or a structure that already exists in the acoustic features or tokenizer representations. The isolated-note inputs already have clear harmonic structure, and pitch helix-like geometry may naturally appear in acoustic features such as CQT or log-mel spectrograms. In addition, both Jukebox and MusicGen use learned audio tokenizers/codecs before the Transformer, such as the VQ-VAE in Jukebox and EnCodec in MusicGen. This paper analyzes Transformer decoder layers, but does not report the Helicality of tokenizer/codec representations. Therefore, if the helix already exists in raw acoustic features or tokenizer outputs, the current results cannot fully show that the Transformer/MFM has learned a new pitch geometry. I suggest adding raw feature baselines, codec/tokenizer baselines, and randomly initialized model baselines.

2. The title and some conclusions are somewhat over-generalized. The title uses "music foundation models", but the experiments only analyze Jukebox and MusicGen, both of which are generative music models. The paper does not analyze MERT, MuQ, MusicFM, LAION-CLAP, or other understanding-oriented music foundation models. Therefore, a more accurate conclusion would be that pitch helix-like structures are observed in two generative MFMs, rather than that MFMs in general encode pitch with a helical structure.

3. Although the results are interesting, some findings are not very surprising. Octave equivalence and harmonic structure are already fundamental to pitch perception, and the inputs are only isolated notes. Therefore, observing a geometry related to pitch height and pitch class is not completely unexpected. The paper’s contribution is the careful quantification and analysis of this structure, but without stronger controls, the uniqueness of the insight is still limited.

Other comments:

1. I suggest explaining the input and intermediate representation extraction process for MusicGen more clearly. The paper states that the text encoder is omitted, the music signals are input into the frozen EnCodec, and the Transformer decoder is then analyzed. However, the relation between EnCodec tokens/latents and Transformer hidden states could be explained in more detail.

2. I suggest weakening statements such as "MFMs embed pitch in a helical structure". A more careful statement would be: "helix-like pitch geometry can be observed in the Transformer representations of two generative MFMs under isolated-note inputs."

Overall assessment:

Overall, I think this is an interesting, careful, and clearly written paper. It proposes a more musically interpretable analysis than standard probing, and it provides useful observations through Helicality and artificial harmonic-controlled signals. However, the main issue is the lack of important baselines, such as raw acoustic features, tokenizer/codec representations, and randomly initialized models. Because of this, the paper does not fully rule out a natural alternative explanation: the observed helix may mainly come from the harmonic structure of the input audio or the model’s front-end tokenizer, rather than from an internal musical knowledge structure learned by the Transformer/MFM. For this reason, I think the work has clear potential, but the current evidence is not yet sufficient to fully support the main conclusion.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Internal representations of large music generative models encode the helical structure of pitch in a manner that depends on the model as well as the harmonic structure of the signal.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

Overall comments The authors propose a helicality measure for representations and evaluate two music generative models on artificial signals as well single MIDI synthesized notes. They observe variable helicality among instruments and in layer-wise representations, specifically that latter layers have higher helicality. Moreover, helicality trends seem to be well correlated across the two models. Multiple regression analysis on artificial signals suggests that harmonic profile affects measured helicality, such that octaves and even harmonics positively contribute to helicality, i.e. the proxy for a helix-like embedding, while odd harmonics contribute negatively to helicality.

The problem statement, mathematical formulation and analysis are interesting and of value to the community. However, there are some flaws in the experimental methodology, and the takeaways from this paper could be discussed and presented better. Given the clear hypothesis and validation, what are the broader implications? I think the paper would benefit from evaluating smaller as well as non-generative models to contextualize the findings for the MIR community, evaluating using real-world data, and improving the discussion to gain better intuition from the findings.

Nevertheless, I believe this is a useful paper for the community. I would encourage the authors to conduct some follow up studies to address earlier comments in an extended or future version of this article.

Section wise comments: Introduction Motivation needs more discussion about the implications of this study for topological understanding, mechanistic interpretability, and downstream usefulness. Reading prior cited work helps, but the core motivation needs to be clarified in this work. Why are these MFMs chosen? Moreover, since this is an analysis paper, including other representations such as MERT or the encoder used in music language models such as Music Flamingo [4], may yield more reusable insights.

Method Why are the transformer decoder layer outputs the best choice for this experiment, as opposed to the input audio tokens? For instance, Wei et al. also investigated the codec outputs in these models using symbolic data [1]. While decoder layers performed better than tokens [1], this choice needs to be justified briefly in methods and ablated in the experiments. How is the proposed model to evaluate helicality different from the one proposed in [2]? This warrants some discussion.

Experiments Setting hyperparameters to optimize for helicality seems a bit circular. A results section showing what values worked and why may be more informative–what and how important are the reference helix parameter choices for different instruments? Why not evaluate also with real-world audio? While the artificial signals data provide a controlled testbed to study the influence of harmonic structure on helicality, a dataset with isolated instrument recordings like SOL [3] may provide better insights than the MIDI dataset. You could then also evaluate the influence of playing style for instance. Figure 10 and 11: are these results for Jukebox only? Instrument ordering is moderately consistent between models, but it's unclear whether that is the case for layer-wise and instrument-level distributions.

Following the reviews and meta-review, we revised the paper to clarify its methodology and limitations. Section 3.1 now clarifies that the input audio is encoded by each model’s pretrained audio tokenizer and that the resulting token sequences are passed to the Transformer decoder before intermediate representations are extracted. We also revised the figure captions to describe the presented results more accurately.

In response to the meta-reviewer’s main concern, we acknowledged that the observed helix-like geometry may already arise at the acoustic feature or tokenizer level. Because additional feature- and tokenizer-level controls were not included in this revision, we toned down our claims. We limited our conclusions to observations from the two analyzed generative MFMs, Jukebox and MusicGen. We also explicitly stated that whether the geometry originates in the Transformer or in earlier representations remains unclear.