Abstract:

Neural audio representation learning has enabled a range of content-oriented applications, but the resulting features remain limited for tasks involving audio processing. Furthermore, it is not obvious what processing-aware representations should capture: the processing itself, abstracted away from source content, or the processed audio that retains it. Existing approaches implicitly commit to one or the other and also differ in their models, data, and evaluation, obscuring which design choices drive their behavior. We address both questions within a unified framework of three objectives: processing consistency, description alignment, and equivariance via forward prediction. We compare all combinations of the objectives under a controlled setup and reveal their relative strengths and interactions. Our framework produces both a transformation embedding and a processed-audio embedding, and we find that the two play complementary roles: distance-based tasks favor the former, while probe-based tasks favor the latter. Combined with improvements in network architecture and training pipeline, our representations outperform prior baselines across retrieval, probe-based evaluation, and style transfer.

Meta Review:

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Representing audio transformations separately from the processed content is useful as those representations have different performance characteristics in downstream tasks

Q23. Main review and comments for the authors (to be completed before the discussion phase). Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

The paper provides a novel exploration of representation learning techniques for audio transformations. The core ideas of processing consistency, description alignment and equivariance learning have merit, and could even be transferred to different application domains altogether. These ideas are then convincingly validated by the rigorously conducted experiments, as the proposed approach achieves substantially better performance than current state of the art.

The paper is also very well written, clearly laying out the methods and experiments.

Since I don't have any major complaints about this paper, I recommend acceptance.

Minor comments:

  • Would be nice to have the Figure 1 extended to show how z_T^+ and \hat(z)_y are produced
  • Could add a footnote to Table 1's "Condition" Column to note that these are only using chain length 1, and a small reasoning why we focus on single-chain for the analysis of the different conditions (WS, WI etc)
  • L338 Briefly describing the justification already here would allow for better reading flow
  • L353 Paragraph should mention that these results are (also) in Table 2

Q25. Meta-review and final comments for authors (to be completed after the discussion phase)

All reviewers agree that this paper represents a meaningful scientific contribution, as the proposed concept of capturing audio transformations and their associated audio information separately is novel and experimentally verified to have practical relevance.

There are some non major experimental shortcomings and some claims in the paper are stronger than the evidence supports (particularly around equivariance, and compositional understanding). Furthermore, the rule of the loss terms is under-explained.

Therefore I recommend to accept this paper, and encourage the authors to further improve it regarding the claims made and general clarity.

Review 1:

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

By using a combination of contrastive objectives in a multi-view setup, partially separated audio representations that encode (1) the audio processing, abstracted from source content (transformation embedding), and (2) the processed audio embedding itself can be learned.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

Summary: The paper presents an interesting shift in thinking of encoding audio processing effects within latent spaces, challenging the way audio processing (e.g., reverb) may be used to create views of the same audio for learning a single embedding space invariant to such augmentations. Instead, they propose to learn both a processed audio embedding and a transformation embedding, where the former ideally encodes more source information and the latter encodes more processing information. They present a thorough, well-structured ablation of their proposed contrastive objective variations across many downstream evaluation methods and show that their method can mostly outperform baselines across tasks. Importantly, they find that it is difficult to fully separate these embeddings, as one is derived from the other (z_T is learned from z_y), and their results reflect this.

Overall I find the paper a very interesting read and well-structured study. The weaknesses presented below are addressable with a bit of deepened analysis and clarification of claims, leading to my weak accept recommendation.

Major strengths:

  • The proposed method challenges typical thinking of audio processing as an augmentation method towards learning a single invariant embedding space, and instead examines how we can retain this information separate from the audio content information within a latent space. The suite of downstream methods also presents a strong framework for other work in this space.
  • The paper is well-structured overall; the methods and objectives are clearly explained in a model-agnostic way before the architecture is introduced in 3.1 which makes it easy to follow. The data, training, and evaluation setup is clearly explained.
  • While not highlighted so much by the authors, they introduce a PyTorch audio processing library which could definitely be used by the community in the future.
  • The related work (specifically as discussed in the objective design sections) is well-justified and clearly connected to proposed methodology with appropriate distinction.

Major weaknesses:

  • The effect of L_y is not entirely clear/analyzed. In Table 1, on it's own it is quite a bit worse across retrieval tasks, and even hurts when added in combination with the other losses. More explanation is needed to understand why you think this interaction affect is occurring or why this loss is much weaker than the others.
  • Along the same lines as above, from Tables 1 and 2, it seems like L_p is the dominating loss, which comes more from the encoding of the processing information explicitly and does not at all affect z_y. Why do you think this is the case? Even when the other two losses are combined the results are very similar to L_p only.
  • While it is appreciated that the authors are up-front that the separation of transformation/processed embedding is inherently partial, some of the results are a bit overstated along these lines. The UMAP plot specifically I found to be not the most convincing, in particular "z_y forms tight clusters by instrument, while z_T largely disperses" - even qualitatively this is a bit overstated. Referencing Table 2, source cls across z_t and z_y for the full loss combination (used for UMAP), the margin between the two is relatively small, and z_T still certainly encodes a lot of source information based on this probing result, so this weakens the claims from the UMAP plot a bit. I wonder if some clustering metrics could strengthen the UMAP claims a bit.
  • Something I found slightly strange is that z_T and z_y are not "equally" learned within the proposed framework, so treating them as such is potentially not a fair comparison. What I mean by this is z_y is computed from the frozen VAE of SAO, plus a trained transformer encoder on top of that (and only affected by L_y), while z_T is derived from z_y and trained using all of the objectives (when active). Much more learning is focused in developing z_T than z_y here, which should be acknowledged in the framing of the method.

Minor comments:

  • Some pieces of Table 1/2 could potentially be presented as figures rather than large tables. For example, in Table 1, varying chain length presents a nice opportunity for a figure such as a bar/line chart with chain length on the x-axis. The tables are a bit hard to digest back to back.
  • The blind vs. non-blind setting is briefly introduced in 2.1, but could be re-mentioned as related results are presented (e.g. Table 3/4.6) as at that point the distinction was no longer clear.
  • Reference [41] is "FlashOptim", correct in the references but not in the mention in 4.1 - mentioned as "FlashAdamW".
  • Figure 1 makes it look as though E_A is frozen SAO encoder alone, would be good to include that this also has the trainable transformer encoder layers.
  • Line 309 in 3.2 could start a sub-section on the loss ablation specifically, to break this up a bit.
  • 4.6 felt a bit like an after-thought, or should be included within 4.4 as the results are in Table 3, which is explained at the end of 4.4. Maybe start another section for Table 3 instead and merge in 4.6.
Review 2:

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

A framework combining contrastive, descriptive, and equivariant losses to learn dual audio embeddings tailored specifically to audio effects and processing chains.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This paper studies representation learning for audio transformations and audio effect chains. It proposes three objectives processing consistency, description alignment, and forward predictio, to learn two embeddings: z_T, which should organize audio by the applied transformation, and z_y, which should retain both source and processing information. The embeddings are evaluated through retrieval, processor/category and parameter probes, source leakage analysis, style transfer, and blind/non blind comparisons.

The distinction between “the transformation” and “the processed audio” is valuable, but some claims are stronger than the evidence supports. The external baseline comparison is not cleanly controlled and the style-transfer evaluation lacks some perceptual evidence.

Strengths The separation between z_T and z_y is one of the strongest parts of the paper. It is reasonable that distance-based tasks prefer an embedding organized by processing, while probe based tasks may prefer a richer embedding that keeps source information.

The paper compares all non-trivial combinations of L_T, L_P, and L_y. The results reveal meaningful trade-offs.

A library of roughly 110 processors across EQ, filters, dynamics, distortion, reverb, delay, and modulation is valuable.

The source classification analysis and cross content retrieval conditions help separate processing geometry from content leakage.

Weaknesses Baseline comparisons are not equivalent comparisons. The proposed system differs from prior systems in architecture, frozen SAO frontend, processor library, synthetic data generation, hard-negative batch construction, and supervision signal. This is especially important for L_P meaning: when active, the model is trained using processor type and normalized parameter information. This is a stronger signal than blind contrastive learning from audio alone. Comprisons against blind methods such as Fx-Encoder and Fx-Encoder++ should therefore be separated from description supervised or parameter-informed models.

“Equivariance” is stronger than what is demonstrated. L_y is described as equivariance via forward prediction. However, the implementation is closer to action conditioned latent prediction. Which means given an input embedding and a processing embedding, predict the processed audio embedding. The paper does not show stronger equivariant properties such as composition, inverse consistency, repeated transformation consistency, or a shared transformation direction across sources. The authors should either use the more modest term “forward prediction” or add experiments showing compositional structure.

Chain length results may not prove compositional understanding. The paper presents improved retrieval accuracy for longer chains as evidence that the representations scale to more complex processing. However, longer chains may simply leave a larger and more distinctive acoustic footprint. A six-effect chain can be easier to distinguish than a single subtle EQ, compressor, or saturation effect. Stronger evidence would include unseen processor orderings or varying one parameter while the rest of the chain is fixed.

Overall, I lean toward Weak Accept. The paper addresses an important problem, proposes a useful distinction between transformation and processed audio embeddings, and provides a strong internal ablation study. However, some of the broader claims about superiority to prior work, equivariance, and compositional generalization are stronger than the current evidence supports, and the external baseline comparisons are not fully controlled enough to isolate the source of the gains in my opinion.

Review 3:

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Different training objectives and configurations for transformation representations lead to distinct geometry and clustering of the latent embeddings. Processing-focused embeddings perform better on distance-based retrieval tasks while content embeddings are better for probing.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

The paper studies representation learning for audio processing and transformations. It frames the problem around two complementary embeddings: a transformation embedding (z_T) that captures the processing done on the audio, and a processed-audio embedding (z_y) which retains both the processing and content. Three training objectives are proposed, including processing consistency (L_T), description alignment (L_P), and equivariance (L_y). The framework is evaluated on episodic retrieval, MLP probes, an embedding analysis, style transfer, and a blind/non-blind pretraining comparison. All seven non-trivial combinations of the objectives are ablated, and the approach is reported to outperform prior baselines.

Strengths

  1. The z_y cross-content collapse (table 3) is a sharp result: the failure of z_y on cross condition retrieval directly supports the central insight on the distinction between geometry and recoverability.
  2. The style transfer experiment extend results to concrete applications: using embedding cosine distance on z_T as optimization objective leads to lower MRSTFT, which further demonstrate the robustness of the embedding geometry.
  3. The non-blind comparison is a valuable diagnosis: it identifies source estimation as the main bottleneck and explains the performance degradation in source-mismatch conditions seen across the paper. A useful pointer for future work.
  4. Open-source commitment and reproducibility: the processor library could be a good resource for the community and people working in this area.

Major weaknesses

  1. Why L_T and L_P are both needed is unclear and buried in the content: on paper these two losses seem redundant because L_P already provides the cross-source consistency that L_T targets. And in the results we do see that L_P outperforms L_T in many tasks. I think what really makes them distinct is the candidate sets, which provides different contrastive supervision and is discussed in 4.2.
  2. Source distribution is the dominant factor in retrieval, but understated in the discussion: in table 1, the proposed methods perform significantly better when source audio is seen. It is therefore unclear how much the model's performance is dependent on data domain. The comparison with other models in the same columns is also misleading because P/A/T are defined with respect to the paper's training data and don't apply directly to others.
  3. Key claims are asserted before justification: the paper repeatedly states a conclusion and supply justifications much later. For example, the distinction between the embedding's geometry and probe recoverability is central to justify the decisions to use z_T for retrieval and z_y for probing, yet it's only discussed in section 4.4 after presenting all main results.
  4. Statistics are only reported for style transfer: retrieval, probe, and embedding comparison are all missing variance estimates, while several claims rest on small differences.
  5. Missing specifications: the non-blind variant is only described as "additionally conditioned on z_x" (line 113). How z_y and z_x are combined and used as input to the inverse predictor, and whether there are any architecture changes, are not stated. This directly affects reproducibility of table 3, which has some of the strongest results of the paper.

Minor weaknesses

  1. Figure 1 is overloaded: it is unclear that z_T+ comes from a positive view that shares the same processing but different source.
  2. The stronger configurations are more content-leaking: this is slightly contradicting the purpose of the transformation embedding and is worth some clarification.

Overall this paper presents a clean conceptual contribution through a representation learning framework for audio processing. The main issue is the structure and organization of its content, which makes it a bit tough to read. Therefore I am leaning accept, conditioned on necessary reorganization and clarification.

Post-Review Updates:
  • Split the original episodic-retrieval table into a source-condition table, a chain-length figure, and a generalization analysis table.
  • Clarified the distinct roles and interactions of the three objectives, particularly the differences between processing consistency and description alignment.
  • Added silhouette scores to the embedding analysis, moderated the UMAP claims, and clarified the distinction between geometric organization and probe-based recoverability.
  • Expanded and reorganized the blind/non-blind discussion, including the non-blind architecture, test-time input estimation, and oracle comparison.
  • Added a public code link and camera-ready metadata.
  • Polished the text and presentation in general.