Synthesizer inversion is challenging for two main reasons: 1) Distinct parameter configurations can produce perceptually similar sounds. 2) Parameter-space losses often fail to reflect rendered audio similarity, while the synthesizer being a non-differentiable black box prevents simple audio-domain supervision. To address the one-to-many mapping induced by the first challenge, we formulate synthesizer inversion as conditional generation over discrete synthesizer parameters and use masked discrete diffusion as the generator. This treatment additionally avoids the fixed-order assumption of autoregressive models and the continuous-relaxation mismatch of flow matching when modeling categorical synthesizer controls. To address the second challenge, we further fine-tune the model with GRPO-style audio-domain rewards computed from rendered outputs. Experiments on Dexed show that, after supervised training, the discrete diffusion model is competitive with autoregressive and flow-matching baselines, and reward-based fine-tuning further improves out-of-domain audio matching performance. Code and demos are available at: https://github.com/DDSynth-RL/DDSynthRL.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

DDSynth-RL is treating Dexed synthesizer inversion as masked discrete diffusion over discrete parameter/MIDI tokens and then fine-tunes it with GRPO post training from the non-differentiable black-box renderer.

Q23. Main review and comments for the authors (to be completed before the discussion phase). Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

The paper makes a clean, well-motivated contribution. Based on the intrinsic characteristic of the problem, they apply masked discrete diffusion to synthesizer parameter inversion. The contribution stands on the discrete diffusion framing. the RL stage is a promising but currently underperform baselines. The experimental validation is solid and comprehensive.

Strength: (1) The one-to-many + non-differentiable-renderer characterization cleanly motivates a generative formulation plus reward-based fine-tuning. (2) Discrete diffusion beats flow matching and matches a tuned ordered AR model (probably beats random order AR model)

Weakness: The major issue I have with the paper is on the experimental results on GRPO: Table 1 results shows that it degrades every in-domain metric compared to Stage 1 base model significantly. And for out of domain results, it flipped. The author does not explain why this happen. The author does not provide the details on the dataset for GRPO training either (The only detail maybe only the use of audio renderer for audio rewards). It make me wonder do they use Nsynth audio data as targets for GRPO fine tuning. Hope the authors can address this or provide evidence or explanation on this performance.

Another experimental issue is the FM backbones, in section 4.2.3, they use AST style transformer, where as AR and DD shares the same transformer, which can cause performance difference, details on the backbones are missing here.

Minor issue: Does discrete diffusion beat random order AR? It seems so for wMFCC and CLAP from figure 3. Probably can add random order AR into table 1.

Q25. Meta-review and final comments for authors (to be completed after the discussion phase)

The paper is a clear, well-motivated contribution with strong reproducibility: it formulates synthesizer inversion as masked discrete diffusion over discrete synthesizer-parameter tokens and adds a GRPO fine-tuning stage.

Contributions: (1) The discrete-diffusion formulation addresses the task's one-to-many nature and avoids the autoregressive fixed-order assumption. (2) A properly-tuned autoregressive baseline with an order ablation that fills a gap in the literature. (3) Out-of-domain gains from GRPO fine-tuning.

Issues: (1) GRPO is applied only to the discrete-diffusion model, so the RL gains cannot be separated from the model choice, and the "discrete diffusion is better" claim is clean only against flow matching. (2) The GRPO training data is unspecified (leaving the OOD comparison unclear). (3) The post-fine-tuning in-domain degradation is unexplained. (4) the flow-matching baseline differs in encoder and architecture. (5) the GRPO objective is a non-standard variant whose prior work should be cited.

Suggestions: (1) Evaluation is objective-only, can add one listening test. (2) There are reproducibility gaps to be filled: audio encoder, loss-weight heuristic, GRPO hyperparameters. (3) Add an AR-GRPO baseline.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

This paper addresses the task of synthesizer inversion, building on recent generative work on the task. In particular, it focuses on dealing with the task's intrinsic ill-posedness using a masked discrete diffusion strategy, and improving robustness to out-of-distribution audio using a GRPO fine-tuning procedure.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This paper addresses the task of synthesizer inversion, building on recent generative work on the task. In particular, it focuses on dealing with the task's intrinsic ill-posedness using a masked discrete diffusion strategy, and improving robustness to out-of-distribution audio using a GRPO fine-tuning procedure. The result is a slight in-domain improvement over existing methods and a strong out-of-domain improvement. The work also fills in a gap in the literature by training an autoregressive generative model over synthesizer parameters, though this is viewed as a baseline rather than the "main event" of the paper.

Broadly, my view is that this is a very strong piece of work, and I am happy to recommend it for acceptance. It takes a principled and effective approach to a longstanding problem which is receiving renewed research interest, and presents its methodology clearly and its experimentation convincingly. I have some minor gripes about certain design decisions, which I have detailed below, but these mostly do not detract from its clear strength as a scientific contribution, and so my recommendation remains a "strong accept".

Strengths:

• Strong empirical results and a clear improvement over sensible baselines
• Sensible and (mostly) well justified modeling decisions
• Ablation of autoregression order is a helpful detail that also paves the way for an interesting line of future work
• Sensible representation of synthesizer parameter estimation as a Markov Decision Process
• Smoothed target distributions are a nice way to handle naturally ordinal synthesizer parameters
• A generally solid framing of the task and motivations for researching it. Particularly, I enjoyed that synthesizer parameters were highlighted as a symoblic substrate for research. This echos the varied and powerful analysis-by-synthesis methods which, in the deep learning era, have largely fallen out of favour.
• I agree with the paper's observation that "there is currently a lack of standard benchmark for evaluating this task". This would be a very valuable direction for future work!

Criticisms:

The proposed method, of course, requires that parameters be discrete variables. This effectively means that the majority of software synthesizers (Dexed being a notable exception, where the majority of parameters are effectively implemented as 7-bit ordinal discrete values) require discretisation of parameters in order to be used with the algorithm.

The proposed discretisation scheme for continuous parameters (25 bins per parameter) seems very coarse. I was surprised to see that it performs so well — it's nonintuitive to me that such aggressive discretisation should allow for such a close audio match. With this in mind, I wonder if an analysis of quantisation error might have been a valuable addition (e.g. what happens if we resynthesise the validation dataset using discretised parameters?) I also wonder if the authors tested the method on a synthesizer with a predominantly continuous parameter space?

Further, the paper discusses "the continuous-relaxation mismatch of flow matching when modeling categorical synthesizer controls". Yet, there is considerable work in other domains (molecule generation, tabular data generation, etc.) on mixed discrete-continuous flow matching, and frameworks for general exponential family variational flow matching, which suggest this mismatch is either not such an issue, or is effectively surmountable. Coarse discretisation of all continuous parameters, on the other hand, does certainly introduce bias. Is it possible, then, that another explanatory factor underlies these results? e.g. testing only on a synthesizer with a predominantly discrete parameter space?

Equation 2 introduces the token loss weight hyperparameters, and section 4.3 explains that these are manually tuned to increase weights for parameters that "converge slowly or have stronger influence on the audio". Heuristics are fine, of course, but it's important for reproducibility to detail them. How are these weights computed? How is "influence" on the audio evaluated, especially taking into account the gating effect some parameters exert on each other?

The GRPO algorithm's behaviour is known to vary quite widely as a function of its hyperparameters, particularly the group and batch sizes. I would recommend the authors to include these in the camera ready version.

Unless I'm mistaken, the Le Vaillant et al dataset uses a fixed FM algorithm (i.e. operator routing). I feel this is a shame! The algorithm parameter is usually framed as an obstacle in FM/Dexed parameter estimation, and thus often fixed to a pre-determined value, but given the impressive performance of this model, it would have been valuable to see if it is capable of dealing with this challenging parameter!

Dependence on hand designed presets is also perhaps a weakness of the proposed method, not least because it limits scalability of the method, as well as its application to new synthesizers that don't boast such a large community of active, preset-producing users. It would have been nice to compare this to randomly generated datapoints to ascertain whether the performance benefits of this design choice outweigh these limitations.

Despite these criticisms, I remain convinced that the paper would be a valuable contribution to this year's ISMIR conference.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Models processing Discrete token representations will achieve better results at Audio Synthesizer Inversion than models using continuous representations, and using reinforcement learning for fine-tuning will improve the accuracy on out-of-domain cases.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

The work achieved here is very good and provides a real improvement to certain weak points of previous approaches.

I do like:

• The work is reproducible thanks to its codebase and the use of available datasets/systems
• The good idea to use language models and compare auto-regressive vs non auto-regressive on this task
• The introduction and comparison/justification of the different approaches will make people new to this task better able to understand its challenges

The cons:

• I couldn't find any information regarding the audio encoder in the paper itself; I don't think it's said which encoder has been use for audio tokenization and neither why this choice. Maybe one sentence about this could be added somewhere.
• I understand the interest of the GRPO ablation which basically allow better exploration of the model on the out of domain case; but the fact that it has been applied only to the Discrete Diffusion Non-AR model despite the evaluation showing that the AR model seems to have comparable performances on In-Domain & OOD cases seems to be lacking justification. Is there a reason for not having experienced GRPO Tine-Tuning on AR Model?
• Still on this note, it feels like the conclusions kind of over-attributes the gain to discrete diffusion. Yes, GRPO improves the DD model, but because it wasn't applied to the AR model, it feels like concluding that discrete diffusion may be the better approach is true and only compared to flow matching; and that we may have missed an opportunity here to conclude on AR models too.
• In addition, the evaluation is only performed using objective metrics; a subjective experiment would have been very valuable in addition before making any kind of conclusion on the model comparison. There is even a strong motivation given in the discussion section for the necessity of such experiment, which even lead to the conclusion that "This observation suggests that human-designed preset datasets may be more suitable than randomly sampled parameter datasets for synthesizer inversion.". This is a very interesting finding and I believe that by itself, it justifies the need for such additional subjective listening evaluation.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

DDSynth-RL uses discrete diffusion to handle the non-fixed order of synthesis parameters and GRPO to incorporate audio distance information when a black-box synthesiser is used, surpassing the previous state-of-the-art flow-matching approach.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

Solid work. Huge plus for sharing the source code. Had a few questions about the model architecture and got answers after I quickly read the code. The overall quality is great enough to give a definite accept. I only have a few comments and suggestions; please see below.

A few details are unclear. Like, "the position to fill is epsilon-greedily based on normalised-entropy confidence". Would be great to explain exactly how it works. The corresponding part of the source code is also not easy to read. I had a hard time understanding what exactly equation 9 is doing, maybe because I'm not familiar with RL. The log \pi_ref doesn't do anything since it's frozen and not optimised; basically, a bias term. It doesn't look similar to the GRPO formulation I searched online. Maybe it's a variation of it. Please provide a corresponding reference, with adequate explanation. I also suggest simplifying it to the following form: L_GRPO = 1/K \sum_{k} (\beta - A_k) l_k This is much clearer, as it shows that if A_k > \beta, the corresponding \log \pi (\tau_k) will be minimised, and vice versa. The 1) criteria for designing the parameter prediction order for the AR baseline and 2) how the loss weights of the parameters are adjusted should be briefly discussed and clarified. It's very unclear from the text.

Section 4.4 uses the same audio metrics as GRPO. It might be better to move the introduction of the metrics and the references to section 3.4 and simply state that the same metrics are also used for evaluation. Talking about this, it's surprising to see that in the in-domain case, fine-tuning on those metrics actually yields worse performance on those metrics than the stage 1 baseline (Table 1). Do you have any explanation for this? In addition, it would be great to elaborate on the rationale for two-phase GRPO tuning and why only CLAP and CREPE are used in the second phase. In the listening samples, the second-phase baselines yield worse pitch predictions than the multi-reward one. Just curious about the motivation here.

Out of curiosity, have you checked the parameter prediction order from your DDSynth model? Does it follow a similar trend as your manually designed order, or not at all? Would love to see some ablation study on this.

L137: "The second direction (of discrete diffusion) comes from discrete diffusion models." This reads so awkwardly to me. Maybe give the second direction a different name? I don't have a clear answer, though.

Following the reviewers' and meta-reviewer's suggestions, we clarified the related work and contribution, added key details about the audio encoder, GRPO training data and decoding procedure, corrected the GRPO notation, and revised the discussion of the in-domain degradation and the comparison between DD and AR.

Due to the six-page limit, we could not include every implementation detail and hyperparameter. The code, configurations, and audio demos are publicly available for readers interested in further details.

The suggested additional experiments, including quantization analysis, AR-GRPO, continuous-parameter synthesizers, and subjective evaluation, are valuable directions. Due to the limited camera-ready timeline, they could not all be completed in this work, and we hope to explore them in the future.