We present MIDI-LLM, a recipe that improves multitrack text-to-MIDI generation via adapting Large Language Models (LLMs). MIDI-LLM expands an LLM's text vocabulary to include MIDI tokens and employs a two-stage training pipeline: (i) unimodal continued pretraining on music-adjacent text and standalone MIDIs, and (ii) multimodal supervised finetuning on text–MIDI pairs. Our instantiation of MIDI-LLM based on Llama 3.2 (1B) outperforms the recent Text2midi model in both text control and musical quality, and readily integrates with optimized inference ecosystems like vLLM. To align with real-world songwriting workflows, we further finetune our MIDI-LLM on the TheoryTab dataset for text-conditioned lead sheet (i.e., melody + chords) generation and infilling. A comprehensive ablation study validates the synergy between LLM text pretraining, standalone MIDI pretraining, and supervised text-to-MIDI finetuning. Finally, an in-the-wild blind user study conducted in a real-world creative workflow at scale with 58 Hookpad Aria users and 4,002 generated outputs demonstrates that our MIDI-LLM achieves the highest acceptance rate in zero-to-one lead sheet generation over baselines without text control or LLM pretraining, confirming its efficacy in human-AI music co-creation.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Implementation, documentation, and extensive evaluation of a text-to-midi generation model.

Q23. Main review and comments for the authors (to be completed before the discussion phase). Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This is an ambitious paper, documenting the construction of a text-to-midi generative model from pretraining, to posttraining, to evaluation of a variety of downstream tasks. One high-level critique of this paper is that because it does so much, there isn't a lot of space to elaborate on the nuances of each component of the work. Overall I think the work is very well done, and most of the challenge is about fitting it all in to the 6 page format. I do think the paper makes pretty good decisions about what to emphasize within the space constraints.

The are a lot of methodological decisions that aren't subjected to scientific much scrutiny, but they mostly look reasonable and defensible to me. I notice that some details are missing. E.g., I don't see mention of the context length without which it's a bit hard to make sense of the batch sizes described in Section 4.2.1. I also don't see how (text, midi) sequences are prepared and batched (sequence packing; padding)? If there are plans to release code, that mitigates my remaining concerns about the details.

I am pretty skeptical of the quality of MidiCaps annotations as a link between the text and music modalities. But revisiting the construction of datasets for text-to-midi modeling is out of scope for the present work. MidiCaps is the natural choice right now for this type of multimodal modeling work, for all its faults.

Structurally, I'm a bit confused about the organization of Sections 2, 3, and 4. I'm not sure why tokenization deserves its own section, or what the principle is behind what content goes into Section 3 vs Section 4. But this doesn't materially detract from the readability of the paper.

I am curious about the quality of the model fit on standalone MIDIs. With less that 2B midi tokens, I'd expect to Llama 3.2 1B to overfit. Maybe 25K steps is not enough to see overfitting, but in that case I wonder whether the model is underfit. It's a bit hard for me to interpret the LM Loss numbers in Table 3; are these loss on mixtures of text + midi? In that case, I'd expect LM Loss under "No PT" ablation and "Text PT" ablation to be terrible (much worse than reported) since the midi head in those ablations is random initialization, so I must not understand what these numbers mean. Since you're using the same vocabulary as anticipatory music transformer, it seems like it should be possible to directly compare loglosses (modulo possible issues with different train/test splits)?

Qn: how does 1B pretrained Llama model become 1.5B midi-LLM (Table 2)?

The FAD metrics also work towards midi generator quality, but the comparison is only to Text2Midi (in principle you could compare to other Midi generators right?). I'm also not sure I believe in FAD on synthesized audio as a measure of midi quality: it seems almost like an adversarial attack on this metric, which I'd expect to be more sensitive to local timbre (constant, since all models use the same synthesizer) and might miss the longer-term structural elements.

All that said about model quality, I've listened to the demos and clearly they are doing something reasonable. The focus of this paper isn't about producing the best midi generator; it is about producing a multi-modal music generator.

The evaluation protocols are very thorough. Comprehensive automatic evaluations, broad set of tasks, and corroboration with human judgement.

Q25. Meta-review and final comments for authors (to be completed after the discussion phase)

I think this paper is great: ambitious, practical recipe, strong headline results, thorough ablations, well-written. Concerns raised by reviewers are relatively minor concerns about the strength of certain experimental results and how to interpret baseline comparisons; it is tough to do apples-to-apples comparisons in this space (e.g., as R1 notes, parameter-matching doesn't make much sense when the point of this work is to exploit existing large LM infrastructure). Overall the experiments are extensive and well-executed.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Adapting a standard LLM with MIDI tokens and staged text/MIDI training yields a practical text-controllable MIDI generator, especially useful for musical ideation.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

Summary

This paper presents MIDI-LLM, a recipe for adapting a standard large language model to text-conditioned MIDI generation and infilling. The method expands the LLM vocabulary with AMT-style MIDI tokens, continues pretraining on music-adjacent text and standalone MIDI, and then performs supervised finetuning on paired text-MIDI data. The resulting model is further adapted to TheoryTab lead-sheet generation and evaluated through offline metrics, ablations, CFG analysis, inference benchmarking, and an in-the-wild user study. The paper addresses an important gap between symbolic music generation and modern text-conditioned generative modeling, and it does so with a practical design that benefits from existing LLM serving infrastructure. The empirical results are strong, the ablation study is informative, and the user study gives the work a realistic creative-workflow grounding. I believe the paper merits acceptance, although several additional baselines and scaling analyses would make the claims more complete.

Strengths

The central modeling choice is practical and well motivated. Instead of introducing a specialized symbolic-music architecture, the paper adapts a standard LLM by expanding its vocabulary with MIDI tokens. This makes the contribution useful beyond the specific model instance, because it connects text-to-MIDI generation with the broader LLM training and serving ecosystem. The inference results in Table 2 support this point convincingly: despite using a larger model than Text2midi, MIDI-LLM achieves much higher real-time factors, improving from 0.56 to 3.33 at batch size 1 and from 1.06 to 11.48 at batch size 4, with FP8 further improving throughput. This makes the infrastructure argument more than a convenience claim.

The comparison with Text2midi shows large empirical gains on the MidiCaps + LMD setting. MIDI-LLM improves FAD from 0.818 to 0.173 and CLAP from 0.187 to 0.221 in bf16, suggesting gains in both musical quality and text alignment. While there are some confounds in this comparison, the magnitude of the improvement is substantial and establishes MIDI-LLM as a strong text-to-MIDI system.

The ablation study is one of the strongest parts of the paper. Table 3 does not merely show that adding more stages helps; it gives a useful picture of the roles played by different sources of pretraining. Llama text pretraining improves several controllability measures, especially some music-theoretical attribute controls, but can underperform on multiple music-quality metrics. MIDI-focused pretraining improves quality-related metrics but tends to weaken or destabilize controllability. The full MIDI-LLM recipe achieves the lowest validation NLL and a strong overall balance across quality and control, even though it does not dominate every individual metric. This supports the authors' claim that text knowledge, MIDI-domain modeling, and paired text-MIDI alignment are complementary.

The paper also provides useful analysis beyond aggregate offline metrics. The CFG study in Fig. 2 is particularly insightful: from-scratch generation benefits from moderate CFG, whereas infilling requires much stronger guidance because the surrounding MIDI context can dominate the text prompt. The cross-prompt rigidity analysis is a good diagnostic, showing that without CFG, a substantial fraction of time-aligned infill bars can share the same chordal chroma profile under different text prompts. This analysis helps explain why text control is easier in zero-to-one generation than in context-constrained infilling. The in-the-wild user study is a valuable addition. Evaluating 58 users and 4,002 generated suggestions in an actual lead-sheet co-creation interface gives the paper a stronger claim about practical utility than offline metrics alone would allow.

Weakness

The ablation design is informative, but it still leaves some important stage-combination questions unresolved. In particular, the table does not clearly include a baseline that starts from the pretrained Llama model and applies only paired text-MIDI SFT, without the standalone MIDI continued-pretraining stage. This would be a useful comparison because it would test whether large-scale paired text-MIDI data is sufficient once the model already has strong text knowledge.

The external system comparison is strong but not fully controlled. MIDI-LLM is around 1.5B parameters, while Text2midi is around 0.3B parameters. This fivefold scale difference makes it difficult to attribute the gains solely to the proposed training recipe, tokenization choice, or LLM compatibility. I do not think this invalidates the result, since the paper's practical goal is partly to exploit existing LLM infrastructure, but it does limit the causal interpretation of Table 2. A smaller MIDI-LLM model, or a larger Text2midi-style baseline if available, would make the comparison more decisive.

Suggestions

A related suggestion is to discuss scaling more explicitly, especially because the motivation for using an LLM is not only larger capacity but also the possibility of scale-dependent or emergent abilities. The current results show that a 1.5B MIDI-LLM works well, but they do not tell us whether larger LLM backbones would merely improve FAD/CLAP gradually, or whether they would unlock qualitatively different behavior such as stronger long-range musical planning, better interpretation of abstract style descriptions, more reliable instruction following under multiple simultaneous controls, or better resolution of conflicts between text prompts and MIDI context. These are precisely the kinds of abilities one might expect to matter for text-conditioned symbolic composition. Even a limited scaling analysis, such as comparing smaller and larger LLM backbones or varying the amount of MIDI continued pretraining, would help determine whether the gains are primarily architectural, data-driven, scale-driven, or tied to emergent text-to-music reasoning abilities. This matters because symbolic music datasets are much smaller than general text corpora, and downstream TheoryTab finetuning is only 34M tokens. It would be useful to know whether MIDI-LLM continues to benefit from larger text-pretrained models, or whether the approach quickly becomes bottlenecked by available MIDI and paired text-MIDI data.

Overall Assessment

This is a strong and practically relevant paper. Its main contribution is not just adapting an LLM to MIDI tokens, but showing that this adaptation can produce better text-to-MIDI generation, efficient inference through existing LLM serving tools, and useful behavior in a real creative interface. The paper is also unusually clear in diagnosing where text control works well and where it remains difficult, especially in infilling.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

The work constructed a new MIDI LLM for human-AI music collaboration.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

Strengths:

• The paper introduced a new MIDI LLM.
• The experimental details and demos are rich to reproduction and understanding.

Weakness:

• The relevant baselines are lacking, such as MIDI-GPT, ChatMusician, NotaGen, etc. Comparing MIDI-LLM (1.5B) only with a smaller model Text2midi (0.3B) is not enough and robust to illustrate its effectiveness.
• The speedup largely comes from inference acceleration framework vLLM, and lower precision fp8 and bf16, which any LLM-shaped model gets for free. So it says more about the LLM serving ecosystem than about the proposed method. A fairer comparison would optimize the Text2midi baseline too, or at least separate the algorithmic contribution from the engineering one.
• Qwen2.5-Omni-augmented infilling examples are not varified by humans.
• The user-study result of From Scratch generation is not very strong (only p<0.1).
• Acceptance rate is coarse and does not reveal the necessary detailed evaluation. More fine-grained subjective ratings, such as text adherence, musical coherence, and novelty, are necessary for evaluation.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Adapting existing LLMs with a MIDI vocabulary and a two-stage training pipeline yields an efficient text-to-MIDI model that enhances zero-to-one musical ideation for human creators.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This paper presents MIDI-LLM, a recipe for adapting a pretrained language model to text-to-MIDI generation and infilling. The model expands the LLM vocabulary with AMT-style MIDI tokens, continues training on music-related text and standalone MIDI, and then finetunes on paired text-MIDI data. The authors evaluate it on text-to-multitrack MIDI generation, specialize it for TheoryTab lead-sheet tasks, and test it in an in-the-wild co-creation study. The paper is well motivated and highly relevant to ISMIR. It addresses the important problem of making symbolic music generation more controllable through text while preserving the editability of MIDI. The proposed method is not conceptually complex, but it is practical and well executed. The comparison with Text2midi supports the main claim, showing improved quality, stronger text control, and faster inference through existing LLM serving infrastructure.

A key strength is the ablation study. The authors do not only report the final model, but also examine how different training stages affect musical quality and text controllability. The results suggest that the gains come from combining language knowledge, MIDI pretraining, and paired text-MIDI supervision. The analysis of classifier-free guidance is also useful, especially the observation that text control is harder during infilling because the surrounding MIDI context can dominate the prompt. The user study is another valuable contribution. It evaluates the system in a realistic lead-sheet co-creation workflow with 58 users and 4,002 generated suggestions. MIDI-LLM performs best for from-scratch generation, which supports the claim that text control is especially useful for early-stage musical ideation.

The user study results are mixed. In the infilling task, the AMT baseline remains competitive and is stronger in several cases. The authors provide reasonable explanations, such as possible conflict between text prompts and the existing musical context, but these explanations are not directly tested. Some evaluation details could also be strengthened. The quantitative comparison uses FAD and CLAP, which require rendering the generated MIDI into audio. This adds an extra variable, since the metrics may partly reflect the quality of the synthesizer rather than the symbolic composition itself. The TheoryTab symbolic metrics are reasonable, but they are still indirect measures of musical usefulness. The controllability results would also be easier to interpret with more detail about the custom genre and mood classifiers.

Overall, I recommend weak accept. The paper is clearly written and useful for the MIR community. While the method mainly adapts existing ideas rather than introducing a fundamentally new modeling paradigm, the execution is strong and the evaluation is broad.

Added links to open-source materials, including live demo, model checkpoint, and inference code.

De-anonymized Hooktheory / Hookpad user study context.