Current real-time AI improvisation systems lack the communication awareness human musicians rely on: rather than treating communication as a foundational algorithm design concern, most systems layer interaction strategies post-hoc onto generative algorithms through explicit controls and predefined modes. This gap persists in part because no formalized, machine-readable communication model with musicians exists. To address this, we study how expert musicians communicate in free (non-idiomatic) improvisation, unconstrained by prior discussion or agreement. Through a collaborative co-design process with expert improvisers, we derive a communication model that (1) captures how free improvisers negotiate musical ideas and enter stable musical spaces, and (2) is formalized as a machine-readable annotation scheme. We further present the H2H (Human-to-Human) Music Improvisation dataset: six hours of audio-visual expert duo improvisations with clean per-player stems and per-player annotations of both their own intentions and their perception of their partner's intentions. To our knowledge, this is the first such dataset for free improvisation. Together, the communication model and the dataset offer a new lens and resource for studying musician communication and may in future inform the design of AI musical partners that communicate by design.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

It is possible to model free music improvisation starting from human-to-human interaction observations.

Q23. Main review and comments for the authors (to be completed before the discussion phase). Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

This paper presents a communication model about improvisation and an H2H (Human-to-Human) Music Improvisation dataset. The contributions were developed through a co-design process with five expert improvisers. The model proposes a sequence of negotiation, proposal, and stability states, suitable for free improvisation and for use as a machine-readable annotation scheme. The dataset provides six hours of recordings with per-player stems and per-player intention labels. The findings of the study indicated that the stability status is agreed upon between paired musicians, but not for proposal and negotiation. The paper argues that this is expected due to the gap between perception and intention.

Strengths

• Original piece of work.
• Well anonymised!
• Clear contributions: 1) a communication model about improvisation as a machine-readable annotation scheme; and 2) H2H (Human-to-Human) Music Improv dataset: an annotated multi-modal human-to-human improvisation dataset with per-player stems and per-player intention labels. Computed two alignment metrics: overall alignment and per-state alignment.

Weaknesses

• Overall: Lack of methodological detail and arguably lack of diversity in the sample. Were there any women involved in this study, for example?
• Only 15 references.
• There is missing the AI usage statement.
• The reference style should be revised.
• [L55-57] The connection between the gap and the need for a machine-readable communication model and an annotated dataset is weak. There should be a clearer justification of why these two particular assets are relevant.
• When citing the SOMAX system, it's better practice only to cite one paper, not three: "L105 [7, 8, 10]"
• Section 3 - Communication model: Is the co-design process based on any particular methodology or theory? Providing a bit more detail on this can help others implement a similar approach (or inquire about the proposed approach).
• L205-207: "discussed across several sessions and between researchers from the improvisation practice, machine learning, and HCI," => how many sessions and how many researchers?
• L216: "subjected to thematic analysis;" => Please provide the reference by Clarke and Braun.
• L295: "All recruited musicians" => How many?
• L317-319: "As one co-designer put it, 'everything that you ask the player to do that’s not simply playing the music is going to change the outcome of the music.'" => If this is a quote, it should be clearer who said this.
• Section 4.2: Providing a bit more information about the profile of the improvisers can help understand what the biases were, which should be made explicit: was it a diverse group of improvisers (i.e. age, gender, geography...) or was it a homogeneous cluster? These limitations/biases should be acknowledged to contextualise the study and dataset better.
• L398-L400: "We transcribed recordings from the co-design sessions, including pre- and post-play discussions, and conducted the matic analyses." => This sentence should be moved to section 3.1. It is unconventional to talk about the data analysis in the discussion section.

Typos

• L108-109 "jam_bot fine-tunes transformer-based" => ? This sentence looks unfinished.
• The reference style should be revised.

Q25. Meta-review and final comments for authors (to be completed after the discussion phase)

Strenghts

The paper relates to an audio-visual dataset for music improvisation, with strengths including:

• Well-written paper (R1).
• Provides a new dataset including audio, video and formal annotations (R1, R2, R3).
• The quality of the recording of the dataset (R3).

Major Criticisms and Areas for Improvement

Reviewers identify critical gaps in framing the contribution and contextualisation of free jazz improvisation, in particular:

• Overstatement of the implications of the paper's contribution (R2).
• The proposed communication model simplifies a complex and rich activity (R2).
• The application of the annotation scheme is unclear (R2).
• Incomplete literature review (R3).
• The dataset only covers duos and is small (R3).
• The ethics statement is lacking (R3, MR).

Recommendations and Overall Evaluation

Reviewers suggest the following changes:

• Include some basic signal analysis (R1).
• Contextualise free jazz improvisation better (R3).
• Add more methodological detail and justify why there is a lack of gender diversity in the sample (MR).
• Include the AI usage statement (R3, MR).

Addressing the comments above, as detailed in the reviews, can strengthen the paper considerably.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Creating a dataset about free improvisation, annotated by the musicians themselves, creates some surprising insights around disagreement and provides data for future research.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

The paper is well written and provides a new dataset including audio, video and formal annotations of simple free improvisation settings.

The dataset is welcome. The paper could have been more interesting with some basic signal analysis beyond the simplistic agreement analysis provided.

As I have expressed in the literature section, I believe that the paper paints itself into a corner as it doesn't describe how the free improvisation fits in the wider world of music making.

Some of these shortcomings, especially the literature and framing can easily be added because there's also some redundancy in the papers introduction that could be dropped.

In summary, I'm not excited, but appreciative.

Some detailed comments:

27: throughout history ... always (redundant) 64: "as an interdisciplinary group..." reads journalistic; I recommend not infusing a scientific publication with personal biographical details 86: introduction is repetitive: you already said what you're proposing in the abstract and the paragraphs above, only to repeat it once more in the numbered bullets 117: does it matter that it's earlier? what is fundamental about it? 119: not clear whether this is an actual implementation (like the ones above) or "just" a model. It seems it's just a model, as you use the word "theorise" 160: section 2 as a whole doesn't convince me that there is something such as "free improvisation" 373 etc: why is it "sec." and not just time? Figure 4: text is too small

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

Model for annotation of free improvisation procedure, and a preliminary dataset with video + annotations.

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

The paper addresses a relevant and timely problem: the lack of communication-aware design principles in real-time AI improvisation systems. The proposed H2H Music Improv dataset is potentially valuable, especially because it includes expert duo improvisations, clean per-player stems, and annotations of both self-intended and perceived partner intentions. This type of material could become a useful resource for the MIR, computational creativity, and human-AI music interaction communities.

However, I find that the paper somewhat overstates the implications of its contribution. The abstract and framing suggest that the work provides a foundation for overcoming important limitations of current AI improvisation systems, but the paper does not actually propose, implement, or evaluate an AI system using the model. As a result, the connection between the communication model and concrete algorithmic design remains somehow speculative. The proposed communication model is also rather coarse-grained. Reducing the interactional dynamics of free improvisation to stages such as initialization, negotiation, and stability may be useful as a first descriptive framework, but it simplifies the richness, ambiguity, and simultaneity of communication in free improvisation. Similarly, while the annotation scheme is presented as machine-readable, it is not fully clear how directly it can support computational modelling beyond supervised classification of relatively broad interactional states. The authors should clarify whether the main contribution is intended to be a theoretical model of musician communication, an annotation framework, a dataset, or a practical step toward AI improvisation systems. At present, the strongest contribution appears to be the dataset, as the broader claims about AI systems are underdeveloped. Despite this, the dataset and annotation framework provide an initial empirical resource for studying communicative dynamics in expert free improvisation, which may in future inform the design of interactive AI systems.

Q18. Write ONE line (in your own words) with the main take-home message from the paper.

The authors present an audio-visual dataset of two-party, call-and-response free improvisation with intention annotations

Q23. Main review and comments for the authors. Please summarize strengths and weaknesses of the paper. It is essential that you justify the reason for the overall evaluation score in detail. Keep in mind that belittling or sarcastic comments are not appropriate.

Overall review: I am recommending weak accept purely because of the dataset; the musicians who participated in this study appear to be knowledgable improvisers, and it would be useful for the community to have access to their recordings. The methods for data gathering were scientific and clean, and that reporting was both necessary and helpful. I note some weaknesses with the paper writing in my review limitations section. If accepted, I would encourage the authors to spend time fairly placing their definition of free jazz in the context of other practices.

Strengths: This is an interesting and well curated dataset. I listened to some of the recordings, and I heard clear demonstrated proficiency with free practice. Especially in the current generative tech climate, ethically recording this dataset is an impressive achievement. The care taken to ensure that the sound quality is as pure as possible is appreciated, and the recording set up is reasonable. I was happy to read that the musicians were still able to see each other through glass; visual communication is necessary for collective improvisation. I am confident that this dataset will enable numerous interesting studies in computational musicology and music information retrieval.

Limitations

The related work section lacks both ends of the computer-music jamming literature spectrum (classic AI to gen AI). I would encourage the authors to cite Arne Eigenfeldt’s Musebot work for the classical, rules based agentic AI approach, and Yusong Wu’s work (Realchords, Realjam, etc) for the generative AI side.

The improvisation sessions only cover musician pairs, which severely limits the sonic fabric which has been created and recorded. I would have loved to see a session with all five musicians present, and any other configuration, including solo musician work, would have significantly strengthened this contribution.

I understand that the musicians had to be in separate rooms to control for audio bleed, but having the ablated study where some of the recordings were in separate rooms with the glass and low latency setup, and some of the recordings were in the same room would have been fascinating to study (how do musicians change their interactions when they are breathing the same air and occupying the same acoustic space?)

The paper talks about AI music, but the dataset is small relative to the general datasets for generative tasks (this point does not negate my strength point for MIR tasks). For instance, MusicGen (2023) was trained on 20,000 hours of audio. Additionally, some of the related work AI music system are symbolic based, rather than audio based. This dataset is audio-based, and would, in my opinion, be severely weakened by audio-to-midi conversion.

The analysis of free jazz in sections 1, 2, and 3 does not do a sufficient job of placing this musical framework in the general free jazz practice. I am personally familiar with Anthony Braxton’s work, which structures improvisation not around melodies but around “language types” (long sound, trills, angular attacks, intervallic forming, etc). One good source to reference is Heretic: Modeling Anthony Braxton's Language Music (Brown et al, MMRP, 2019), or Hunter Brown’s masters thesis the same year. Additionally, there is some discussion of roles in section 5.1.3, but the discussion is limited to the idea of “leading and following”, without mentioning other role based music, including other concrete roles (soloist, contrarian, etc) and abstract roles such as character embodiment. The risk with not acknowledging the deep practice of free jazz is that the paper reads like an anthropological study, where researchers observe practitioners without they themselves participating in the practice. I will note that the study participants demonstrate clear understand the free jazz practice, both in their playing and their report descriptions, including “gestures”, “energy”, and “texture” as part of their process in section 5.1.1.

The ethics statement is lacking. The most important aspect (IRB approval and compensation) is covered, but there is no discussion on the ethics of studying free jazz. Free jazz has a deep history that ought to be respected and acknowledged. I personally know free jazz musicians who would be extremely opposed to any studies taking place relating to their practice. The benefit (in my opinion: we can study an abstracted method of human communication and the methods through which augmented sonic communication practices are possible) must outweigh the cost (potential appropriation, to name one).

Limitations in the provided audio: I listened to around 10 minutes across the four available tracks.

• I only heard melodic and chordal structure, very little if any experimentation with general sound or extended techniques.
• The call and response framework the musicians are working with encourages them to have a response to every idea that they hear, so I did not hear any part of the tracks that involved intentional silence.
• The musicians did not play with the space, whether due to video anonymization or the studio setup (dancing, acoustic space manipulation, etc).

Nitpicks: Line 46 “This” is an unclear pronoun reference Line 190 Tense change; should “is” be “was”? Line 310-311 Tense change, should this be “annotated”

We thank the reviewers and meta-reviewer for their constructive comments. This summary documents the changes made after the review phase.

Main revisions: We moved the contribution list from bullet points into the closing paragraphs of the Introduction, and clarified in the abstract and Introduction that the paper's main contributions are the communication model and the audio-visual dataset, offered as a resource for future system design. We better motivated the scope of free improvisation within collaborative music making, and broadened related work to include more agent-based systems.

Specific changes:

• Situated free improvisation within collaborative music making and motivated our focus (R1).
• Placed Voyager within the multi-agent landscape and clarified it is an implemented system (R1, R3).
• Rephrased the abstract and Introduction to scope the contributions to the model and dataset (R2).
• Expanded related work to include Heretic and the Musebot ensemble as agent-based systems (R3).
• Justified the duet configuration and separate-room setup (R3).
• Added HCI session details, musician demographics, and HCI references (MR).
• Updated the reference style and added the AI usage statement (MR, R3).
• Addressed all proofreading comments.

On ethics: we acknowledge R3's concern about the cultural history surrounding free jazz. Our study, however, concerns broadly free (non-idiomatic) improvisation, not only limited to free jazz. Our participants are active free-improvisation practitioners who framed their own work in these terms. All gave informed consent to the recording and public release of their performances. We take this consent, and the practitioners' own framing, as the basis for the study's ethical standing.

We see suggestions such as dataset signal analysis and a same-room ablation as promising directions for future work.