P11-10: Human Performance in Lyric Topic Classification: Evaluation and Implications
Varvara Papazoglou, Robert Gaizauskas, Alexandros Stamatiadis
Subjects: Evaluation, datasets, and reproducibility ; Lyrics and other textual data ; MIR data fundamentals ; User-centered evaluation ; Cognitive and user-centered MIR ; Automatic classification ; Analysis tasks
Presented Remote
Automatic systems for lyric topic classification typically rely on single-label gold annotations derived from curated external sources, but it remains unclear how well these labels reflect listener interpretations or how well listeners agree with each other on song topics. In this paper, we present, to our knowledge, the first systematic study of human performance on lyric-based topic classification and compare it with a logistic regression baseline on the same dataset. We conduct an online annotation study in which participants assign three ranked topic labels to songs based solely on their lyrics. We evaluate inter-annotator agreement, as well as individual and aggregated human accuracy against Songfacts-derived ground truth labels.
The results show that lyric topic classification is challenging and inherently subjective. Specifically, individual annotators match the gold label 28% of the time (53% within their top three), comparable to an automatic baseline (27%/48%); aggregated human judgements reach 42%/69%, exceeding the strongest automatic configuration on the same songs in top-3 accuracy (33%/63%). Inter-annotator agreement is low to moderate (pairwise τ = 0.29–0.38) but rises consistently when measured against group consensus (τ = 0.40–0.51), indicating that annotators diverge in the details within a partially shared interpretation. These findings suggest that single-label evaluation may underestimate valid human judgements and motivate richer annotation and evaluation schemes for the task.
Based on the very helpful, constructive feedback we received, we made the following changes:
- Added explicit stats to the abstract.
- SD and medians reported (when needed). Not in the case of AAA, as this is a metric from the literature that uses mean by definition.
- Fixed all reported typos and improved clarity of mentioned ambiguous/contradictory/unclear points.
- Website traffic stats added.
- More citations about lyrics in MIR, and discussion now links results to existing research.
- Correlation between ACD score and human performance was tested.
- A table was removed, as recommended by the meta-reviewer, to allow for more space to address comments and suggestions.
- Supplementary material added, as promised (Zenodo link, due to no option to re-upload to CMT).
We also recalculated our results and fixed small arithmetic errors that only had a marginal impact on some (but not the overall) results.
A lot of text was rephrased and restructured to allow for space for more crucial details (especially related to reviewers' recommendations), without this affecting the key points and core content of our paper.
Due to space constraints, we could not add more information about the lyrics websites mentioned, and we also could not use our research questions as headings in the discussion for the same reason.