Transcript Research on Music Perception and Action at Haskins Laboratories

Research on Music Perception and
Action at Haskins Laboratories:
A Fast-Forward Review
(Or: 25 Years in 8 Minutes)
Bruno H. Repp
Haskins Laboratories
300 George Street
270 Crown Street (1971-2005)
“Haskins Laboratories is an independent,
international, multidisciplinary
community of researchers conducting
basic research on spoken and written
language.” [What am I doing there?]
Beginnings: Melody and Text
Mary Lou Serafine
(recent picture)
I started out as a speech researcher,
working on speech perception and
acoustic phonetics. However, in the
1980s I started a collaboration with
Mary Lou Serafine and Robert Crowder
on a musical topic of common interest.
Robert G. Crowder
(1939-2000)
Our research showed that in a test of memory for unfamiliar songs, melodies
are more difficult to recognize if they are combined with different lyrics. A
series of experiments explored the conditions under which this integration of
melody and text occurs.
Serafine, M. L., Crowder, R.G., & Repp, B. H. (1984). Integration of melody and text in memory for songs. Cognition, 16,
285-303.
Serafine, M. L., Davidson, J., Crowder, R. G., & Repp, B. H. (1986). On the nature of melody-text integration in memory
for songs. Journal of Memory and Language, 25, 123-135.
Crowder, R. G., Serafine, M. L., & Repp, B. H. (1990). Physical interaction and association by contiguity in memory for the
words and melodies of songs. Memory & Cognition, 18, 469-476.
The “Composer’s Pulse”
Manfred Clynes
At a conference in 1985, I encountered Manfred Clynes and
his theory of composer-specific expressive microstructure,
which intrigued me. With his help, I synthesized piano
performances exhibiting “appropriate” and “inappropriate”
microstructure, obtained aesthetic judgments from listeners,
and analyzed the timing of expert pianists’ performances of
a Beethoven sonata movement. The results were mixed.
Repp, B. H. (1989). Expressive microstructure in music: A preliminary perceptual assessment of four composers'
"pulses". Music Perception, 6, 243-274.
Repp, B. H. (1990a). Composers' pulses: Science or art? Music Perception, 7, 423-434.
Repp, B. H. (1990b). Further perceptual evaluations of pulse microstructure in computer performances of
classical piano music. Music Perception, 8, 1-33.
Repp, B. H. (1990c). Patterns of expressive timing in performances of a Beethoven minuet by nineteen famous
pianists. Journal of the Acoustical Society of America, 88, 622-641.
Performance Analyses I
Now I was hooked. I decided to abandon speech research and focus on music.
Stimulated by my analysis of Beethoven performances, I conducted detailed
measurements and statistical analyses of expert pianists’ expressive timing in
performances of Robert Schumann’s “Träumerei”, op. 15, No. 7, which enabled
me to give objective descriptions of commonalities and individual differences
among famous artists’ interpretations. Later I analyzed both expressive timing and
dynamics in MIDI recordings of graduate student pianists’ performances, which
were demonstrably less individual than the expert performances.
Repp, B. H. (1992b). Diversity and commonality in music performance: An analysis of timing microstructure in Schumann's
"Träumerei". Journal of the Acoustical Society of America, 92, 2546-2568.
Repp, B. H. (1993b). Objective performance analysis as a tool for the musical detective. Journal of the Acoustical Society of
America, 93, 1203-1204.
Repp, B. H. (1995c). Expressive timing in Schumann's "Träumerei": An analysis of performances by graduate student pianists.
Journal of the Acoustical Society of America, 98, 2413-2427.
Repp, B. H. (1996b). The dynamics of expressive piano performance: Schumann's "Träumerei" revisited. Journal of the Acoustical
Society of America, 100, 641-650.
Perception of Tempo and Timing
My “Träumerei” analyses led to several perceptual and performance studies using
this music. They investigated the optimal shape of a ritardando, the scaling of
expressive timing with changes in tempo, and the perceived tempo of an
expressively timed performance.
Repp, B. H. (1992c). A constraint on the expressive timing of a melodic gesture: Evidence from performance and aesthetic
judgment. Music Perception, 10, 221-242.
Repp, B. H. (1994a). Relational invariance of expressive microstructure across global tempo changes in music performance: An
exploratory study. Psychological Research, 56, 269-284.
Repp, B. H. (1994b). On determining the basic tempo of an expressive music performance. Psychology of Music, 22, 157-167.
Repp, B. H. (1995b). Quantitative effects of global tempo on expressive timing in music performance: Some perceptual evidence.
Music Perception, 13, 39-57.
Performance Analyses II
Making use of extensive MIDI performance data I had collected from Yale
graduate student pianists, I measured and analyzed everything I could lay my
hands on, including pedal timing and its dependence on tempo, arpeggio
timing, asynchronies within and between hands, and error patterns. (Manfred
Clynes called me “Reppmesser”.)
Repp, B. H. (1996c). Pedal timing and tempo in expressive piano performance: A preliminary investigation. Psychology of Music, 24, 199-221.
Repp, B. H. (1996c). Patterns of note onset asynchronies in expressive piano performance. Journal of the Acoustical Society of America, 100,
3917-3932.
Repp, B. H. (1996e). The art of inaccuracy: Why pianists' errors are difficult to hear. Music Perception, 14, 161-184.
Repp, B. H. (1997c). Some observations on pianists' timing of arpeggiated chords. Psychology of Music, 25, 133-148.
Repp, B. H. (1997d). Expressive timing in a Debussy Prelude: A comparison of student and expert pianists. Musicae Scientiae, 1, 257-268.
Repp, B. H. (1997f). The effect of tempo on pedal timing in piano performance. Psychological Research, 60, 164-172.
Repp, B. H. (1997g). Variability of timing in expressive piano performance increases with interval duration. Psychonomic Bulletin & Review, 4,
530-534.
Piano Acoustics and Technique
During that time, I also conducted a few studies of piano acoustics, and of the
production and perception of legato and staccato articulation on the piano,
using simple musical materials.
Repp, B. H. (1993a). Some empirical observations on sound level properties of recorded piano tones. Journal of the Acoustical
Society of America, 93, 1136-1144.
Repp, B. H. (1995a). Acoustics, perception, and production of legato articulation on a digital piano. Journal of the Acoustical
Society of America, 97, 3862-3874.
Repp, B. H. (1997b). Acoustics, perception, and production of legato articulation on a computer-controlled grand piano. Journal of
the Acoustical Society of America, 102, 1878-1890.
Repp, B. H. (1998f). Perception and production of staccato articulation on the piano. Unpublished manuscript.
Digression: The “Tritone Paradox”
Diana Deutsch
A stay in the Netherlands in 1993 enabled me to
pursue research on the auditory illusion called
“tritone paradox” and its presumable connection
to language and speech, proposed by Diana
Deutsch. My results did not settle the issue but
led to some new paradoxes.
Repp, B. H. (1994c). The tritone paradox and the pitch range of the speaking voice: A dubious connection. Music Perception, 12,
227-255.
Repp, B. H. (1997e). Spectral envelope and context effects in the tritone paradox. Perception, 26, 645-665.
“Average” Performances
Inspired by recent findings in the psychological literature showing that
composite faces constructed by averaging digitized images are judged to
be more attractive than most individual faces, I synthesized composite
music performances by averaging the timing and dynamic patterns of
individual performances and found that they, too, were preferred to most
individual performances (when realized on a digital piano). I also found
that the average timing patterns of groups of expert and student pianists
were remarkably similar, suggesting a common norm.
Repp, B. H. (1997a). The aesthetic quality of a quantitatively average music performance: Two preliminary experiments. Music
Perception, 14, 419-444.
Repp, B. H. (1997d). Expressive timing in a Debussy Prelude: A comparison of student and expert pianists. Musicae Scientiae, 1,
257-268.
Action-Perception Parallels in
Expressive Timing
In a series of perceptual studies I demonstrated that the detectability of a
small local change in the timing of an otherwise metronomic synthesized
music performance is closely related to the average timing pattern of
performances of the same music: If an interval is typically lengthened in
performance, its artificial lengthening is difficult to detect, presumably
because it sounds “normal”.
Repp, B. H. (1992a). Probing the cognitive representation of musical time: Structural constraints on the perception of timing perturbations.
Cognition, 44, 241-281.
Repp, B. H. (1995d). Detectability of duration and intensity increments in melody tones: A partial connection between music perception and
performance. Perception & Psychophysics, 57, 1217-1232.
Repp, B. H. (1998a). Obligatory "expectations" of expressive timing induced by perception of musical structure. Psychological Research, 61,
33-43.
Repp, B. H. (1998c). Variations on a theme by Chopin: Relations between perception and production of deviations from isochrony in music.
Journal of Experimental Psychology: Human Perception and Performance, 24, 791-811.
Repp, B. H. (1999g). Relationships between performance timing, perception of timing perturbations, and perceptual-motor synchronization in
two Chopin preludes. Australian Journal of Psychology, 51, 188 203.
Performance Analyses III
In my most ambitious performance analysis project, I measured the timing
of 115 recorded performances of the opening phrase of Chopin’s Etude in E
major, Op. 10, No. 3, and statistically extracted four underlying independent
timing patterns whose weighted combinations yielded good approximations
of the observed patterns. I also analyzed the measured dynamic patterns
and investigated the (very weak) relation between these objective measures
and aesthetic judgments of the performances.
Repp, B. H. (1998d). A microcosm of musical expression: I. Quantitative analysis of pianists' timing in the initial measures of
Chopin's Etude in E major. Journal of the Acoustical Society of America, 104, 1085-1100.
Repp, B. H. (1999a). A microcosm of musical expression: II. Quantitative analysis of pianists' dynamics in the initial measures of
Chopin's Etude in E major. Journal of the Acoustical Society of America, 105, 1972-1988.
Repp, B. H. (1999f). A microcosm of musical expression: III. Contributions of timing and dynamics to the aesthetic impression of
pianists' performances of the initial measures of Chopin's Etude in E major. Journal of the Acoustical Society of America, 106,
469-478.
More Chopin Etude Studies
The Chopin performance data enabled me to conduct other studies that relied
on them, including experiments on performance imitation, the role of auditory
feedback, mental imagery, and pattern learning.
Repp, B. H. (1998b). The detectability of local deviations from a typical expressive timing pattern. Music Perception, 15, 265-290.
Repp, B. H. (1999b). Detecting deviations from metronomic timing in music: Effects of perceptual structure on the mental
timekeeper. Perception & Psychophysics, 61, 529-548.
Repp, B. H. (1999c). Control of expressive and metronomic timing in pianists. Journal of Motor Behavior, 31, 145-164.
Repp, B. H. (1999d). Effects of auditory feedback deprivation on expressive piano performance. Music Perception, 16, 409-438.
Repp, B. H. (2000). Pattern typicality and dimensional interactions in pianists’ imitation of expressive timing and dynamics. Music
Perception, 18, 173–211.
Repp, B. H. (2002a). The embodiment of musical structure: Effects of musical context on sensorimotor synchronization with
complex timing patterns. In W. Prinz & B. Hommel (Eds.), Common mechanisms in perception and action: Attention and
Performance XIX (pp. 245–265). Oxford, U.K.: Oxford University Press.
Repp, B. H. (2002e). Perception of timing is more context sensitive than sensorimotor synchronization. Perception &
Psychophysics, 64, 703–716.
First Tapping Studies
In some of my recent music studies I had used finger tapping as a method of
tracking expectations about expressive timing. I discovered that small timing
changes in the music that were barely audible were automatically compensated
for in tapping. This awakened my interest in the phase correction process
underlying sensorimotor synchronization. I confirmed my finding in several
studies using phase perturbation methods. From now on , my research focused
on simple rhythms and only rarely used real music.
Repp, B. H. (2000a). Compensation for subliminal timing perturbations in perceptual-motor synchronization. Psychological
Research, 63, 106–128.
Repp, B. H. (2001a). Phase correction, phase resetting, and phase shifts after subliminal timing perturbations in sensorimotor
synchronization. Journal of Experimental Psychology: Human Perception and Performance, 27, 600–621.
Repp, B. H. (2002b). Automaticity and voluntary control of phase correction following event onset shifts in sensorimotor
synchronization. Journal of Experimental Psychology: Human Perception and Performance, 28, 410–430.
Repp, B. H. (2002c). Phase correction in sensorimotor synchronization: Nonlinearities in voluntary and involuntary responses to
perturbations. Human Movement Science, 21, 1–37.
Repp, B. H. (2002d). Phase correction following a perturbation in sensorimotor synchronization depends on sensory information.
Journal of Motor Behavior, 34, 291–298.
Phase and Period Correction
Peter E. Keller
Together with Peter Keller, who spent one year
with me as a post-doc, I investigated how
synchronized tapping adapts to tempo change
in a metronome (i.e., period correction). We
found period correction to be less automatic
and more cognitively controlled than phase
correction. Peter (now at the Max Planck
Institute for Human Cognitive and Brain
Sciences in Leipzig, Germany) has remained a
close collaborator.
Repp, B. H. (2001b). Processes underlying adaptation to tempo changes in sensorimotor synchronization. Human Movement
Science, 20, 277–312.
Repp, B. H., & Keller, P. E. (2004). Adaptation to tempo changes in sensorimotor synchronization: Effects of intention, attention,
and awareness. Quarterly Journal of Experimental Psychology, 57A, 499–521.
Synchronization in Two Modalities
Aniruddh D. Patel
John R. Iversen
Amandine Penel, another post-doc, and I found that
people had difficulty synchronizing with visual
sequences, especially in the presence of auditory
distractors. I also started collaborating with Aniruddh
Patel and John Iversen (Neurosciences Institute,
San Diego). We found that it is difficult to extract a
beat from a visual rhythm.
Repp, B. H., & Penel, A. (2002). Auditory dominance in temporal processing: New evidence from synchronization with
simultaneous visual and auditory sequences. Journal of Experimental Psychology: Human Perception and Performance, 28,
1085–1099.
Chen, Y., Repp, B. H., & Patel, A. D. (2002). Spectral decomposition of variability in synchronization and continuation tapping:
Comparisons between auditory and visual pacing and feedback conditions. Human Movement Science, 21, 515–532.
Repp, B. H., & Penel, A. (2004). Rhythmic movement is attracted more strongly to auditory than to visual rhythms. Psychological
Research, 68, 252–270.
Patel, A. D., Iversen, J. R., Chen, Y., & Repp, B. H. (2005). The influence of metricality and modality on synchronization with a beat.
Experimental Brain Research, 163, 226–238.
Distractors and Syncopation
In a series of experiments, I investigated the influence of auditory
distractors on synchronization with an auditory metronome or on selfpaced tapping. Peter Keller took the lead in investigations of off-beat
synchronization with alternating hands, a surprisingly difficult task.
Repp, B. H. (2003a). Phase attraction in sensorimotor synchronization with auditory sequences: Effects of single and periodic
distractors on synchronization accuracy. Journal of Experimental Psychology: Human Perception and Performance, 29, 290–
309.
Keller, P. E., & Repp, B. H. (2004). When two limbs are weaker than one: Sensorimotor syncopation with alternating hands.
Quarterly Journal of Experimental Psychology, 57A, 1085-1101.
Repp, B. H. (2004b). On the nature of phase attraction in sensorimotor synchronization with interleaved auditory sequences.
Human Movement Science, 23, 389–413.
Keller, P. E., & Repp, B. H. (2005). Staying offbeat: Sensorimotor syncopation with structured and unstructured auditory
sequences. Psychological Research, 69, 292–309.
Repp, B. H. (2006a). Does an auditory distractor sequence affect self-paced tapping? Acta Psychologica, 121, 81–107.
The Synchronization Threshold
In several studies, I determined the fastest rate at which musicians
are able to maintain synchrony with a metronome or a simple rhythm.
This “synchronization threshold” reflects a sensorimotor processing
limit around 8-10 Hz.
Repp, B. H. (2003b). Rate limits in sensorimotor synchronization with auditory and visual sequences: The synchronization
threshold and the benefits and costs of interval subdivision. Journal of Motor Behavior, 35, 355–370.
Repp, B. H. (2005a). Rate limits of on-beat and off-beat tapping with simple auditory rhythms: 1. Qualitative observations. Music
Perception, 22, 479–496.
Repp, B. H. (2005b). Rate limits of on-beat and off-beat tapping with simple auditory rhythms: 2. The role of different kinds of
accent. Music Perception, 23, 167–189.
Repp, B. H. (2007a). Hearing a melody in different ways: Multistability of metrical interpretation, reflected in rate limits of
sensorimotor synchronization. Cognition, 102, 434-454.
Repp, B. H. (2007b). Perceiving the numerosity of rapidly occurring auditory events in metrical and non-metrical contexts.
Perception & Psychophysics, 69, 529-543.
Review Papers
Halfway through the decade I published three review papers.
Repp, B. H. (2005c). Sensorimotor synchronization: A review of the tapping literature. Psychonomic Bulletin & Review, 12, 969–
992.
Repp, B. H. (2006c). Musical synchronization. In E. Altenmüller, M. Wiesendanger, & J. Kesselring (Eds.), Music, motor control, and
the brain (pp. 55–76). Oxford, UK: Oxford University Press.
Repp, B. H. (2006d). Rate limits of sensorimotor synchronization. Advances in Cognitive Psychology (http://ac-psych.org/), 2, 163181.
Perception and Action: Pianists
Günther Knoblich
In 2003 I began regular visits to the Max Planck
Institute for Human Cognitive and Brain Sciences in
Munich, later in Leipzig. Günther Knoblich (now at
Radboud University Nijmegen) taught me much
about perception and action and later invited me to
work with him for two years at Rutgers University,
Newark, when I did not have any grant support.
In three studies involving real music, we showed that pianists can recognize
their own performances and predict their own characteristic timing.
Repp, B. H., & Knoblich, G. (2004). Perceiving action identity: How pianists recognize their own performances. Psychological
Science, 15, 604–609.
Keller, P. E., Knoblich, G., & Repp, B. H. (2007). Pianists duet better when they play with themselves: On the possible role of action
simulation in synchronization. Consciousness and Cognition, 16, 102-111.
Repp, B. H., & Keller, P. E. (2010). Self versus other in piano performance: Detectability of timing perturbations depends on
personal playing style. Experimental Brain Research, 202, 101-110.
Perception and Action:
Tapping and Listening
In other studies, we investigated the feeling of agency in tapping and
showed that pressing piano keys can affect pianists’ perception of
relative pitch in the tritone paradox.
Repp, B. H. (2006b). Does an auditory perceptual illusion affect on-line auditory action control? The case of (de)accentuation and
synchronization. Experimental Brain Research, 168, 493–504.
Repp, B. H., & Knoblich, G. (2007a). Action can affect auditory perception. Psychological Science, 18, 6-7.
Repp, B. H., & Knoblich, G. (2007b). Toward a psychophysics of agency: Detecting gain and loss of control over auditory action
effects. Journal of Experimental Psychology: Human Perception and Performance, 33, 469-482.
Knoblich, G., & Repp, B. H. (2009). Inferring agency from sound. Cognition, 111, 248-262.
Repp, B. H., & Knoblich, G. (2009). Performed or observed keyboard actions affect pianists’ judgments of relative pitch. Quarterly
Journal of Experimental Psychology, 62, 2156-2170.
Rhythm Production and Perception
With various collaborators, I investigated effects of tempo, meter, and
interval structure on the accuracy of rhythm production and perception,
beat tracking, and phase correction in synchronization with moderately
complex rhythms.
Repp, B. H., Windsor, L., & Desain, P. (2002). Effects of tempo on the timing of simple musical rhythms. Music Perception, 19, 565–
593.
Repp, B. H., & Saltzman, E. L. (2002). Influences of metrical structure and grouping on the kinematics of rhythmic finger tapping.
Unpublished manuscript.
Repp, B. H., London, J., & Keller, P. E. (2005). Production and synchronization of uneven rhythms at fast tempi. Music Perception,
23, 61–78.
Repp, B. H. (2008a). Multiple temporal references in sensorimotor synchronization with metrical auditory sequences.
Psychological Research, 72, 79-98.
Repp, B. H., Iversen, J. R., & Patel, A. D. (2008). Tracking an imposed beat within a metrical grid. Music Perception, 26, 1-18.
Repp, B. H., London, J., & Keller, P. E. (2008). Phase correction in sensorimotor synchronization with nonisochronous sequences.
Music Perception, 26, 171-175.
Repp, B. H., London, J., & Keller, P. E. (submitted). Perception-production relationships and phase correction in synchronization
with two-interval rhythms.
Inter-Agent Synchronization
Synchronization between individuals (as in music performance) has
as yet been little investigated but is a special interest of Peter Keller.
I have done some preliminary work with him along these lines.
Keller, P. E., & Repp, B. H. (2008). Multilevel coordination stability: Integrated goal representations in simultaneous intra-personal
and inter-agent coordination. Acta Psychologica, 128, 378-386.
Repp, B. H., & Keller, P. E. (2008). Sensorimotor synchronization with adaptively timed sequences. Human Movement Science, 27,
423-456.
Yale Undergraduate Collaborators
In recent years, I have been sought out by an increasing number of Yale
undergraduates who wish to do research on music cognition. This has
resulted in a number of coauthored publications on various topics.
Resnicow, J. E., Salovey, P., & Repp, B. H. (2004). Is recognition of emotion in music performance an aspect of emotional
intelligence? Music Perception, 22, 145–158.
Repp, B. H., & Doggett, R. (2007). Tapping to a very slow beat: A comparison of musicians and non-musicians. Music Perception,
24, 367-376.
Repp, B. H., & Bruttomesso, M. (2009). A filled duration illusion in music: Effects of metrical subdivision on the perception and
production of beat tempo. Advances in Cognitive Psychology, 5, 114-134.
Repp, B. H., & Jendoubi, H. (2009). Flexibility of temporal expectations for triple subdivision of a beat. Advances in Cognitive
Psychology, 5, 27-41.
Repp, B. H., & Steinman, S. R. (2010). Simultaneous event-based and emergent timing: synchronization, continuation, and phase
correction. Journal of Motor Behavior, 42, 111-126.
Repp, B. H., & Marcus, R. J. (in press). No sustained sound illusion in rhythmic sequences. Music Perception.
Repp, B. H., & Thompson, J. M. (in press). Context sensitivity and invariance in perception of octave-ambiguous tones.
Psychological Research.
Current Research
I am continuing to investigate sensorimotor synchronization in various
contexts, as well as perception of rhythm, meter, and accentuation.
Repp, B. H. (2008b). Metrical subdivision results in subjective slowing of the beat. Music Perception, 26, 19-39.
Repp, B. H. (2008c). Perfect phase correction in synchronization with slow auditory sequences. Journal of Motor Behavior, 40,
363-367.
Repp, B. H. (2009a). Segregated in perception, integrated for action: Immunity of rhythmic sensorimotor coordination to auditory
stream segregation. Quarterly Journal of Experimental Psychology, 62, 426-434.
Repp, B. H. (2009b). Rhythmic sensorimotor coordination is resistant but not immune to auditory stream segregation. Quarterly
Journal of Experimental Psychology, 62, 2306-2312.
Iversen, J. R., Repp, B. H., & Patel, A. D. (2009). Top-down control of rhythm perception modulates early auditory responses.
Annals of the New York Academy of Sciences, 1169, 58-73.
Repp, B. H. (2010). Sensorimotor synchronization and perception of timing: Effects of music training and task experience. Human
Movement Science, 29, 200-213.
Repp, B. H. (in press). Self-generated interval subdivision reduces variability of synchronization with a very slow metronome.
Music Perception.
Repp, B. H. (in press). Do metrical accents create illusory phenomenal accents? Attention, Perception, & Psychophysics.
Thanks to:
• Haskins Laboratories, for allowing me to stay even though my
research is unrelated to the Laboratories’ mission
• NIH and NSF for providing financial support
• The “Master Tappers” (my regular participants: nowadays
graduate students from the Yale School of Music, previously
also including undergraduates and other graduate students),
for their cheerful cooperation and excellent rhythmic skills
• You, the audience, for your attention