Transcript Recent Research in Musical Timbre Perception
Recent Research in Musical Timbre Perception
James W. Beauchamp University of Illinois at Urbana-Champaign Andrew B. Horner Hong University of Science and Technology Michael D. Hall James Madison University, Harrisonburg, VA
Starting Point
•
Timbre experiments are based on musical instrument sounds.
Starting Point
•
Timbre experiments are based on musical instrument sounds.
•
Perform short-time spectral analysis.
Starting Point
• • •
Timbre experiments are based on musical instrument sounds.
Perform short-time spectral analysis.
Identify parameters of ST spectrum:
Starting Point
• • •
Timbre experiments are based on musical instrument sounds.
Perform short-time spectral analysis.
Identify parameters of ST spectrum:
Partial (harmonic) amplitudes Time variation - Spectral envelope (centroid, irregularity, etc.)
Starting Point
• • •
Timbre experiments are based on musical instrument sounds.
Perform short-time spectral analysis.
Identify parameters of ST spectrum:
Partial (harmonic) amplitudes Time variation - Spectral envelope (centroid, irregularity, etc.) Partial (harmonic) frequencies Time variation - Inharmonicity
Methods for Studying Timbre
Stimuli Preparation In Freq. Domain – Simplification – Perturbation – Normalization
Methods for Studying Timbre
Stimuli Preparation In Freq. Domain – Simplification – Perturbation – Normalization Listener Experiments –Discrimination (pairs) –Timbral Distance Estimation –Classification –Identification
Methods for Studying Timbre
Stimuli Preparation In Freq. Domain – Simplification – Perturbation – Normalization Listener Experiments –Discrimination (pairs) –Timbral Distance Estimation –Classification –Identification Data Processing/Presentation –Discrimination (sensitivity) scores/plots –Multidimensional Scaling –Correspondence (
R
2 ) Measurements
Studies Reviewed
• 1999 Discrimination Study • 2006 Discrimination Study • 2006 Multidimensional Scaling (MDS) Study • 2009 Discrimination/Classification Study
1999 Discrimination Study
(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds – clarinet, flute, oboe, trumpet, violin, harpsichord, marimba
1999 Discrimination Study
(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds – clarinet, flute, oboe, trumpet, violin, harpsichord, marimba • Equalize
F
0 , loudness, and duration.
1999 Discrimination Study
(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds – clarinet, flute, oboe, trumpet, violin, harpsichord, marimba • Equalize
F
0 , loudness, and duration.
• Test sounds: Apply six spectrotemporal simplifications.
1999 Discrimination Study
(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds – clarinet, flute, oboe, trumpet, violin, harpsichord, marimba • Equalize
F
0 , loudness, and duration..
• Test sounds: Apply six spectrotemporal simplifications.
• Subjects discriminate between original and simplified sounds.
1999 Discrimination Study Results
Discrim Score • Spectral envelope smoothing • Spectral flux elimination 96% 91% • Amplitude envelopes smoothing • Frequency envelopes smoothing 70% • Freq. envs. harmonic locking • Frequency variations elimination 66% 69% 71%
2006 Discrimination Study
Horner, Beauchamp, and So JAES • Eight sustained musical instrument tones – bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin
2006 Discrimination Study
Horner, Beauchamp, and So JAES • Eight sustained musical instrument tones – bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin • Modified by fixed random transfer function 1 2
H
(
f
) 1 2 , = error level
2006 Discrimination Study
Horner, Beauchamp, and So JAES • Eight sustained musical instrument tones – bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin • • Modified by fixed random transfer function – 1 2
H
(
f
) 1 2 , = error level
F
0 , loudness, duration, centroid preserved
2006 Discrimination Study
Horner, Beauchamp, and So JAES • Eight sustained musical instrument tones – bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin • • Modified by fixed random transfer function – 1 2
H
(
f
) 1 2 , = error level
F
0 , loudness, duration, centroid preserved Typical spectral envelopes:
2006 Discrimination Study
Horner, Beauchamp, and So JAES •
Objective:
To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to discrimination between original and modified tones.
2006 Discrimination Study
Horner, Beauchamp, and So JAES •
Objective:
To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to the discrimination data.
•
Best results:
obtained by
relative-amplitude (harmonic) spectral error:
rase
1
N N
n
1
a K
k
1
A k
n
A
'
k
n a
, 0
a
3.0
K
k
1
A k a
Usually,
a
1 or 2
1 0.9
0.8
0.7
0.6
0.5
0.4
0
2006 Discrimination Study
Horner, Beauchamp, and So JAES Discrimination vs. error level ( ): 0.1
0.2
0.3
e rror le v e l
0.4
0.5
R
2 =0.81
2006 Discrimination Study
Horner, Beauchamp, and So JAES Discrimination vs. rel-amp spec error: 1 0.9
0.8
0.7
0.6
0.5
0.4
0 0.1
0.2
0.3
relative-amplitude spectral error
0.4
0.5
R
2 =0.90
for
a
=1.0
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones – bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • • Ten sustained musical instrument tones – bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin
F
0 , loudness, duration, attack & decay times, and average spectral centroid are equalized.
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones – bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin •
F
0 , loudness, duration, attack & decay times, and average spectral centroid are equalized.
• Two types of tones:
static
(flux removed) and
dynamic
(flux retained).
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones – bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin •
F
0 , loudness, duration, attack & decay times, and average spectral centroid are equalized.
• Two types of tones:
static
(flux removed) and
dynamic
(flux retained).
• Subjects estimate timbral dissimilarity between instruments.
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones – bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin •
F
0 , loudness, duration, attack & decay times, and average spectral centroid are equalized.
• Two types of tones:
static
(flux removed) and
dynamic
(flux retained).
• Subjects estimate timbral dissimilarity between instruments.
• Data processed by two multi-dimensional scaling (MDS) programs (SPSS & Matlab).
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: •
E
ven
/O
dd: Ratio of even and odd harmonic rms amplitudes.
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: •
E
ven
/O
dd: Ratio of even and odd harmonic rms amplitudes •
S
pectral
IR
regularity: Degree of jaggedness of a spectrum.
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: •
E
ven
/O
dd: Ratio of even and odd harmonic rms amplitudes •
S
pectral
IR
regularity: Degree of jaggedness of a spectrum.
For Dynamic Tones Only: •
S
pectral
C
entroid
V
ariation: Standard deviation of the spectral centroid normalized by average value.
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: •
E
ven
/O
dd: Ratio of even and odd harmonic rms amplitudes •
S
pectral
IR
regularity: Degree of jaggedness of a spectrum.
For Dynamic Tones Only: •
S
pectral
C
entroid
V
ariation: Standard deviation of the spectral centroid normalized by average value.
•
S
pectral
IN
coherence: Degree of spectral change relative to the average spectrum (same as
flux
).
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Static Tone Case Correlations: E/O: R=0.78
SIR: R=0.69
SPSS algorithm Stress=0.12
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Static Tone Case Correlations: E/O: R=0.79
SIR: R=0.75
Matlab algorithm Stress=0.12
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Dynamic Tone Case SPSS algorithm Correlations: E/O: R=0.71
SCV: R=0.68 SIN: R=0.56 SIR: R=0.39
Stress=0.17
2006 MDS Study
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Dynamic Tone Case Matlab algorithm Correlations: E/O: R=0.69
SCV: R=0.68 SIN: R=0.53 SIR: R=0.40
Stress=0.15
2009 Study
Hall and Beauchamp (Canadian Acoustics) •
Goals/Purpose Exp. 1. Relative importance of spectral vs. temporal cues:
Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope.
2009 Study
Hall and Beauchamp (Canadian Acoustics) •
Goals/Purpose Exp. 1. Relative importance of spectral vs. temporal cues:
Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope.
•
Exp. 2. Relative importance of spectral envelope (formant) structure vs. spectral centroid:
Compare discrimination/classification performance for interpolated tones vs. tones obtained by filtration which matches the centroids of the interpolated tones.
2009 Study
Hall and Beauchamp (Canadian Acoustics) •
Experiment 1 Method Reference stimuli:
Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)
2009 Study
Hall and Beauchamp (Canadian Acoustics) • •
Experiment 1 Method Reference stimuli:
Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)
Test stimuli: A
4 4 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs time envelope between the violin and trombone timbres.
Vn I 10 I 20 I 30 Temporal I 01 I 11 I 21 I 31 I 02 I 12 I 22 I 32 I 03 I 13 I 23 Tr
2009 Study
Hall and Beauchamp (Canadian Acoustics) • • •
Experiment 1 Method Reference stimuli:
Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)
Test stimuli: A
4 4 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs time envelope between the violin and trombone timbres.
Interpolation steps:
Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or
both
(3 steps gives the opposite timbre.).
2009 Study
Hall and Beauchamp (Canadian Acoustics) • • • •
Experiment 1 Method Reference stimuli:
Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)
Test stimuli: A
4 4 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs time envelope between the violin and trombone timbres.
Interpolation steps:
Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or
both
(3 steps gives the opposite timbre.).
Subjects’ tasks:
- 1) to discriminate tone pairs. - 2) to classify tones as ‘violin’, ‘trombone’, or ‘other’.
2009 Study
Hall and Beauchamp (Canadian Acoustics)
Experiment 1 Results
Discrimination: reference stimuli:
Violin Trombone
6.00
5.00
4.00
3.00
2.00
1.00
0.00
-1.00
1 2
Step Size
3 6.00
5.00
4.00
Amplitude Envelope 3.00
Spectral Envelope Both 2.00
1.00
0.00
-1.00
1 2
Step Size
3 Amplitude Envelope Spectral Envelope Both Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.
2009 Study
Hall and Beauchamp (Canadian Acoustics) Classification:
Experiment 1 Results Violin Responses Trombone Responses
1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
1.00
0.90
0.80
0.60
0.50
TrHybrid 0.20
0.10
0.00
Vn V nH yb rid Tr H yb rid
Spectral Envelope
Tr Vn V nH yb rid Tr H yb rid
Spectral Envelope
Tr Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.
Amplitude Envelope
Vn VnHybrid TrHybrid Tr
2009 Study
Hall and Beauchamp (Canadian Acoustics)
Experiment 2 Method
•
Reference stimulus:
Original impoverished violin.
2009 Study
Hall and Beauchamp (Canadian Acoustics)
Experiment 2 Method
•
Reference stimulus:
Original impoverished violin.
•
Test stimuli:
- 1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept).
2009 Study
Hall and Beauchamp (Canadian Acoustics)
Experiment 2 Method
•
Reference stimulus:
Original impoverished violin.
•
Test stimuli:
- 1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept).
- 2) 3 violin tones low-pass filtered in steps to match the spectral centroids of the 3 interpolated tones of 1).
2009 Study
Hall and Beauchamp (Canadian Acoustics)
Experiment 2 Method
•
Reference stimulus:
Original impoverished violin.
•
Test stimuli:
- 1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept).
- 2) 3 violin tones low-pass filtered in steps to match the spectral centroids of the 3 interpolated tones of 1).
•
Subjects’ tasks:
Discrimination and classification as in Exp. 1. (Which has the greater effect? Interpolation or filtration?)
2009 Study
Hall and Beauchamp (Canadian Acoustics)
Experiment 2 Results
Discrimination: 5.00
4.50
4.00
3.50
3.00
2.50
2.00
1.50
1.00
0.50
0.00
Classification:
Violin Responses
1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
Formant Structure
Stimulus Type
Spectral Centroid 1 1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
2
Step Size
3
Trombone Responses
Formant Structure Spectral Centroid Vn VnHybrid TrHybrid Tr Formant Structure
Stimulus Type
Spectral Centroid
Conclusion Summary
•
1999 discrimination study:
– Spectral envelope detail and spectral flux are important for dynamic musical sounds, and they are more important than temporal detail.
Conclusion Summary
•
1999 discrimination study:
– Spectral envelope detail and spectral flux are important for dynamic musical sounds, and they are more important than temporal detail.
•
2006 discrimination study:
– The ability to hear differences between dynamic tones with matched spectral centroids and randomly altered spectra correlates strongly with relative spectral amplitude differences.
•
Conclusions
2006 MDS study:
– Using centroid and attack/decay normalized tones, there is strong evidence that
even/odd ratio
and other spectral envelope details are important for timbral differences of impoverished (static) and dynamic musical instrument tones.
•
Conclusions
2006 MDS study:
– Using centroid and attack/decay normalized tones, there is strong evidence that
even/odd ratio
and other spectral envelope details are important for timbral differences of impoverished (static) and dynamic musical instrument tones.
•
2009 discrimination/classification study:
– Using spectral interpolation with respect to both spectral and temporal dimensions on impoverished violin and trombone tones: 1) Spectral differences were found to be more important than temporal differences. 2) Detailed spectral differences were much more important than mere spectral centroid differences.