Recent Research in Musical Timbre Perception

James W. Beauchamp University of Illinois at Urbana-Champaign Andrew B. Horner Hong University of Science and Technology Michael D. Hall James Madison University, Harrisonburg, VA

Starting Point

•

Timbre experiments are based on musical instrument sounds.

Starting Point

•

Timbre experiments are based on musical instrument sounds.

•

Perform short-time spectral analysis.

Starting Point

• • •

Timbre experiments are based on musical instrument sounds.

Perform short-time spectral analysis.

Identify parameters of ST spectrum:

Starting Point

• • •

Timbre experiments are based on musical instrument sounds.

Perform short-time spectral analysis.

Identify parameters of ST spectrum:

 Partial (harmonic) amplitudes Time variation - Spectral envelope (centroid, irregularity, etc.)

Starting Point

• • •

Timbre experiments are based on musical instrument sounds.

Perform short-time spectral analysis.

Identify parameters of ST spectrum:

 Partial (harmonic) amplitudes Time variation - Spectral envelope (centroid, irregularity, etc.)  Partial (harmonic) frequencies Time variation - Inharmonicity

Methods for Studying Timbre

Stimuli Preparation In Freq. Domain – Simplification – Perturbation – Normalization

Methods for Studying Timbre

Stimuli Preparation In Freq. Domain – Simplification – Perturbation – Normalization Listener Experiments –Discrimination (pairs) –Timbral Distance Estimation –Classification –Identification

Methods for Studying Timbre

Stimuli Preparation In Freq. Domain – Simplification – Perturbation – Normalization Listener Experiments –Discrimination (pairs) –Timbral Distance Estimation –Classification –Identification Data Processing/Presentation –Discrimination (sensitivity) scores/plots –Multidimensional Scaling –Correspondence (

R

2 ) Measurements

Studies Reviewed

• 1999 Discrimination Study • 2006 Discrimination Study • 2006 Multidimensional Scaling (MDS) Study • 2009 Discrimination/Classification Study

1999 Discrimination Study

(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds – clarinet, flute, oboe, trumpet, violin, harpsichord, marimba

1999 Discrimination Study

(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds – clarinet, flute, oboe, trumpet, violin, harpsichord, marimba • Equalize

F

0 , loudness, and duration.

1999 Discrimination Study

(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds – clarinet, flute, oboe, trumpet, violin, harpsichord, marimba • Equalize

F

0 , loudness, and duration.

• Test sounds: Apply six spectrotemporal simplifications.

1999 Discrimination Study

(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds – clarinet, flute, oboe, trumpet, violin, harpsichord, marimba • Equalize

F

0 , loudness, and duration..

• Test sounds: Apply six spectrotemporal simplifications.

• Subjects discriminate between original and simplified sounds.

1999 Discrimination Study Results

Discrim Score • Spectral envelope smoothing • Spectral flux elimination 96% 91% • Amplitude envelopes smoothing • Frequency envelopes smoothing 70% • Freq. envs. harmonic locking • Frequency variations elimination 66% 69% 71%

2006 Discrimination Study

Horner, Beauchamp, and So JAES • Eight sustained musical instrument tones – bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin

2006 Discrimination Study

Horner, Beauchamp, and So JAES • Eight sustained musical instrument tones – bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin • Modified by fixed random transfer function 1  2  

H

(

f

)  1  2  ,  = error level

2006 Discrimination Study

Horner, Beauchamp, and So JAES • Eight sustained musical instrument tones – bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin • • Modified by fixed random transfer function – 1  2  

H

(

f

)  1  2  ,  = error level

F

0 , loudness, duration, centroid preserved

2006 Discrimination Study

Horner, Beauchamp, and So JAES • Eight sustained musical instrument tones – bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin • • Modified by fixed random transfer function – 1  2  

H

(

f

)  1  2  ,  = error level

F

0 , loudness, duration, centroid preserved Typical spectral envelopes:

2006 Discrimination Study

Horner, Beauchamp, and So JAES •

Objective:

To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to discrimination between original and modified tones.

2006 Discrimination Study

Horner, Beauchamp, and So JAES •

Objective:

To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to the discrimination data.

•

Best results:

obtained by

relative-amplitude (harmonic) spectral error:



rase

 1

N N



n

 1

a K



k

 1

A k

 

n



A

'

k

 

n a

, 0 

a

 3.0

K



k

 1

A k a

Usually,

a

 1 or 2

1 0.9

0.8

0.7

0.6

0.5

0.4

0

2006 Discrimination Study

Horner, Beauchamp, and So JAES Discrimination vs. error level (  ): 0.1

0.2

0.3

e rror le v e l

0.4

0.5

R

2 =0.81

2006 Discrimination Study

Horner, Beauchamp, and So JAES Discrimination vs. rel-amp spec error: 1 0.9

0.8

0.7

0.6

0.5

0.4

0 0.1

0.2

0.3

relative-amplitude spectral error

0.4

0.5

R

2 =0.90

for

a

=1.0

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones – bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • • Ten sustained musical instrument tones – bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin

F

0 , loudness, duration, attack & decay times, and average spectral centroid are equalized.

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones – bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin •

F

0 , loudness, duration, attack & decay times, and average spectral centroid are equalized.

• Two types of tones:

static

(flux removed) and

dynamic

(flux retained).

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones – bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin •

F

0 , loudness, duration, attack & decay times, and average spectral centroid are equalized.

• Two types of tones:

static

(flux removed) and

dynamic

(flux retained).

• Subjects estimate timbral dissimilarity between instruments.

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones – bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin •

F

0 , loudness, duration, attack & decay times, and average spectral centroid are equalized.

• Two types of tones:

static

(flux removed) and

dynamic

(flux retained).

• Subjects estimate timbral dissimilarity between instruments.

• Data processed by two multi-dimensional scaling (MDS) programs (SPSS & Matlab).

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: •

E

ven

/O

dd: Ratio of even and odd harmonic rms amplitudes.

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: •

E

ven

/O

dd: Ratio of even and odd harmonic rms amplitudes •

S

pectral

IR

regularity: Degree of jaggedness of a spectrum.

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: •

E

ven

/O

dd: Ratio of even and odd harmonic rms amplitudes •

S

pectral

IR

regularity: Degree of jaggedness of a spectrum.

For Dynamic Tones Only: •

S

pectral

C

entroid

V

ariation: Standard deviation of the spectral centroid normalized by average value.

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: •

E

ven

/O

dd: Ratio of even and odd harmonic rms amplitudes •

S

pectral

IR

regularity: Degree of jaggedness of a spectrum.

For Dynamic Tones Only: •

S

pectral

C

entroid

V

ariation: Standard deviation of the spectral centroid normalized by average value.

•

S

pectral

IN

coherence: Degree of spectral change relative to the average spectrum (same as

flux

).

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Static Tone Case Correlations: E/O: R=0.78

SIR: R=0.69

SPSS algorithm Stress=0.12

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Static Tone Case Correlations: E/O: R=0.79

SIR: R=0.75

Matlab algorithm Stress=0.12

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Dynamic Tone Case SPSS algorithm Correlations: E/O: R=0.71

SCV: R=0.68 SIN: R=0.56 SIR: R=0.39

Stress=0.17

2006 MDS Study

Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Dynamic Tone Case Matlab algorithm Correlations: E/O: R=0.69

SCV: R=0.68 SIN: R=0.53 SIR: R=0.40

Stress=0.15

2009 Study

Hall and Beauchamp (Canadian Acoustics) •

Goals/Purpose Exp. 1. Relative importance of spectral vs. temporal cues:

Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope.

2009 Study

Hall and Beauchamp (Canadian Acoustics) •

Goals/Purpose Exp. 1. Relative importance of spectral vs. temporal cues:

Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope.

•

Exp. 2. Relative importance of spectral envelope (formant) structure vs. spectral centroid:

Compare discrimination/classification performance for interpolated tones vs. tones obtained by filtration which matches the centroids of the interpolated tones.

2009 Study

Hall and Beauchamp (Canadian Acoustics) •

Experiment 1 Method Reference stimuli:

Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)

2009 Study

Hall and Beauchamp (Canadian Acoustics) • •

Experiment 1 Method Reference stimuli:

Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)

Test stimuli: A

4  4 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs time envelope between the violin and trombone timbres.

Vn I 10 I 20 I 30 Temporal I 01 I 11 I 21 I 31 I 02 I 12 I 22 I 32 I 03 I 13 I 23 Tr

2009 Study

Hall and Beauchamp (Canadian Acoustics) • • •

Experiment 1 Method Reference stimuli:

Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)

Test stimuli: A

4  4 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs time envelope between the violin and trombone timbres.

Interpolation steps:

Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or

both

(3 steps gives the opposite timbre.).

2009 Study

Hall and Beauchamp (Canadian Acoustics) • • • •

Experiment 1 Method Reference stimuli:

Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)

Test stimuli: A

4  4 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs time envelope between the violin and trombone timbres.

Interpolation steps:

Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or

both

(3 steps gives the opposite timbre.).

Subjects’ tasks:

- 1) to discriminate tone pairs. - 2) to classify tones as ‘violin’, ‘trombone’, or ‘other’.

2009 Study

Hall and Beauchamp (Canadian Acoustics)

Experiment 1 Results

Discrimination: reference stimuli:

Violin Trombone

6.00

5.00

4.00

3.00

2.00

1.00

0.00

-1.00

1 2

Step Size

3 6.00

5.00

4.00

Amplitude Envelope 3.00

Spectral Envelope Both 2.00

1.00

0.00

-1.00

1 2

Step Size

3 Amplitude Envelope Spectral Envelope Both Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.

2009 Study

Hall and Beauchamp (Canadian Acoustics) Classification:

Experiment 1 Results Violin Responses Trombone Responses

1.00

0.90

0.80

0.70

0.60

0.50

0.40

0.30

0.20

0.10

0.00

1.00

0.90

0.80

0.60

0.50

TrHybrid 0.20

0.10

0.00

Vn V nH yb rid Tr H yb rid

Spectral Envelope

Tr Vn V nH yb rid Tr H yb rid

Spectral Envelope

Tr Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.

Amplitude Envelope

Vn VnHybrid TrHybrid Tr

2009 Study

Hall and Beauchamp (Canadian Acoustics)

Experiment 2 Method

•

Reference stimulus:

Original impoverished violin.

2009 Study

Hall and Beauchamp (Canadian Acoustics)

Experiment 2 Method

•

Reference stimulus:

Original impoverished violin.

•

Test stimuli:

- 1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept).

2009 Study

Hall and Beauchamp (Canadian Acoustics)

Experiment 2 Method

•

Reference stimulus:

Original impoverished violin.

•

Test stimuli:

- 1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept).

- 2) 3 violin tones low-pass filtered in steps to match the spectral centroids of the 3 interpolated tones of 1).

2009 Study

Hall and Beauchamp (Canadian Acoustics)

Experiment 2 Method

•

Reference stimulus:

Original impoverished violin.

•

Test stimuli:

- 1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept).

- 2) 3 violin tones low-pass filtered in steps to match the spectral centroids of the 3 interpolated tones of 1).

•

Subjects’ tasks:

Discrimination and classification as in Exp. 1. (Which has the greater effect? Interpolation or filtration?)

2009 Study

Hall and Beauchamp (Canadian Acoustics)

Experiment 2 Results

Discrimination: 5.00

4.50

4.00

3.50

3.00

2.50

2.00

1.50

1.00

0.50

0.00

Classification:

Violin Responses

1.00

0.90

0.80

0.70

0.60

0.50

0.40

0.30

0.20

0.10

0.00

Formant Structure

Stimulus Type

Spectral Centroid 1 1.00

0.90

0.80

0.70

0.60

0.50

0.40

0.30

0.20

0.10

0.00

2

Step Size

3

Trombone Responses

Formant Structure Spectral Centroid Vn VnHybrid TrHybrid Tr Formant Structure

Stimulus Type

Spectral Centroid

Conclusion Summary

•

1999 discrimination study:

– Spectral envelope detail and spectral flux are important for dynamic musical sounds, and they are more important than temporal detail.

Conclusion Summary

•

1999 discrimination study:

– Spectral envelope detail and spectral flux are important for dynamic musical sounds, and they are more important than temporal detail.

•

2006 discrimination study:

– The ability to hear differences between dynamic tones with matched spectral centroids and randomly altered spectra correlates strongly with relative spectral amplitude differences.

•

Conclusions

2006 MDS study:

– Using centroid and attack/decay normalized tones, there is strong evidence that

even/odd ratio

and other spectral envelope details are important for timbral differences of impoverished (static) and dynamic musical instrument tones.

•

Conclusions

2006 MDS study:

– Using centroid and attack/decay normalized tones, there is strong evidence that

even/odd ratio

and other spectral envelope details are important for timbral differences of impoverished (static) and dynamic musical instrument tones.

•

2009 discrimination/classification study:

– Using spectral interpolation with respect to both spectral and temporal dimensions on impoverished violin and trombone tones: 1) Spectral differences were found to be more important than temporal differences. 2) Detailed spectral differences were much more important than mere spectral centroid differences.