Transcript MusicVisualization - Informatics

Visualization of Music
Donald Byrd
School of Informatics & Jacobs School of Music
Indiana University
Updated 19 Mar. 2008
Copyright © 2003-08, Donald Byrd
19 Mar. 08
1
Basic Representations of Music & Audio (1)
Digital Audio
Audio (e.g., CD, MP3):
like speech
Time-stamped
Time-stamped
Events Events
(e.g., MIDI file): like
unformatted text
Musiclike
Notation
Music Notation:
text with complex
formatting
15 Nov. 07
2
Basic Representations of Music & Audio (2)
Audio: no (explicit) structure
Events/MIDI: simple structure
Notation: very complex structure
15 Nov. 07
3
Basic and Specific Representations vs. Encodings
Basic and Specific Representations (above the line)
Audio
Time-stamped Events
Waveform
Time-stamped MIDI
Csound score
Time-stamped expM IDI
.WAV
Red Book (CD)
SMF
Csound score
Music Notation
Gamelan not.
Notelist
expM IDI File
Tablature
CM N
M ensural not.
M usicXM L
Finale
ETF
Encodings (below the line)
15 Nov. 07
4
Audacity vs. Don & Gabi’s Audio Segmentation
demos
19 Mar. 08
5
Islands of Music: Visualization of a Music
Collection
Cooper, Matthew; Foote, Jonathan; Pampalk, Elias; & Tzanetakis, George (2006).
Visualization in Audio-Based Music Information Retrieval. Computer Music Journal 30(2),
pp. 42—62.
15 Nov. 07
6
“All at once” & “one alone” Visualizations in R of
Chopin Preludes
all 24 Preludes, Op. 28
“bag of notes” (order is
ignored) for the C
major Prelude only
19 Mar. 08
7
Variations2 Timeliner
19 Nov. 08
8
Malinowski’s Music Animation Machine
(from http://www.musanim.com/)
15 Nov. 07
9
Langner & Goebl’s “Performance Worm”
15 Nov. 07
10
Types of Music Visualizations (1)
• What music is visualized?
– Abstractions of music, no particular works
– Collections of pieces
– Individual pieces
• Performances, scores or both?
• What features are visualized?
• How is visualization organized?
•
•
•
•
By time (horizontal structure)
By features of harmony, scale, etc.
Temporal resolution: bag of notes (or measures), etc.
Can be 2 or more ways at once
19 Mar. 08
11
Types of Music Visualizations (2)
• Dimensionality
– 2D (on paper or screen)
– 3D (simulated on screen, virtual reality, etc.)
• How dynamic is the visualization?
– Changing (“animation”) resulting from scrolling isn’t
really dynamic
• If highlighting follows time, “semi-dynamic”
• Ex: Malinowski’s Music Animation Machine
– Truly dynamic visualizations are less common
• Ex: Chew’s MuSA.RT, Langner & Goebl’s “Performance
Worm”
19 Mar. 08
12
Visualization Pitfalls
• Disorientation (navigation confusion)
– Common with…
• Simulated 3D: most people aren’t good at it
• Rotation helps a lot, esp. if user controlled
• Multiple visualizations
– …but can occur w/ any poorly-designed interface
– Music visualization w/ audio has extra danger, unless synched
• Distracting glitziness
• Unresponsive
• Misleading
– Cf. Edward Tufte’s books
• The more parameters, the harder to make everything clear
– Music has loads of parameters!
rev. 19 Mar. 08
13
Why is Visualization Relevant to Music
Informatics?
• Affects music representation
• Brings up HCI (Human/Computer Interaction) issues
• A powerful tool for finding patterns in information
15 Nov. 07
14
The Obvious Static Visualization: CWMN
The first few measures of Variation 8 of the “Twinkle” Variations
15 Nov. 07
15
Static Aligned Visualizations of Music via Text:
Standard MIDI File (file dump)
•
•
•
•
•
•
•
•
•
•
•
•
•
0: 4D54 6864 0000 0006 0001 0003 01E0 4D54
16: 726B 0000 0014 00FF 5103 0B70 C000 FF58
32: 0402 0218 0896 34FF 2F00 4D54 726B 0000
48: 0055 00FF 0305 5069 616E 6F00 9048 3881
64: 6480 4840 0C90 4A38 8164 804A 400C 904B
80: 3881 6480 4B40 0C90 4D38 8164 804D 400C
96: 904F 3883 4880 4F40 1890 4F38 8360 9050
112: 3883 4880 4F40 1890 4D38 8330 8050 4018
128: 804D 400D FF2F 004D 5472 6B00 0000 3200
144: FF03 0550 6961 6E6F 8F00 9041 2B81 6480
160: 4140 0C90 4330 8164 8043 400C 9044 3181
176: 6480 4440 0C90 4647 8164 8046 4001 FF2F
192: 00 .
15 Nov. 07
MThd.........‡MT
rk......Q..p¿..X
.....ñ4./.MTrk..
.U....Piano.êH8Å
dÄH@.êJ8ÅdÄJ@.êK
8ÅdÄK@.êM8ÅdÄM@.
êO8ÉHÄO@.êO8É`êP
8ÉHÄO@.êM8É0ÄP@.
ÄM@../.MTrk...2.
...Pianoè.êA+ÅdÄ
A@.êC0ÅdÄC@.êD1Å
dÄD@.êFGÅdÄF@../
16
The Obvious Static Visualization: A Complex Case
Complications on staff 2:
• Editorial additions (small notes)
• Instruments sharing notes only some of the time
• Mixed durations in double stops
• Multiple voices (divisi notation)
• Rapidly gets worse with more than 2!
15 Nov. 07
17
Static Visualizations Aligned by Time: Hendrix
Experience, L.L.C.
15 Nov. 07
18
Static Visualizations Aligned by Time: Audacity vs.
Don & Gabi’s Audio Segmentation demos
• 2 programs don’t communicate => manual alignment!
19 Mar. 08
19
Static Visualizations Aligned by Time: SyncPlayer
• Above figure is SyncPlayer 2 (not available yet)
• SyncPlayer 1.5 aligns MIDI, “structure”, lyrics
• …synchronizes w/ playback
19 Mar. 08
20
Static Visualizations Aligned by Time: Sonic
Visualiser
• Shows whatever plugins want, synchronized with playback
15 Nov. 07
21
Static Visualization over Time: Variations2 Timeliner
• Shows structure & annotations synchronized with playback
• Structure completely controlled by user
19 Nov. 08
22
Static Visualization over Time: Chopin Mazurkas
Project (Harmony)
(from http://www.mazurka.org.uk/info/keyscape/)
15 Nov. 07
23
Semi-Dynamic Visualization over Time:
Malinowski’s Music Animation Machine
(from http://www.musanim.com/)
15 Nov. 07
24
Dynamic Visualization over Time: Langner &
Goebl’s “Performance Worm”
15 Nov. 07
25
Does the Ultimate System Exist? (1)
• Goal: visualizer that can do anything with music
– Handle any no. & combination of visualizations
– Static visualizations: audio, any kind of notation, structural
diagrams, etc.
– Real-time visualizations: video, etc.
– Automatic (or near-automatic) synchronization
– Modular design (in layers) is vital
– Completely general framework
– Well-documented API (preferably plug-ins) for user-written
visualizers
– Support OS-level technologies (QuickTime, etc.)
– Easy-to-learn UI allowing high degree of control
• If it could do all that, should be useful for lots besides
music!
19 Mar. 08
26
Does the Ultimate System Exist? (2)
• Sonic Visualiser 1.2
– Plug-in architecture for user-written visualizers
– User can have a visualizer in a separate pane, or as a layer in existing
pane
– Panes can have different zoom levels
– Can automatically synchronize audio to audio (like SyncPlayer)
– Seems to be able to do anything with audio, but…
19 Mar. 08
27
Does the Ultimate System Exist? (3)
• SyncPlayer 2 (& features in 1.5 demo)
–
–
–
–
–
Plug-in architecture for user-written visualizers
Can automatically synchronize audio to audio
Seems to be able to do anything with audio, & handles CWMN
Visualizers can’t share a window(?)
Not good for video, etc.
19 Mar. 08
28
Everything over Time: Byrd’s GMV (“General
Multimedia Visualizer”)
COMING SOON
19 Mar. 08
29