Transcript Representation of Musical Information

Other Topics in Music
Organization/Representation
Donald Byrd
School of Informatics
Indiana University
Updated 29 April 2006
Copyright © 2003-06, Donald Byrd
1
Classification: Surgeon General’s Warning
• Classification (ordinary hierarchic) is dangerous
–
–
–
–
Almost everything in the real world is messy
Absolute correlations between characteristics are rare
Example: some mammals lay eggs; some are “naked”
Example: musical instruments (piano as percussion,
etc.)
• Nearly always, all you can say is “an X has
characteristic A, and usually also B, C, D…”
• Leads to:
– People who know better claiming absolute correlations
– Arguments among experts over which characteristic is
most fundamental
– Don changing his mind
30 Jan. 06
2
Comparison of Music-IR Task Classifications
• Typke, Rainer, Wiering, Frans, & Veltkamp,
Remco C. (2005). A Survey of Music Information
Retrieval Systems
– Overview of 17 systems for content-based retrieval of
music in both audio & symbolic forms
– Includes “map” of systems showing tasks & users for
which each is most appropriate
– Horizontal axis (tasks) has similar idea to my Similarity
Scale for Content-Based Music IR
– Main difference: Typke et al have “artist”
– Doesn’t fit hierarchy, but useful--and dangerous!
4 April 06
3
Music Recommender Systems (1)
• Guest speaker: Justin Donaldson
– PhD student, IU Computer Science
– Intern, MusicStrands
• Pandora’s approach
– Classification by experts with controlled vocabulary
– “Music genome”
• MusicStrands approach
– Co-occurrence, network analysis, with limited guidance by expert
• Examples: FOAFing the Music, Last.fm, MusicIP Mixer,
Musicmatch Jukebox, MusicStrands, Pandora
5 April 06
4
Music Recommender Systems (2)
• All(?) existing systems try to find music similar to what
you give them
• Instead, do the opposite
– Tim Crawford to Don (2004): I don't want to find more music like
what I already know, I want music as different as possible from it!
• Jeremy Pickens' example: Eigenradio
– http://eigenradio.media.mit.edu/christmas_2004.html
– NOT a good example!
• How to automate? Genre classification?
5 April 06
5
Music Recommender Systems (3)
• Systems: FOAFing the Music, Last.fm, MusicIP
Mixer, Musicmatch Jukebox, MusicStrands,
Pandora
• Pandora’s “music genome” idea
– Assumes all music based on small number of “genes”
– Content-based
– Requires annotation by human experts
• Last.fm
– Conventional collaborative approach(?)
• Others?
7 April 06
6
Maps, Visualizations, & Metrics (1)
• Example 1: Justin Donaldson’s 3-D visualization
• Map requires metric (similarity measure) => positions in ndimensional space
• n can be huge, except for visualization
• Example 2: Pampalk et al, “Exploring Music Collections
by Browsing Different Views” (ISMIR 2003, CMJ 2004)
–
–
–
–
Organization by spectrum, periodicity, metadata
Uses self-organizing maps (SOMs)
SOMs can focus on audio analysis and/or metadata
Maps of same collection aligned => can move from one view to
another
10 April 06
7
Maps, Visualizations, & Metrics (2)
• With good similarity measure, easy to find similar stuff or
different stuff!
– Automatically (searching, filtering)
– Do-it-yourself (browsing)
•
•
•
•
What’s a good similarity measure for music?
Usual interpretation: what are good features to use?
Cf. Eigenradio: has objective features, but need subjective
Cf. Pampalk comment on main difficulty
10 April 06
8
Digital Music Libraries
• iTunes: no. 1 commercial system
– Popular & simple but not always easy to find music
with
– Not a real “music library”: does very little
• Variations2: research project => production system
10 April 06
9
What is a Digital Library?
• Not just library with computers & on-line catalog!
• DL as collection/information system
• “collection of information that is both digitized and organized”
-- Michael Lesk, NSF
• “networked collections of digital text, documents, images,
sounds, scientific data, and software” -- PITAC report
• DL as organization
• “organization that provides resources to select, structure, offer
intellectual access to, interpret, distribute, preserve integrity of,
and ensure persistence over time of collections of digital
works...”
-- Digital Library Federation
• “Elephant in the Room” for all DLs: persistence over
time = preservation
13 April 06
10
What is a Digital Music Library?
• Music has many special needs
• Content formats
– Need audio, scores; want video, maybe MIDI, etc.
• Search capabilities
– for content and metadata
• Intellectual Property Rights (IPR) => access
control important
• Traditional library catalogs don’t handle music
well
– One reason: lack of music-specific metadata
13 April 06
11
Variations and Variations2
•
•
•
•
Digital library of music sound recordings & scores
Original concept 1990, online since 1996
Variations2 started as pure research project
Now production system; replaced Variations in
2005
– Accessible by all in Music Library; other locations
restricted for IPR reasons
– Used daily by large student population
– Currently: 11,500 titles, 15,000 hours of audio
• Over 6 TB uncompressed, 1.6 TB compressed (MP3, AAC)
– Opera, songs, instrumental music, jazz, rock, world
music, etc.
13 April 06
12
Some Metadata and Digital Library Buzzwords
• MARC: metadata standard for library catalogs
–
–
–
–
From the Library of Congress
Old (1970’s): fixed format, etc.; “bibliographic”
Standard for maintaining & exchanging bibliographic information
Simple relationships, elaborate details
• Dublin Core (DC): general-purpose metadata standard
– From Dublin Core Metadata Initiative (DCMI)
– New (1990’s): XML, etc.; “metadata”
– Simple, general, extensible
– Terminology: http://dublincore.org/documents/dcmi-terms/
• Open Archives Initiative (OAI): metadata consumer
• FRBR: metadata standard for library catalogs
– From IFLA, with support from Library of Congress, etc.
– New (>2000)
– Complex relationships, elaborate details
18 April; rev. 25 April
13
Functional Requirements for Bibliographic
Records (FRBR)
• Represents much more complex relationships than MARC
– MARC records refer explicitly to subject headings (LCSH), URLs
– …and implicitly (via uniform names & titles) to other MARC
records
– …but not consistently!
– FRBR (like Variations2) records always refer to each other
• FRBR Entities
– Group 1: Products of intellectual & artistic endeavor
– Group 2: Those responsible for the intellectual & artistic content
– Group 3: Subjects of works
• Much of following from by Barbara Tillett (2002), “The
FRBR Model (Functional Requirements for Bibliographic
Records)”
25 April
14
FRBR Entities
• Group 1: Products of intellectual & artistic endeavor
1. Work (completely abstract)
2. Expression
3. Manifestation
4. Item (completely concrete: you can touch one)
– Almost heirarchic; “almost” since works can include other works
• Group 2: Those responsible for the intellectual & artistic
content
– Person
– Corporate body
• Group 3: Subjects of works
–
–
–
–
–
Groups 1 & 2 plus
Concept
Object
Event
Place
25 April
15
Relationships of Group 1 Entities: Example
w1 J.S. Bach’s Goldberg Variations
e1 Performance by Glenn Gould in 1981
m1 Recording released on 33-1/3 rpm sound disc in 1982
by CBS Records
i1a, 1b, 1c Copies of that 33-1/3 rpm disc acquired in 1984-87 by the
Cook Music Library
m2 Recording re-released on compact disc in 1993 by
Sony
i2a, i2b Copies of that compact disc acquired in 1996 by the Cook
Music Library
m3 Digitization of the Sony re-released as MP3 in 2000
25 April
16
Relationships of Group 1 Entities (1)
Work
is realized through
Expression
Intellectual/
artistic content
Physical recording of
content
Manifestation
Item
25 April
is embodied in
is exemplified by
17
Relationships of Group 1 Entities (2)
Work
is realized through
Expression
is embodied in
recursive
one
Manifestation
is exemplified by
many
Item
25 April
18
FRBR vs. Variations2 Data Models
FRBR
• Group 1
–
–
–
–
Work
Expression
Manifestation
Item
Variations2
– Work
– Instantiation
– Media object
• Group 2
– Person
– Corporate body
(any named
organization?)
– Contributor (person
or organization)
– Container
Items in blue are close, though not exact, equivalents.
25 April; rev. 26 April
19
Elephant in the Room for Music DLs: Getting
Catalog Information into FRBR or Variations2
• 2005 MLA discussion
– Cataloging to current standards (MARC) is very expensive
– FRBR and Variations2 both much more demanding
– Michael Lesk/NSF: didn’t like funding metadata projects because
“they always said every other project should be more expensive”!
– Libraries seem to be moving to FRBR anyway
• Idea 1: collaborative cataloging (ala OCLC)
• Idea 2: take advantage of existing cataloging
– Variations2: simple “Import MARC” feature
– VTLS: convert MARC => FRBR is much more ambitious
• Good ideas, but probably not enough
• Idea 3: user-contributed metadata?
13 April 06
20
Variations2 Hands-on
• Possibilities
–
–
–
–
–
–
–
–
–
Search using Variations2 search window
Search using IUCAT
External (WWW or other) access via reserve lists, etc.
Create playlist
Add bookmarks
Create listening drill from playlist
Export to make a Web page
Use Opus window
Use Timeliner
10 April 06
21
Variations2 Data Types
• Work is realized through
• Instantiation (recording or score) is embodied in
• Container (CD, LP, edition of scores, etc.)
• Contributor (person or organization)
– Contributor to work: composer, lyricist, etc.
– Contributor to instantiation: performer, conductor,
engineer, etc.
10 April 06
22
Works & Work Relationships
• Work concept is new to Variations2/FRBR
• Much more important to organize music than
(e.g.) books
– Language of title says very little about content
– Important relationships: song & album, aria & opera,
etc.
• Work relationships can be very complex
– Part/whole
– Arrangement
– Version (improvisation, etc.)
12 April 06
23
Style Genres & Genre Classifications
• Genre Classifications are a mess
• No consistency between classifications
– All-Music Guide: <=4 levels: 2 top-level (pop/classical), 34
second-level
– Amazon.com: ca. 23 main genres
– GarageBand.com: 47 genres, flat
– ID3 tags (in MP3's): 80 genres, flat; WinAmp version: 126 genres,
flat
– Ishkur's EM Guide ("Electronic music" only): <=3 levels: 7 toplevel
– iTunes: 37 genres, flat
– MIREX 2005 : 9, 38 leaves
10 April 06
24
Style Genres & Genre Classifications
• No wonder: what makes a musical style is very subtle!
• In many cases, "correct" genre can't be determined without
knowledge of the lyrics, even understanding
• …or even intent of creators
• Dave Datta (2005): automatic genre-classification
programs are finding something, & probably useful, but
may not be genres as people understand them
• Turntablism is a separate genre--or is it? If it's done
"mildly", what you'd hear is mostly the genre of the
underlying music!
13 April 06
25
Transcription of Polyphonic Audio
• Cf. OMRAS experiments: an important research problem
• Why is it so difficult?
• Guest speaker: Ian Knopke, IU fellow in music informatics
14 April 06
26
Music Plus One (1)
• Chris Raphael’s accompaniment system
• Goals of ``Music Plus One'' are similar to ``Music Minus
One'' (MMO)
• But, with MMO, soloist must follow accompaniment
• Goals: program must:
– Respond in real time to soloist's tempo changes & expression
– Learn from past performances so it assimilates soloist's
interpretation in future
– Bring sense of musicality to performance
– Components: Listen and Play
14 April 06
27
Music Plus One (2)
• Listen
– As soloist plays, signal analyzed to determine what notes have
played & exactly when
– Greatly simplified because computer “knows” score (MIDI file)
– But must be robust to inaccuracies & embellishments by soloist
while maintaining accuracy in matching signal to soloist's part
– Uses Hidden Markov Model (HMM)
• Play: fuse knowledge sources
– Output of Listen
– Score (notes, rhythms, etc.)
– To improve over successive rehearsals, use collection of past solo
performances by soloist
– Performances of the accompaniment by a live player
14 April 06
28
Accompaniment System “Variations”
•
•
•
•
Music written for system (beyond human capabilities)
Erase the soloist => greatly enlarge repertoire
Detecting beat in audio (alignment with score)...
Much easier than detecting without the score (e.g., Digital
Performer)
• For use by Variations2?
– Example: Sacrificial Dance of Le Sacre du Printemps
20 April 06
29
“Music as Different as Possible”
• Results are… interesting
– Two teams used Cage’s 4’ 33”
– All lists good, none great
• Not much world music!
–
–
–
–
Team A: Ives’ The Unanswered Question
Team B: Prefuse 73’s B2 Living Life
Team C: Japanese gagaku Keibairaku No Kyu (Taishiki-Cho)
Team D: Balinese monkey chants
• Electronic/computer music can be much more extreme
– Xenakis’s Bohor: few definite-pitched sounds
– Dodge’s In Celebration: wild synthesized “voice”
• “You sit in a chair, touched by nothing, feeling the old self…”
• Consider language + basic parameters of sound
– Pitch, duration/rhythm, dynamics, timbre
17 April 06
30
Expectation and Perception with Sponges,
Dinosaurs, and Music
• Sponges
– Contamination of kitchen surfaces before & after
cleaning with sponge surprised researchers
• Dinosaurs
– 1922 audience fooled by test reel for The Lost World
• Music
– Don’s experience with Kurzweil flute sound
– Hammond organ Model A compared to pipe organ (ca.
1940)
• In “blind” test, experts & students couldn’t tell them apart
16 April 06
31
Science, Scholarship, and Critical Thinking
• Good research is very hard
– Electronic Musician article on analog summing
• The issue isn’t just science…
– D. Huron on what he learned about music scholarship
• It isn’t just scholarship…
– 1922 audience fooled by test reel for The Lost World
• It’s critical thinking
– ALWAYS evaluate information sources
– ALWAYS consider biases, including your own
• Darwin’s attitude about his biases
• “Most people would rather die than think…”
17 April 06
32
Intellectual Property Rights (IPR) (1)
• IPR is huge problem for music IT, including IR, both
research & use
– No one knows the answers! Different in different countries!
• For music, U.S. copyright is complex “bundle of rights”
– mechanical right: use in commercial recordings, ROMs, online
delivery for private use
– synchronization right: use in audio/visual works (movies, TV, etc.)
– More complex than for text works because performing art
• U.S. Constitution: balance rights of creators and public
– “To achieve these conflicting goals and serve the public interest
requires a delicate balance between the exclusive rights of authors
and the long-term needs of a knowledgeable society.” —Mary
Levering, U.S. Copyright Office
– After some time, work enters Public Domain
26 April 06
33
Intellectual Property Rights (IPR) (2)
• Law supposed to balance rights of creators & public, but…
– Time till Public Domain getting longer & longer
– “Joke”: When will old Disney movies be Public Domain?
– Sonny Bono Copyright Extension Act: not till 70 years after death!
– Digital Millenium Copyright Act (DMCA) restricts owner’s rights
– Rep. Smith’s bill (in Congress soon?) even worse
• “Fair Use”: U.S. limit on exclusive rights of copyright
owners
– Traditionally used for excerpts for reviews, etc.
– Not well-defined. Four tests:
–
–
–
–
1.
2.
3.
4.
Purpose and character of use, including if commercial or nonprofit
Nature of copyrighted work
Amount and substantiality of portion used relative to work as a whole
Effect of use on potential market for or value of copyrighted work
• Law also has educational exemptions
26 April 06
34
Intellectual Property Rights (IPR) (3)
• NB: I’m not a lawyer!
• IPR in practice
– Mp3.com sued & shut down
– Peer-to-Peer networks: Napster, Gnutella, FreeNet
– Church choir director arranged work, did free performance;
donated to publisher => sued
• Example: Student wants to quote brief excerpts from
Beethoven piano sonatas in class paper, in notation
• Do they need permission from owner?
–
–
–
–
Beethoven dead for more than 70 years => in Public Domain
…but not all editions
Still, don’t need permission: Fair Use applies
For recording, probably not P.D., but Fair Use applies
26 March, rev. 15 April
35
Music IR as Music Understanding
• Dannenberg (ISMIR 2001 invited paper) argued central
problem of music IR is music understanding
• …also basis for much of computer music (composition &
sound synthesis) and music perception and cognition
– “A key problem in many fields is the understanding and
application of human musical thought and processing”
• Don: No understanding yet => sidestep intractable
problems!
• Cf. “how people find information” vs. “how computers
find information”
14 April 06
36
Detecting Beats/Tempo in Audio without a score (1)
• Related tasks: tempo detection & beat
detection/slicing
• What can you do with them?
– Create loops
– Change tempo radically with no artifacts
– Ask Will Pierce
• State of the art in commercial products
– Digital Performer Beat Detection Engine™
– “Employing sophisticated transient detection
technology…”
• Likely to work only with very simple texture
28 April 06
37
Detecting Beats/Tempo in Audio without a score (2)
• What else can you do? More advanced stuff:
– Change swing feel to straight 8ths (Digital Performer)
• State of the art in research systems
– MIREX 2005 Audio Tempo Extraction contest
• www.music-ir.org/mirex2005/index.php/Audio_Tempo_Extraction
–
–
–
–
Looking for notated & perceived tempo
…and phase (= upbeat)
Music w/stable tempo, wide variety of styles, many non-Western
Texture?
• With beat slicing and audio similarity:
– Violate IPR laws (Scrambled Hackz!)
• Interview: www.wired.com/news/columns/0,70664-0.html
• Video: www.popmodernism.org/scrambledhackz/?c=4
28 April 06
38
Intellectual Property Rights in The Real World
• NB: I’m not a lawyer!
• A common way for people to decide what’s OK
– Consider ethics: any problem?
– Consider practical effects: any problem?
– If no and no, go ahead
• Example: Member of this class wants to share copyrighted
music with others in the class
– Ethics: it depends
– Practical effects: ordinarily none
• Thorny issue: at what point is sampling a problem?
• Deeper, thornier issue: does IPR make sense?
– Joey Morwick: maybe not
– Ian Clarke (Freenet), Sven Koenig (Scrambled Hackz), promoters
of XOR circumvention: absolutely not!
27 April 06
39
Conclusion; Thank You
• Please, please think for yourself
– ALWAYS evaluate information sources
– ALWAYS consider biases, including your own
• Schoenberg: “This book I have learned from my
students”
• Don Byrd: “This course I have learned
from my students”
– I’ve learned a lot about music and technology
• I’m available!
28 April 06
40