Transcript Style Space: What we are learning about cultural variability and categories by studying 1000000 Manga pages research project by Software Studies Initiative, University of California.

Style Space:
What we are learning about cultural
variability and categories by studying
1000000 Manga pages
research project by Software Studies Initiative, University of
California - San Diego (UCSD) and
Multimodal Analysis Lab, National University of Singapore (NUS)
project report by Lev Manovich
June 2010
team @Software Studies Initiative
Sunsern Cheamanunkul (PhD student, Computer Science)
Jeremy Douglass (Post-doc, Software Studies Initiative)
William Huber (PhD student, Visual Arts; GRA, Software Studies)
Lev Manovich (director, Sofware Studies Initiative)
So Yamaoka (PhD student, Computer Science)
team @Multimodal Analysis Lab
Bertrand Grandgeorge (statistical analysis, programming)
Bin (Ong Kian Peng) (visualization)
Cultural Variability
In descriptive statistics, statistical dispersion (also called
variation) is defined as variability or spread in a variable.
Common examples of measures of statistical dispersion are
the variance and standard deviation.
What kinds of variance can we observe in cultural phenomena,
artifacts and processes?
Is independently produced culture more
varied than the products of cultural
industry?
Did the adoption of digital tools led to more
diversity?
Does globalization indeed decreases
cultural diversity as it commonly believed?
our sample:
883 Manga titles consisting from
41863 chapters
1074790 pages
source: onemanga.com
(top site for manga scans translated
into English)
title: Biomega
author: Nihei Tsutomu
Manga in a bookstore in Dubai (March 2010)
questions:
What is the variability of such large sample
of human creativity? Are there substantial
stylistic differences between Manga
created in different countries? Or Manga
created for different audiences Or different
Manga genres? Or different titles? How
unique is each Manga artist? What kinds of
patterns of variability can we find? How
can we visualize these patterns? How to
define and model cultural variability?
methodology:
images:
image processing of every page to
extract features
exploratory data analysis
statistical analysis
supervisualization
gender/genre/country tags:
statistical analysis
visualization
image processing:
using our software running on
supercomputers at Department of
Energy Research Center (NERSC) we
analyzed each page to measure 8
global visual features
Brightness mean
Std
Entropy
Sobel (the amount of edges found)
Contrast
Correlation
Energy
Homogeneity
preparing data for exploratory analysis:
identify special pages (color, cover, etc)
remove special pages from average calculations
calculate measurement averages and stdev per book, per title, per
author
calculate PCA of feature averages and stdev
final data:
1) data per page: 54 columns 1074790 rows
2) measurement averages per book: 54 columns 41863 rows
3) measurement averages and PCA per series: 54 columns 883 rows
4) measurement averages and PCA per author: 54 columns 110 rows
exploratory data analysis and
supervisualization
main software:
Mondrian (exploratory data analysis)
Matlab (statistical analysis)
VisualSense
(custom Flash visualization application which supports image graphs
- up to 1000 small images)
ImageGraph
(custom imageJ macro for non real-time rendering high-res
visualizations - no image number limit - were able to render
visualization with 1 million Manga pages)
HIPerSpace Explorer
(custom application for interactive visualization running on display
walls - real-time performance with up to 4000 images of any size)
Use of low-level statistical image features
for Manga analysis (and cultural analytics in
general)
Many low-level statistical features in combination are used for computational
analysis of visual style and other characteristics of the data (PCA, machine
learning, etc)
Some single features only relevant for such analysis and don’t have direct
meaning for humans; other correspond to concepts which have perceptualcognitive meaning (depends on the data set) - for example, average
saturation for paintings, or enthropy for Manga images (indicates presence of
textures which likely also means more detail, more realism and more
production labor)
In the case of the low-level statistical features which do have direct
perceptual/cognitive meaning, plotting single feature over time or
using 2 features in a scatter plot creates meaningful visualization
which can reveal patterns in time or relations between different data
sets
Even if such visualizations may not immediately tell the reason for a
pattern, the difference in feature values is likely to be important - it
indicates that something is changing in time, or that data sets are
different in some ways - the researcher can then examine the data to
learn the meaning of these patterns. Therefore a line plot of even
most simple features (such as brightness mean) reveals patterns of
change (i.e., there must be some reason for the changes in
brightness values)
1 million Manga pages organized by visual properties
Each page is rendered as one point
X = stdev Y=entropy
Style space: the space of all possible
variations (feature values)
Style: a part of the space defined by a
range of feature values*
Mass: the portion of the space filled in by
the actual artifacts
Density: number of observed artifacts per
standard volume unit of style space
* which range of feature values and on how many dimensions can’t be
determined theoretically
it depends on the mass in the style space, historical context, and the “style
literacy” - the ability of a viewer to read distinctions
a group of artifacts which taken by themselves can be divided into a few
styles would appear as having one style if other very different artifacts are
added to the space
X = mean of brightness Y = mean of stdev. All possible images would be inside the semi-circle.
With small data sets (such as a few hundred paintings by a modern
artist) the style space is filled in uneven manner, and there are gaps
between images
With certain larger cultural sets we may expect the data to form some
approximation of a Gaussian distribution - the central area will be denser
(i.e. will contain most of the data)
Core: if most of the data is inside a single region where density > p, we
call this cluster “core”
( p: core threshold )
average distance: average distance between any two artifacts (inside a
core if it exists)
Examples of small cultural data sets which do not have cores:
Left: 123 paintings by Piet Mondrian, 1904-1917.
Right: 205 paintings by Mark Rothko, 1934-1970.
X = mean of brightness
Y = mean of saturation
Each object in a set is sufficiently different from the rest (as we may expect for a modern artist who aims to create
works which are never precise copies of each other) - average distance is large
X = mean of brightness Y = mean of stdev.
X = mean of brightness Y = mean of stdev.
example of a short Manga title (433 pages) - a pattern with core
X = mean of brightness Y = mean of stdev.
1,000,000 Manga pages.
X = brightness mean Y = brightness stdev.
Manga universe has a core
When we graph a part of our Manga data set larger than a
particular number (such as 50,000 pages or %5 of the data), we
see a single cluster with density > p
p - the difference in feature values below the level of human
perception
ADD the density number for mean feature for Manga pages core
Within the part of the space defined by the core, all possible
variations are actualized
Cultural evolution can explore all possibilities inside parts of style
space
histogram of mean averages per title
histogram of stdev averages per title
example of a smaller data set with core but lower density:
4137 Magnum photos (Calcutta, Hong Kong, Moscow, 1947-2009)
X - mean; Y- stdev
Visual style in Manga core varies
continuosly (on some dimensions)
therefore if we are to assign discrete
linguistic categories to regions of Manga
universe, the cuts would be arbitrary
We can state this because:
1) quantitative analysis of images allows us to describe differences in
data with arbitrary level of precision (or at least below the threshold of
human perception)
2) the analysis of Manga pages reveals a core (in some visual dimensions)
- a region with density > p
1 million Manga pages organized by visual properties
Each page is rendered as one point
X = stdev Y=entropy
5% of our data set (50K pages) organized by visual properties
X = stdev Y=entropy
visualization full size: 40000x40000 pixels
Image size: 2%
The edge of Manga universe:
pages which have high stdev, low entropy
The edge of Manga universe:
pages which have
middle stdev values and high entropy values
Manga universe has voids some stylistic possibilities are never
observed in practice
Since many visual features vary together we can’t expect that a 2D
scatter graph of two features will be completely filled with data
points even in the case of a massive data set
However some stylistic possibilities which can in principle appear
are still never observed in our Manga set
Cultural evolution does not actualize all possibilities
Cultural variability is not infinite
X = stdev Y = entropy
all 1 million pages (black); all pages of Korean titles (red). This graph suggests that even with much larger
sample the style space will still have significant voids.
Manga universe is
inhomogeneous and anisotropic
(some directions are more important)
Within Manga mass, some choices are used more often than
others
Cultural evolution is not anisotropic
genre and style
Manga tags: folksonomy which became taxonomy
Fans use categories to tag genres
First developed by fans these categories now are standard in the
industry
Our data set contains 35 distinct tags assigned by fans
Tags code genre (action, comedy, drama, fantasy, etc.),
country of production (Japan, Manwha/Korea, Manhua/China)
audience (Shoujo/teenage girls, Shōnen/teenage boys, Seinen/young
men, Josei/young women)
tag frequency (number of titles with each tag in our data set)
Manga folksonomy: overlapping
categories
Most titles have multiple genre tags
Example: tags for first 10 titles in our data set
X = number of tags assigned to a title
Y = frequency
Groups with different genres tags
strongly overlap in style space
blue - titles with tag drama.romance (25)
violet - titles with tag drama.adventure.fantasy (22)
X - PCA1 Y - PCA2
Parallel coordinates
Classical view of categories (Aristotle): categories are discrete entities
defined by sets of properties shared by their members; categories have firm
boundaries; each concept is either inside or outside a given category (i.e.,
apple and mandarin are fruits; cucumber and cabbage are vegetable)
Prototype theory (Rosch and Lakoff): categories are graded membership;
some objects are more central to a category than others (i.e. apple is “more”
of a fruit than a mandarin);
category boundaries are fuzzy; categories may have more than one focal
point
in Manga style space all categories (i.e. groups of titles as defined by user
tags) strongly overlap
The overlap in style space parallels the
overlap in genre space
most titles are marked by fans with a number of genre tags as opposed to
just one genre tag
(The data sets contains groups of titles with a unique tag combinations but these groups are too small for meaningful statistical test of style
difference. Similarly, we can’t test if the differences in style between data
subsets which only have one single genre tag and the rest are important
because such subsets are even smaller)
What can we learn about
genres - or cultural categories
in general - by using cultural
analytics?
If we assume that tags data reflects fans perception of Manga
fans (as opposed to tagging technology), genres are
combinations of modular traits
However it maybe safer to assume that different
representational technologies will shape the models of
cultural categories (genre, gender, etc) in their image
For example network visualization may suggest that genres
are nodes in a network
The same applies to models and methods from scientific
fields we can borrow to talk about categories: phenotypegenotype (biology), feature vector, machine learning
(computer science), fuzzy sets (mathematics), semantic
features (linguistics)
connections between 7 most frequent genre tags in Manga data set
(not including gender tags)
Phrase Net visualization (manyeyes)
details about Phrase Net visualization
technique from manyeyes (in our case each
“phrase” means the tags for a single title; 876
titles total):
The program will create a network diagram of
the words it found as matches. Two words will
be connected if they occurred in the same
phrase. The size of a word is proportional to
the number of times it occurred in a match; the
thickness of an arrow between words tells you
how many times those two words occurred in
the same phrase. The color of a word indicates
whether it was more likely to be found in the
first of second slot of a pattern. The darker the
word, the more often it appeared in the first
position.
connections between 10 most frequent genre tags in Manga data set
(not including gender tags)
connections between all genre tags combinations in Manga data set
(not including gender tags)
shōjo
manga
refers to a story serialized in a shōjo manga
magazine (a magazine marketed to girls between 10 and 18)
shounen
manga
refers to a story serialized in a shounen
manga magazine (a magazine marketed a male audience between the
ages of 10 and 18)
gender and genre: connections between top 5 tags in our data set
(including gender and genre tags)
gender and genre: connections between top 12 tags in our data set
(including gender and genre tags)
Note that none of top genre tags is gender exclusive
manga.titles.X_PCA1.Y_PCA2.red_shounen_boys
manga.titles.X_PCA1.Y_PCA2.red_shoujo_girls
parallel coordinates
Shonen - pink Shoujo - blue
X = stdev Y = entropy
Shonen - pink Shoujo - blue
This difference is statistically significant
feature: mean
Kruskal-Wallis Test (Nonparametric ANOVA)
Dunn's Multiple Comparisons Test
Comparison
P value
shoujo
/girls vs. shounen /boys
*** P<0.001 shoujo
/girls vs. seinen /men
*** P<0.001
shounen /boys vs. josei /women
*** P<0.001
seinen /men vs. josei /women *** P<0.001
shounen /boys vs. seinen /men
ns P>0.05
shoujo /girls vs. josei /women
ns P>0.05
analysis of 1 million Manga pages
provides empirical evidence that visual
style is used to construct gender
differences
Note: since fans can only assign one gender tag for each title this finding
may be an artifact of the tagging system
In Manga gender/style spaces strongly
intersect
The gender/style spaces do not form
exclusive sets
Note: since fans can only assign one gender tag for each title this finding
may be an artifact of the tagging system
histograms of mean averages for shoujo/girls titles (blueå) and shounen/boys titles
(pink)
x axis: 0 (min) - 255 (max)
since the number of title for each category is different, histograms are scaled to the
same dimensions
in progress: is there a difference in style
between artists who only create titles for
one gender and the artists who create titles
for both?
the following network visualization connects artists to gender categories of
their titles
it shows artists with titles for one audience segment vs. the artists with titles
for > 1 segment
what categories are more stylistically
diverse?
standard deviation for all titles for each
gender/age category (calculated over PCA
1):
shoujo/girls 10.7 (235 titles)
josei/women 14.8 (19 titles)
shounen/boys 13 (375 titles)
seinen/men 20 (183 titles)
popularity and style: individual authors
and titles
first 3 titles on onemanga.com
1) One Piece (title id 2)
2) Bleach (title id 3)
3) Naruto (title id 4)
top 2008 titles in Japan (sales)
(source: comipress.com/article/2008/12/31/3733):
1) One Piece (title id 2)
2) Naruto (title id 4)
3) 20th Century Boys (title id 159)
4) Bleach (title id 3)
5) Nana (title id 8)
All data (1 million pages) organized by visual properties - rendered in red
pages for titles id 2, 3, 4 rendered in blue
X = stdev Y=entropy
3 most popular titles have almost identical
mean and stdev averages
brightness mean
1) One Piece (title id 2) - 184.16
2) Bleach (title id 3) - 182.65
3) Naruto (title id 4) - 180.794
brightness stdev
1) One Piece (title id 2) - 91.775503
2) Bleach (title id 3) - 91.646764
3) Naruto (title id 4) - 91.127849
manga.titles.Xmean.Ystdev.red_title_id_2_3_4
manga.titles.X_PCA1.Y_PCA2.red_title_id_2_3_4
Legend for the graph on the previous slide:
All titles in PCA space
X axis - PCA dimension 1
Y axis - PCA 1 dimension 2
PCA values were calculated over means of 8 features (averaged
per title)
red points: first 3 titles id in our data set
(these are the most popular titles with English language readers):
1) One Piece (title id 2)
2) Bleach (title id 3)
3) Naruto (title id 4)
top 2008 titles in Japan (sales)
(source: comipress.com/article/2008/12/31/3733):
1) One Piece (title id 2)
2) Naruto (title id 4)
3) 20th Century Boys (title id 159)
4) Bleach (title id 3)
5) Nana (title id 8)
the top 3 titles have significant differences on other visual dimensions
One Piece
Naruto
Bleach
manga.titles.parallel_coordinates.red_title_id_2_3_4
manga.titles.parallel_boxplot.red_title_id_2_3_4
1) OnePiece
2) Bleach
3) Naruto
variability of top 5 titles
One Piece (title id 2)
Naruto (title id 4)
20th Century Boys (title id 159)
Bleach (title id 3)
Nana (title id 8)
manga.titles.X_log_PCA1_var.Y_log_PCA2_var.red_top_5_titles
in progress: identifying patterns and clusters
in style space for genres, authors, titles
Matrix of scatter plots visualization technique allows us to compare patterns
across many titles
montage,manga.pages.Xstdev_Yentropy.12_titles_similar_length
montage.manga.pages.Xstdev_Yentropy.12_titles_of_one_author
examples of different patterns
manga.pages.all.Left_all.Middle_title71.Right_title262
next slide: Scatter plot matrix of all Manga titles in our data set.
X - mean, Y -stdev
Signatures: artists, titles, genres
Parallel coordinates visualization techniques allows us to visually
represent unique “signatures” of artists, titles and genres
to add:
Signatures: artists, titles, genres
to add:
Signatures: artists, titles, genres
in progress: how many authors have
distinct stylistic signatures?
stylistic signature = average feature values for all titles of a given author
fall within a range < r
r can be defined as a fixed percentage of the value range (for instance %15)
or by perceptual examination - the difference in values within r range
should not be perceived as changing style
based on visual examination - confirm with stats:
Our of 14 artists which have more than 4 titles in our data set
10 have distinct stylistic signatures (%71)
Out of 19 artists which have 4 titles in our data set
14 have distinct stylistic signatures (%73)
example of an author with distinct style: Nihei Tsutomu
parallel boxplot (each red line corresponds to one of his titles)
example of an author which varies his style between titles: Watase Yuu
parallel boxplot (each red line corresponds to one of his titles)
cultural outliers
“An outlying observation, or outlier, is one that appears to deviate
markedly from other members of the sample in which it occurs.”
Grubbs, F. E.: 1969, Procedures for detecting outlying
observations in samples. Technometrics 11, 1–21.
Outliers: The Story of Success
title of the 2008 book by Malcolm Gladwell
histogram of PCA 1 for variability per Manga title
NOISE
abara
example of outliers: titles with most variability:
NOISE and abara by Nihei Tsutomu (red points)
X - PCA 1; Y - PCA 2
scatter plot matrix for 8 features - book averages
red points - NOISE and abara by Nihei Tsutomu
parallel coordinates
graph for 8 features
red lines - book
averages in NOISE
and abara by Nihei
Tsutomu
in both titles the
values for most
features lie outside
of the center
abara
NOISE
Examination of title pages suggests a hypothesis: Tsutomu systematically varies style
from book to book
Visualization confirms this hypothesis: changes in feature values over title length
(book averages) for NOISE and abara by Nihei Tsutomu
Each line represents changes in one feature (book average) over title length
NOISE: temporal patterns at different scales
each time series corresponds to page in one book
book title pages are excluded
books 2 - 5: pages become gradually lighter
books 3 - 6: overall arc pattern
in progress:
Exploring temporal patterns in all 883 Manga
titles
Does visual style change over the life of a
title?
longest title (number of pages):
15978 pages (Haime no Ippo)
top 3 titles:
One Piece: 10849 pages / 563 chapters
Bleach: 8089 pages / 403 chapters
Naruto: 12997 / 594 chapters
abara: 384 pages
NOISE: 183 pages
numbers correspond to manga pages available on onemanga.com in Fall 2009
title: One Piece. Length: 10000 pages+
X: page number Y: mean of page brightness
color indicates position of the page in One Piece title
blue - beginning; red - end (as of fall 2009)
X - stdev; Y - entropy
One Piece:
book 5
(page 10)
One Piece:
book 200
(page 5)
One Piece:
book 400
(page 10)
One Piece:
book 588
(page 10)
in progress:
recognizing patterns of change
We can identify different patterns of temporal change across a title:
no change,
abrupt changes,
gradual changes,
acceleration,
etc.
(This is similar to types of temporal changes in animation - however in Manga
some changes may happen over many years of publication and thousands of
pages)
Once we identified examples of such patterns in a single Manga titles, we can
then use computer analysis to detect these patterns in the rest of the title, and
also in other titles
Some titles have a consistent style
We may also expect that a title like Naruto
which has been running for many years will
change its title over time (we can visualize
this)
However, even in some very short titles
visual style can vary dramatically - making it
meaningless to talk about a single “style” of
such titles
example of a title where style
changes significantly:
title_1922.pages_all.Xstdev.Yentopy
color indicates position of the page in the title sequence
title_1922.pages_all.Xstdev.Yentopy
comparison: developments of artists over time