#### Transcript Modeling Clutter Perception using Parametric Proto

Modeling Clutter Perception using Parametric Proto-object Partitioning Chen-Ping Yu1, Wen-Yu Hua3, Dimitris Samaras1, Gregory Zelinsky1,2 1Dept of Computer Science, 2Dept of Psychology, Stony Brook University; 3Dept of Statistics, Penn State University Experiments and Results Method Introduction - The Problems (1) Model human clutter perception using proto-objects. (2) Estimate “set size” for realistic scenes. - Superpixel Graph An image is first pre-processed into superpixels using SLIC [3], then it is formulated into a graph, where the nodes are the superpixels. Each pair of adjacent nodes are connected with a weighted edge. - Parameters a lower-bound parameter 𝜖 ∈ {0.01, 0.02, …, 0.20}, and a percentile parameter 𝜏 ∈ {0.5, 0.6, …, 0.9}. - What is Visual Clutter? A “confused collection” or a “crowded disorderly state”. Increasing visual clutter leads to poorer performance in many behavioral tasks (e.g. visual search). - What is a Set Size Effect? A drop in search performance with an increase in the number of objects [1]. However, an object count is difficult to quantify in real world scenes. - Optimization MLE using the Nelder-Mead algorithm, and Nonlinear Least Squares (NLS). - MLE Highest Spearman’s ρ = 0.8038, with 𝜖 = 0.14 and 𝜏 = 0.8. 10-fold Cross Validation = 0.7599. - NLS Highest Spearman’s ρ = 0.7966, with 𝜖 = 0.14 and 𝜏 = 0.4. 10-fold Cross Validation = 0.7375. Superpixel Graph SLIC k = 1000 - Edge Weights: Earth Mover’s Distance The edges are weighted by the dissimilarity between the pair of nodes, in terms of Intensity, Color, and Orientation. We use Earth Mover’s Distance as the dissimilarity distances. EMD is defined to minimize the following with an optimal flow : How can we quantify set size or the number of objects in these scenes? - Goal Correlate the model's clutter ranking of our 90 image dataset with the behavioral clutter rankings using Spearman's ρ. WMM-mle WMM-nls Mean-shift [6] Graph based [7] Power Law [8] Edge Density [9] Feature Congestion [10] # of Objects (SUN) [2] Color-cluster clutter [11] 0.8038 0.7966 0.7262 0.6612 0.6439 0.6231 0.5337 0.5255 0.4810 Correlations between human clutter perception and all the evaluated methods. WMM is our Weibull mixture model. Our method runs in 20 seconds using 800x600 images, on an Intel Core i7 3.0 Ghz machine with 8 Gb RAM. - What are Proto-objects? Regions of locally similar features. They can be objects, object parts, or just pieces that come together to form objects. - What does our Clutter Model do? It segments proto-objects from an image, then counts the number of proto-objects as an estimate of visual clutter. Contributions where and some dissimilarity metric (i.e. the L2 distance) between and are the two signatures to be compared, and in . denotes - Edge Labeling for Superpixel Clustering Each edge is labeled as Similar or Dissimilar, based on a similarity-threshold The dissimilar edges are removed to form superpixel clusters, which are merged to form proto-objects. . - Clutter Model Our model successfully predicts the degree that a person will perceive an image as cluttered, and out-performs all other existing models of clutter perception. 0.15 0.15 - Parametric Modelling of Earth Mover’s Distance Statistics We show that Earth Mover’s Distance statistics (EMD) follow a Weibull distribution for efficient parametric modeling. 0.11 0.11 0.86 0.63 0.28 0.75 0.86 0.12 0.04 = 0.6 0.21 0.82 0.81 0.21 0.93 0.65 Dataset - 90 800x600 real world images, sampled from the SUN Database [2] - Divided into 6 groups, each with a different range of object counts (from SUN09). 0.38 protoobjects 0.81 0.65 0.68 0.68 0.71 merge 0.93 0.32 0.32 - Clutter Dataset We obtained a clutter ground truth by having people rank order a subset of images from SUN09 [2] from least to most cluttered. 0.04 0.31 0.31 0.82 0.75 0.12 0.77 0.77 - Proto-object Segmentation Unsupervised image partitioning by our novel parametric EMD model. 0.35 0.35 0.63 0.28 Four sample images from our dataset. Human clutter ranking from left to right: 6, 47, 70, 87; Proto-object model’s ranking using the best-tuned parameter setting (𝜖 = 0.14, 𝜏 = 0.8): 7, 40, 81, 83. 0.77 0.77 0.71 0.38 0.05 0.23 0.05 0.23 0.75 0.75 - Compute using Weibull-Mixture-Model EMD is identical to Mallow’s Distance, , when P and Q have the same total mass [4], and Lp-based distance statistics follow a Weibull distribution [5]. Therefore, a twocomponent WMM (similar/dissimilar) can be used for the computation of . - Clutter rankings (15 raters) and object segmentations (SUN) available for each image - Mean correlation between all pairs of human ranking: Spearman’s ρ = 0.6919 Orientation Weibull-Mixture Model (WMM): Color Intensity 3 5 7 31 32 33 51 52 53 Similarity Threshold – the crossing point between the two components: Application to parameter-free Image partitioning: Use only 2-component WMM and does not enforce the lower-bound parameter 𝜖. 7 9 10 36 37 39 55 57 1~10 objects 31~40 objects 51~60 objects 15 images 15 images 15 images 90 images total References & Acknowledgment 58 - Normalized Clutter Measure The count of the final proto-objects are divided by the initial # of superpixels to produce our final clutter measure for a given image. [1] J. M. Wolfe. Visual search. Attention, 1998. [2] J. Xiao, J. Hays, K. Ehinger, A. Oliva, and A. Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010. [3] R. Achanta, A. Shaji, L. Smith, A. Lucchi, P. Fua, and S. Susstrunk. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE TPAMI, 2012. [4] E. Levina and P. Bickel. The earth mover’s distance is the mallows distance: some insights from statistics. In ICCV, 2001. [5] G. J. Burghouts, A. W. M. Smeulders, and J.-M. Geusebroek. The distribution family of similarity distances. In NIPS, 2007. [6] D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space analysis. IEEE TPAMI, 2002. [7] P. F. Felzenszwalb and D. P. Huttenlocher. Efficient graph-based image segmentation. In ICCV, 2004. [8] M. J. Bravo and H. Farid. A scale invariant measure of clutter. Jounal of Vision, 2008. [9] M. L. Mack and A. Oliva. Computational estimation of visual complexity. In the 12th Annual Object, Perception, Attention, and Memory Conference, 2004. [10] R. Rosenholtz, Y. Li, and L. Nakano. Measuring visual clutter. Journal of Vision, 2007. [11] M. C. Lohrenz, J. G. Trafton, R. M. Beck, and M. L. Gendron. Amodel of clutter for complex, multivariate geospatial displays. Human Factors, 2009. We appreciate the authors of C3 model, Dr. Burghouts of [5], and Dr. Matthew Asher for discussions and code sharing. This work was supported by NIMH Grant R01-MH064748 to G.J.Z., NSF Grant IIS-1111047 to G.J.Z. and D.S., and the SUBSAMPLE Project of the DIGITEO Institute, France.