Transcript PPT
New Features and Insights for Pedestrian Detection Stefan Walk, Nikodem Majer, Konrad Schindler, Bernt Schiele 1 Outline • • • • • • Authors Abstract Main contributions Algorithms Experiments Conclusion 2 Authors (1/4) • Stefan Walk – Experience • 2007-, PhD Candidate in Computer Science, Technische Universität Darmstadt • 2003-2007, Diploma in Physics, Technische Universität Darmstadt, Germany 2007 – Research interest • People Detection • Detecting from video data (utilizing motion information) – Papers • Multi-cue Onboard Pedestrian Detection (CVPR09) 3 Authors (2/4) • Nikodem Majer – Experience • 2007-, PhD Candidate in Computer Science, Technische Universität Darmstadt – Research interest • … – Papers • … 4 Authors (3/4) • Konrad Schindler – Experience • 2009-: assistant professor, TU Darmstadt, Germany • 2007-2008: post-doc, ETH Zurich • 2004-2006: post-doc, Monash University, Melbourne/Australia • 2001-2003: research assistant, Graz University of Technology, Austria – Research interest • computer vision (3D scene analysis, biologically inspired vision, tracking) • image processing, pattern recognition, machine learning, photogrammetry – Papers • PAMI10, CVPR10, ICCV10… 5 Authors (4/4) • Bernt Schiele – Experience • 1999-2004, Assistant Professor, ETH Zurich, Switzerland • 1997-2000, Postdoctoral Associate and Visiting Assistant Professor, MIT and Cambridge, MA, USA • 1994, Visiting researcher at CMU • AE of PAMI, IJCV, AC of ECCV’08, CVPR’09, ICCV’09, PC of ICCV 2011 – Research interest • Perceptual computing, human-computer interfaces – Papers • … 6 Outline • • • • • • Authors Abstract Main contributions Algorithms Experiments Conclusion 7 Abstract (1/2) • Despite impressive progress in people detection the performance on challenging datasets like Caltech Pedestrians or TUD-Brussels is still unsatisfactory • In this work we show that motion features derived from optic flow yield substantial improvements on image sequences, if implemented correctly—even in the case of low-quality video and consequently degraded flow fields • Furthermore, we introduce a new feature, self-similarity on color channels, which consistently improves detection performance both for static images and for video sequences, across different datasets. In combination with HOG, these two features outperform the state-of-theart by up to 20%. 8 Abstract (2/2) • Finally, we report two insights concerning detector evaluations, which apply to classifier-based object detection in general • First, we show that a commonly under-estimated detail of training, the number of bootstrapping rounds, has a drastic influence on the relative (and absolute) performance of different feature/classifier combinations • Second, we discuss important intricacies of detector evaluation and show that current benchmarking protocols lack crucial details, which can distort evaluations 9 Outline • • • • • • Authors Abstract Main contributions Algorithms Experiments Conclusion 10 Main contribution • First, we introduce a new feature based on selfsimilarity of low level features, in particular color histograms from different sub-regions within the detector window • The second main contribution is to establish a standard what pedestrian detection with a global descriptor can achieve at present, including a number of recent advances which we believe should be part of the “best practice”, but have not yet been included in systematic evaluations • Our third main contribution are two important insights that apply not only to pedestrian detection, but more generally to classifier-based object detection. (1)Bootstrapping is very important. (2)The existing evaluation protocol is insufficient 11 Outline • • • • • • Authors Abstract Main contributions Algorithms Experiments Conclusion 12 Outline • 本文的风格与该实验室文章一贯的风格类似 – 在自己提出的两个数据库上(Caltech Pedestrian, TUD-Brussel)测试 当前人体检测领域不同的特征与不同的分类器，评价这些算法的优 劣(性能越高的算法关注度越高) – 自己提出新特征并通过实验给出结论——“在原始方法的基础上引 入我们的特征可以进一步提升人体检测系统的性能” • Related Features – Haar-like, VJ 2001年成功用于人脸检测领域 – HOG (Histogram of Oriented Gradient), Dalal 2005年成功用于人 体检测领域 – HOF (Histogram of Flow), Dalal 2006年提出，应用于视频人体检测 – HOG-LBP 王晓宇 2009年应用于人体检测领域，高性能 – CSS (Color Self-similarity), 本文提出 • Related Classifiers – SVM – MPLBoost (Multiple Pose Boosting), Dollar 2008年提出 13 Haar-like feature (1/2) • Haar-like feature – 图像内部特定模式的两个矩型内部像素和之差 – 采用积分图可以快速计算Haar特征响应值 • Haar特征的变种 – 45, 22.5, 11.25度…，仍然受限于“矩形” – 任意多边形区域形状的Haar特征(CVPR10) 传统Haar特征 Haar特征的积分图计算 14 Haar-like feature (2/2) • 任意形状的Haar特征 – 任意多边形区域的像素和可以等价为一系列梯形区域的像素和 – 梯形区域的像素和等价于两个直角三角形的像素差 – 算法关键是计算直角三角形区域的积分图，参数(x,y,斜率) 15 HOG feature (1/1) • HOG feature-梯度方向直方图 – 输入图像的Gamma校正 – 计算输入图像各像素的梯度幅值与方向 – 梯度幅值高斯加权，使用三线形插值计算各个单元梯度方向的直方 图 – 相邻的单元直方图归一化得到最终的特征向量 HOG特征计算流程 HOG特征的三线性插值 16 HOF feature (1/1) • HOF feature-光流直方图 – 计算输入图像的x、y方向的光流 (例如LK算法等等) – 对于特定区域对，根据对应像素点的x、y方向光流差异，计算光流 梯度幅值与方向 – 根据光流梯度方向使用光流梯度幅值构建直方图 Original 3x3 IMHwd (Internal Motion Boundary wavelet diff.) 17 HOG-LBP (1/1) • HOG-LBP feature：将HOG与LBP串联起来 – HOG：将三线性插值与高斯加权替换为卷积 – LBP (Local Binary Pattern)：局部区域的二值模式 – 该特征在INRIA人体数据库上取得了迄今为止的最好结果 LBP特征示意 18 CSS (1/1) • CSS feature：颜色自相似度 – 对于8x8的图像区域，采用三线性插值计算颜色直方图 – We experimented with different color spaces, including 3x3x3 histograms in RGB, HSV, HLS and CIE Luv space, and 4x4 histograms in normalized rg, HS and uv, discarding the intensity and only keeping the chrominance. Among these, HSV worked best, and is used in the following – 利用这些直方图之间的相似度作为特征向量，作者尝试了L1-norm ，L2-norm, Chi-square distance与直方图交，发现直方图交性能作 为优秀 – 在实现中，对于64x128的窗口划分为8x16=128个8x8区域，得到128 个直方图，直方图相似度一共有128x127/2=8,128个 • Furthermore, second order image statistics, especially co-occurrence histograms, are gaining popularity, pushing feature spaces to extremely high dimensions 19 Classifiers • SVMs – Linear SVM – Histogram Intersection Kernel SVM (HIKSVM) • MPLBoost: Multiple Pose Boosting (In ECCV08 workshop) – 将初始训练样本分成K个子集，同时训练K个强分类器，分类器输出 值是这K个分类器响应值的最大值 – 在训练过程中，只有被所有强分类器错分的样本权值保持不变，否 则该样本权值降低 – 在检测过程中，对于一个扫描窗口，如果有一个强分类器认为是 positive就是positive，如果所有强分类器认为是negative才是 negative 20 Evaluation protocol (1/4) • 人体检测系统衡量标准的不合理之处 – 现阶段用于确定“一个检测窗口是否命中一个人体”依据VOC准则 ，交并比>50% – 没有明确规定如何应对人群中人体与检测框的匹配问题 21 Evaluation protocol (2/4) • We split the set of annotations and detections into considered and ignored sets • Annotations can fall into the ignored set because of size, position, occlusion level, aspect ratio or non-pedestrian label in the Caltech setting • Detections can fall into the ignored set because of size. E.g. if we wish to evaluate on 50-pixel-or-taller, unoccluded pedestrians, any annotation labeled as occluded and any annotation or detection <50 pixels falls in the ignored set 22 Evaluation protocol (3/4) • For considered detections – If they match a considered annotation they count as true positive – If they match no annotation, or only one that has already been matched to another detection, they count as false positive – If they match an ignored annotation they are discarded • For ignored detections – If an ignored detection matches an ignored annotation, it should be discarded – If an ignored detection matches no annotation, it seems reasonable to discard it, but this may introduce a bias – If an ignored detection matches a considered annotation, count it as a true positive 23 Evaluation protocol (4/4) • To summarize, there is no single correct way how to evaluate on a subset of annotations, and all choices have undesirable side effects • It is therefore imperative that published results are accompanied by detections, and that evaluation scripts are made public • As there are boundary effects in almost any setting (all realistic datasets have a minimum annotation size), it must be possible for others to verify that differences are not artifacts of the evaluation 24 Outline • • • • • • Authors Abstract Main contributions Algorithms Experiments Conclusion 25 Database • INRIA人体数据库 • CalTech人体数据库 – – – – 2009年Dollar提出 视频序列 训练集包括192k人体，测试集155k人体 各种困难的情况，光照、遮挡、小尺度(人体高度3像素的都有)、人 群… – 标注非常完善，方便测试检测器的各种特性 • TUD-Brussel数据库 – 2009年Wojek提出 – 视频序列 – 仅有训练集，包括1,326个人，各种尺度各种视角 • 所有实验训练样本尺寸统一64x128，人体大小48x96，对齐 26 Experiment1 – HOG-LBP (1/1) INRIA TUD • However, while we were able to reproduce their good results on INRIA Person, we could not gain anything with LBPs on other datasets. They seem to be affected when imaging conditions change (in our case, we suspect demosaicing artifacts to be the issue) 27 Experiment2 – Color information (1/2) TUD TUD • More than 1fppi is usually not acceptable in any practical application • Self-similarity of colors is more appropriate than using the underlying color histograms directly as feature • On the contrary, adding the color histogram values directly even hurts the performance of HOG 28 Experiment2 – Color information (2/2) • Why CSS is effective? – Self-similarity encodes relevant parts like clothing and visible skin regions • Why directly using color information shows no improvements? – The training data was recorded with a different camera and in different lighting conditions than the test data, so that the weights learned for color do not generalize from one to the other. (Similar reason to Haar feature) 29 Experiment3 – Bootstrap (1/2) • With less than two bootstrapping rounds, performance depends heavily on the initial training set • At least two retraining rounds are required in HOG+linear SVM framework • This problem will be alleviated by using more initial negative samples, not solved 30 Experiment3 – Bootstrap (2/2) • For boosting classifiers (Fig. 3(c))3, the situation is worse: although mean performance seems stable over bootstrapping rounds, the overall variance only decreases slowly—the initial selection of negative samples has a high influence on the final performance even after 3 bootstrapping rounds 31 Experiment4 – Seed & self similarity(1/1) TUD • Self-similarity on HOG blocks shows little improvement • It is important to make sure the result does not depend on the initial selection of negative samples, e.g. by retraining enough rounds with SVMs 32 Experiment5 – CalTech pedestrian (1/2) 33 Experiment5 – CalTech pedestrian (2/2) • Color self-similarity is indeed complementary to gradient information • The motion information contributes greatly on pedestrian detection. The reason that HOF works so well on the “near” scale is probably that during multi-scale flow estimation compression artifacts are less visible at higher pyramid levels, so that the flow field is more accurate for larger people • The performance of all evaluated algorithms is abysmal under heavy occlusion 34 Experiment6 – Haar feature (1/1) TUD • Judging from the available research our feeling is that Haar features can potentially harm more than they help 35 Outline • • • • • • Authors Abstract Main contributions Algorithms Experiments Conclusion 36 Conclusion • 主要结论 – – – – 运动信息会对视频中的人体检测起到很大的促进作用(HOG) 颜色近似度对于人体检测器的性能有很大的提升(CSS) Bootstrap在检测器的学习过程起到关键作用 现阶段的物体检测评价标准不合理… • 次要结论 – LBP仅仅对于INRIA数据库有效 – HOG-linear SVM至少需要2轮bootstrap – 使用Haar特征辅助人体检测可能弊大于利 37 Thanks!! 38