Transcript Concolic 테스팅 과제의 산업적 성취 13 2011년
Software Testing & Verification Group
Moonzoo Kim
http://swtv.kaist.ac.kr
Main Research Theme
• •
To Improve SW reliability through automated testing
– Quality attribute for minimizing malfunctions of systems to reduces damage to human life or valuable properties (i.e., to fight software bug)
Highly reliable SW technology is a key to the success of industrial products
– The portion of SW in embedded devices increases continuously Intelligent Medical Devices Home Service Robots
Introduction of SWTV group
Home Network
Highly Reliable Software-intensive Systems
2/14 Intelligent Mobile Systems
SWTV group.
CS Dept. KAIST
How to Assure Quality of SW?
Introduction of SWTV group
3
SWTV group.
CS Dept. KAIST
Keynote by Bill Gates @ OOPSLA, 2002
• • •
“… When you look at a big commercial software company like Microsoft, … We have as many testers as we have developers. Testers basically test all the time, and developers basically are involved in the testing process about half the time …” “… We're not in the software industry; we're in the testing industry , and writing the software is the thing that keeps us busy doing all that testing.
” “…The test cases are unbelievably expensive; in fact, there's more lines of code in the test harness than there is in the program itself.
Often that's a ratio of about
three to one
.
” Introduction of SWTV group
4
SWTV group.
CS Dept. KAIST
Significance of Automated SW Analysis
• • •
Software has become more ubiquitous and more complex at the same time Human resources are becoming less reliable and more expensive for highly complex software systems Computing resources are becoming ubiquitous and free
– Tencent @ China provides 10TB storage free – Amazon EC2 price: you can use thousands of CPUs @ 0.057$/hr for 3.2Ghz Quad-core CPU •
Remaining task?
– To develop automated and scientific software analysis tools to utilize computing resource effectively and
Introduction of SWTV group
efficiently 5
SWTV group.
CS Dept. KAIST
Research Methodology Science
•Discrete math •Algorithm •PL
SW TV group
Engineering
•System programming •SW Engineering •Security
Embedded Systems Software Engineering
System modeling
Programming Languages
System spec.
Introduction of SWTV group
Requirement properties Req. spec.
assert( φ )
6/14
Algorithms
Automated Testing Counter example(s)
SWTV group.
CS Dept. KAIST
OK or
Ex. Testing a Triangle Decision Program
Input : Read three integer values from the command line. The three values represent the length of the sides of a triangle.
Output : Tell whether the triangle is •
부등변삼각형
,
이등변삼각형
,
정삼각형
Create a Set of Test Cases for this program: (3,4,5), (2,2,1), (1,1,1) ?
• Condition 1: a > 0, b > 0, c > 0 • Condition 2: a < b + c – Ex. (4, 2, 1) is an invalid triangle – Permutation of the above condition :a < b +c, b < a + c, c < a + b • What if b + c exceeds 2
32 (i.e. overflow)?
– long v.s. int v.s. short v.s. char • Developers often fail to consider implicit preconditions – Cause of many hard-to-find bugs
Introduction of SWTV group SWTV group.
CS Dept. KAIST
• # of test cases
required?
①
4
②
10
③
50
④
100
• # of feasible unique
execution paths?
•
11 paths
•
guess what test cases needed “Software Testing a craftsman’s approach” 2 nd ed by P.C.Jorgensen (no check for positive inputs) Introduction of SWTV group 8/60 SWTV group.
CS Dept. KAIST
Research Trends toward Quality Systems
• • •
Academic research on embedded systems has reached stable stage
– just adding new functionalities to a target system is not academic contribution anymore considered as an
Research focus has moved to the quality of the systems from the mere functionalities of the systems
– Energy efficient design, ez-maintenance, dynamic re-configuration, etc
Software reliability is one of the highly pursued qualities
– ASPLOS 2011 Best paper • “S2E: a platform for in-vivo multi-path analysis for software systems” @ EPFL – OSDI 2008 Best paper • “Klee: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs” @ Stanford – NSDI 2007 Best paper • “Life, Death, and the Critical Transition: Finding Liveness Bugs in Systems Code” @ U.C. San Diego
Introduction of SWTV group
9/14
SWTV group.
CS Dept. KAIST
Collaborators of Our Group
Introduction of SWTV group
10/14
SWTV group.
CS Dept. KAIST
Research Directions @ SWTV
Sequential program testing Test cases Concurrent program testing
SWTV group Automated SW Quality Assurence
Thread sched. scenarios
2020-04-30
Fault localization Suspicious stmts. 뮤즈 (MUSE): 프로그램의 수많은 돌연변이들을 활용한버그 위치 추정 기법
SWTV group 11
Introduction of SWTV group
12
SWTV group.
CS Dept. KAIST
Concolic
테스팅 과제의 산업적 성취
2010년 2011년-2012년 2013년-2014년 • •
Phase1(2010) Concolic testing
타당성 조사
Pilot
과제 적용을 통한
concolic testing
타당성 조사 오픈소스 소프트웨어
(busybox)
및 소프트웨어
(SLP(Tizen S
사 전신
) file manager, Samsung security library)
에 적용하여 새로운 버그 발견 • • •
Phase2(2011-2012) Concolic testing
도구 개발
S
사 개발 코드를 테스트하기 위한 자동화된
concolic unit testing
도구
CONBOL
개발
4MLOC
규모 의 삼성 내장형 프로그램 에 적용하여 결함 발견 및 개발자 보고 후 수정 기존
concolic testing
도구를 확장하여
bit
수준 정확도
, BOF
오류 감지
,
실행 속도 개선 수행 • • •
Phase3(2013-2014) Concolic testing
도구 속도 향상 및 오탐 감소
CONBOL
개발 결함 검출 오탐 감소 를 위한
unit-testing
전략
, pre-condition
생성 기술
CONBOL
실행 속도 및 사용성 개선
: 64bit
지원
,
최신
SMT solver
지원 등 주 단위 적용을 통해 삼성 내장형 프로그램 및
open source project
의 실제
Introduction of SWTV group
2020-04-30 13
SWTV group.
CS Dept. KAIST
Concolic
테스팅 과제의 학문적 성취
2010년 2011년 2012년 2013년 2014년 • •
FSE 2011 held in Szeged, Hungary
논문 발표
FSE: SE
분야 학회
TOP 2 Introduction of SWTV group
• •
ICSE 2012 held in Zurich, Switzerland, ICST 2012 held in Montreal Canada
논문
2
건 발표
ICSE: SE
분야
TOP 1
학회 2020-04-30 • • •
ASE 2013 held in Palo Alto, CA, USA
논문 발표 삼성 논문상 동상 수상
ASE: SE
분야
TOP 3
학회 14
SWTV group.
CS Dept. KAIST
Recognition of Success of CONBOL at Samsung Electronics
•
Bronze Award at Samsung Best Paper Award
•
Oct’s Best Practice Award
•
Team leader Dr. Yoonkyu Jang received Samsung Award of Honor
15/23
Introduction of SWTV group
Ph.D Students
• • • •
Topic : Automated testing 2 top intl. journals such as IEEE TSE and FACJ 16 top intl. conf. Including ICSE/FSE/ASE Software system:
– Scalable COncolic testing for Relibaility (SCORE) • •
Topic: Concurrency testing J. of Systems and SW, JSTVR, ISSTA’12, ICST’13, ICST’14, etc.
• • •
Samsung Humantech Bronze award Qualcomm fellowship award Software system:
16 – COncurrent Bug dETection framework (COBET)
SWTV group.
CS Dept. KAIST
Recent Alumni
MS Students
• • • • • •
Topic : concurrent program testing and symbolic execution Best MS thesis award ICSE 2015 SEIP (top S rank), ICST ‘14 Best paper award at KCSE 2014 Best paper award at KIISE paper competition Qualcomm fellowship award
• • •
Topic : automated fault localization Attended ICSE 2014 @ India Preparing OOPSLA 2015 Introduction of SWTV group
•
Topic : concolic testing SWTV group.
CS Dept. KAIST
Questions? Comments?
Introduction of SWTV group
18/14
SWTV group.
CS Dept. KAIST