International Conference on Program Comprehension (ICPC) 2008

A Traceability Technique for Specifications

Aharon Abadi, Mordechai Nisenson and Yahalomit Simionovici

ICPC 2008

Outline



Motivation



Goals



Our Solution: Outline of Traceability Link Process



IR Techniques



Experiments



Conclusions



Future work 2

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Traceability



The ability to link between different artifacts

– Example artifacts: code, user manuals, design documentation, development wikis, etc.



In particular, link code to:

– Relevant requirements – Sections in design documents – Test-cases – Other structured and free-text artifacts 

Also, link from requirements, design documents, etc. to code 3

A Comparison of Traceability Techniques for Specifications

ICPC 2008

What’s Traceability Good For?



Program Comprehension

– Top-down – Bottom-up • Particularly relevant for the maintenance of legacy systems 

Impact analysis

– Keeping non-code artifacts up-to-date 

Requirement Tracing

– Discover what code needs to change to handle a new req.

– Aid in determining whether a specification is completely implemented and covered by tests

4

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Challenges



Scalability

– Large # of artifacts 

Heterogeneity

– Large # of different document formats and programming languages 

Noisy

– Free text information (natural language): conjuctions, prepositions, abbreviations, etc.

– Some information may be outdated, or just plain wrong 

Prior work:

– Recovering Traceability Links in Software Artifact Management Systems using information retrieval methods [Lucia et al., 2007] – Recovering Traceability Links between Code and Documentation [Antoniol et al., 2002, Deerwester et al., 1990, Marcus and Maletic, 2003] A Comparison of Traceability Techniques for Specifications

5

ICPC 2008

Outline



Motivation



Goals



Our Solution: Outline of Traceability Link Process



IR Techniques



Experiments



Conclusions



Future work 6

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Example

/* * The File interface provides … */

public class private

FileImpl

extends

String nativefileName; FilePOA{ /** * Creates a new File … */ }

public

… FileImpl(String nativePath ...){ /** * … */

} Private

String f(..){ …}

7

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Goals



Examine the effectiveness of IR techniques for traceability between code and documentation on “real world” data



Most prior work compared 2 specific algorithms, LSI and VSM

– Is LSI really better?

– How does LSI stack up with other dimensionality reduction techniques?

– How does it compare with other non-dimensionality reduction techniques?



How do different levels of abstraction affect the choice of the best methods?

– How to fit a method and parameters to a dataset?

8

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Outline



Motivation



Goals



Our Solution: Outline of Traceability Link Process



IR Techniques



Experiments



Conclusions



Future work 9

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Traceability Link Process

Query Construction partial code words extraction Words expansion Words ranking (word 1 ,rank 1 ), …,(word m, rank m ) Off line processes documents Document Pre-processing IR-Index Text Preprocessing (

word

1 ,rank 1 ), …,(

word

m, rank m ) sections Sectoring sections Text Preprocessing sections sections

10

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Text Preprocessing

… Copyright owners grant member companies of the OMG permission to make a limited … Text Preprocessing … copyright owner grant member compani omg permiss make limit … • Lower-case , stop-words, number etc. • Stemming

11

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Words Extraction

/* * The File interface provides … */

public class

FileImpl

extends private

String nativefileName; /** * Creates a new File … */ }

public

… /** * … */

} Private

String f(..){ …} words extraction

12

• Class Name • Public Function names • Public function arguments and return type • Comments • Super class name A Comparison of Traceability Techniques for Specifications

ICPC 2008

Words Expansion

…NativePath, fileName, delete_all_elements… Words expansion … NativePath,Native,Path, fileName, File,Name, delete_all_elements, Delete,all,elements … • Use well-known coding standards for sub-words separation

13

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Outline



Motivation



Goals



Our Solution: Outline of Traceability Link Process



IR Techniques



Experiments



Conclusions



Future work 14

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Information Retrieval (IR) Methods



Vector Space Model (VSM) [Salton et al., 1975] implemented by Lucene

– Each document, d , is represented by a vector of ranks of the terms in the vocabulary: v d = [ r d ( w 1 ), r d ( w 2 ), …, r d ( w | V | )] – The query is similarly represented by a vector – The similarity between the query and document is the cosine of the angle between their respective vectors 

Jensen Shannon Similarity Model [Abadi et al., 2008]

– Each document, d , is represented by its empirical probability distribution over words: p d ( w ) – The query is similarly represented – The similarity score is calculated as 1 – JS ( p q , p d ), where JS is the Jensen Shannon Divergence

15

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Dimensionality Reduction Methods



LSI [Deerwester et al., 1990]

– Commonly used in prior studies – An algebraic method – Dimensions represent orthogonal topics 

PLSI [Hofmann, 1999]

– Probabilistic extension to LSI – Based on the assumption that documents are mixtures of topics distributions – Words and documents are conditionally independent given the topic 

SDR [Globerson and Tishby, 2003]

– Based on information theory – Topics are sufficient statistics in information theory terms – These statistics are functions that capture maximum mutual information between words and documents A Comparison of Traceability Techniques for Specifications

16

ICPC 2008

Outline



Motivation



Goals



Our Solution: Outline of Traceability Link Process



IR Techniques



Experiments



Conclusions



Future work 17

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Datasets



Software Communication Architecture (SCA) is an open architecture framework that defines how software and hardware elements operate within a software defined radio.



Common Object Request Broker Architecture (CORBA) is OMG's open, vendor-independent architecture and infrastructure that computer applications use to work together over networks.

Documentation details:

Dataset Size (MB) Sections Vocabulary size SCA CORBA 0.41

1.79

1311 3340 4827 7161 18

Queries details:

Dataset SCA CORBA 7 4 # classes # relevant results / query 6 – 13 5 – 20

A Comparison of Traceability Techniques for Specifications

Total # of relevant results 65 58

ICPC 2008

IR Quality Measures



Precision @ n:

P

(

n

) 

relevant



retrieved n



Recall @ n:

R

(

n

) 

relevant



retrieved relevant



Average precision:

AP



n N

  1

P

(

n

) 

rel

(

n

)

relevant

19

A Comparison of Traceability Techniques for Specifications

ICPC 2008

MAP versus Method

20

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Mean Average Precision (MAP) versus Dimension

21

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Precision versus Recall

22

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Dimensionality of Datasets SCA PLSI Results CORBA

23

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Precision versus Recall over Algorithms for SCA

24

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Precision versus Recall over Algorithms for CORBA

25

A Comparison of Traceability Techniques for Specifications

ICPC 2008

MAP versus Method – Combined over SCA & CORBA

26

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Outline



Motivation



Our Solution: Outline of Traceability Link Process



Similarity measures



IR Techniques



IR Quality Measures



Experiments



Conclusions



Future work 27

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Conclusions



Our Most significant results are:

– Traceability between code and documentation in real world systems is effective via IR techniques.

– For realistic datasets the Vector Space Model and Jensen Shannon model, which did not perform dimensionality reduction where shown to be the most effective.

– SDR was shown to be the best dimensionality reduction model, specifically it is better then LSI.

– As the documentation links are more abstract, the performance of VSM, JS model and SDR become equivalent.



Additional results:

– SDR was shown to be robust to datasets abstractness level – LSI and PLSI are sensitive to datasets abstractness level – We believe that PLSI poor performance is due to the difficulty of modeling very short documents, which could result in severe overfitting A Comparison of Traceability Techniques for Specifications

28

ICPC 2008

Outline



Motivation



Our Solution: Outline of Traceability Link Process



Similarity measures



IR Techniques



IR Quality Measures



Experiments



Conclusions



Future work 29

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Future work



Development of new measures for evaluation of different IR algorithms and datasets, specifically for traceability

– Example: developing a measure of “abstractness” for a specification which will help with tuning of parameters such as dimensionality 

Using dimensionality reduction techniques for creating thesaurus from the indexed data and using it for adding synonyms to the query



Traceability for other types of documents and links



Investigate alternative methods for query construction 30

A Comparison of Traceability Techniques for Specifications

ICPC 2008

References

  

A.D. Lucia, F.Fasano, R. Oliveto, and G. Tortora. Recovering Traceability Links in

Software Artifact Management Systems using Information Retrieval Methods.

ACM Trans. Softw. Eng. Methodol., 16(4):13, 2007.



G. Antoniol, G. Canfora, G. Casazza, A.D. Lucia, and E. Merlo. Recovering Traceability Links Between Code and Documentation. IEEE Trans. Softw. Eng. , 28(10):970-983, 2002.



S.C. Deerwester, S.T. Dumais, T.K. Landauer, G.W. Furnas, and R.A. Harshman. Indexing by Latent Semantic Analysis. Journal of the American Society of Information Science, 41(6):391-407, 1990.



A. Marcus and J. I. Maletic. Recovering Documentation to Source Code Traceability Links using Latent Semantic Indexing. In ICSE the 25 th ’03: Proceedings of International Conference on Software Engineering , 125-135, 2003.



G.Salton, A. Wong, and C.S. Yang. A Vector Space Model for Automatic Indexing. Commun. ACM, 18(11):613-620, 1975.

T.Hofmann, Probabilistic Latent Semantic Indexing. In SIGIR, 50-57, 1999.

A. Globerson and N. Tishby. Sufficient Dimensionality Reduction. Journal of Machine Learning Research, 3:1307-1331, 2003.

31

A Comparison of Traceability Techniques for Specifications

ICPC 2008

Thank You!

32

A Comparison of Traceability Techniques for Specifications