IS 4800 Empirical Research Methods for Information Science Class Notes January 11, 2012

Instructor: Prof. Carole Hafner, 446 WVH [email protected] Tel: 617-373-5116 Course Web site: www.ccs.neu.edu/course/is4800sp12/

I.

Topics for Today

Why does this course exist ?

II.

Evidence-based research in IT III.

Scientific method IV.

What is an empirical research method ?

V.

VI.

Purposes and Products of empirical research Examples, Assignment 1

I. Why does this course exist?

■ Background facts ■ Empirical research means gaining new knowledge through systematic observation and analysis of data ■ Empirical methods usually taught only to graduate student ■ Usually taught only in behavioral/social sciences ■ In the past, little use of empirical research in CS (notable exception is HCI/UI design). This is slowly changing.

Why does this course exist?

■ The need for an empirical approach to research ■ IS is concerned with users and clients as well as the computer and the software ■ Research involving people requires empirical studies, since logical proofs are not possible ■ Many decisions today are based on “anecdotal evidence” or opinions Example: proponents of software design and development methods (functional programming, object-oriented programming, formal methods, design patterns, eXtreme Programming) claim their method improves productivity and/or quality based on their opinion or anecdotal evidence.

Why does this course exist ?

■ Why teach empirical research to undergraduates who are usually not aiming for careers as researchers?

1.

2.

3.

For careers in system admin/IT management For careers in system design/development For careers in IT planning/policy (“governance”) 4.

All educated people should be able to analyze and evaluate claims based on empirical evidence.

Examples: sexting, e-readers

■ Articles in WSJ August 25 2010 ■ “it’s difficult to pinpoint how common the practice (sexting) is. Various surveys have estimated . . . from roughly 4% to 25%.” ■ “ a study of 1,200 e-reader owners . . found that 40% said they now read more than they did with print books.” ■ Companies with IT product lines, legislators, etc. need their questions answered with evidence, not just opinions. Empirical research skills will be increasingly valuable.

II. Evidence-based research in IT: the MIS researcher’s perspective

■ Research on (organizational) impact ■ What is the impact of computerized systems on productivity, group communication, power relationships and organizational practices ?

■ Does the system make the organization more effective in carrying out its mission ?

■ Do the economic benefits of computerized systems exceed their costs?

■ Research on IT management practices ■ What approaches work best for IT governance ?

■ How can risks be identified and minimized ?

Evidence-based research in IT: the computing researcher’s perspective

■ Research on system lifecycle (requirements analysis, design, development) ■ How can system designers translate people’s needs and preferences into requirements ?

■ What are the advantages and disadvantages of proposed software/system development methods ?

Both MIS and computing perspective ■ What problems should be expected when systems are deployed in their target environment ■ What are the best approaches (technical and managerial) to protect the security of systems

Evidence-based research in IT: How the Internet fits in

■ We can study who uses the Internet for what purposes ■ We can study how the Internet affects us as individuals and as a society ■ We can use the Internet as a tool in our research (e.g., for surveys or interviewing) ■ We can study how Internet-based research is different from other types (e.g., different ethical issues arise).

III. The Scientific Method

 Methods of inquiry  Method of authority  Method of logic (deductive reasoning from known facts)  Scientific Method (inductive reasoning) “The Name of the Rose” by Umberto Eco: best selling novel about the transition from authority to reason in 1300’s. (movie starring Sean Connery)

Scientific Explanation

      Empirical Rational Testable Tentative Parsimonious Rigorously evaluated (vs. “anecdotal”)

Steps of the Scientific Method

 The scientific method consists of four steps  Noticing a phenomenon  Formulating testable explanations (hypotheses)  Observing and/or experimenting  Refining and retesting explanations  The scientific method can be tedious and time consuming  But it is the most widely accepted methodology for acquiring knowledge about the world.



Noticing a Phenomenon

 After noticing a phenomenon, you identify VARIABLES or THEMES that appear important in explaining what happens 

Formulating Tentative Explanations

 Initial observations allow you to develop a HYPOTHESIS, or tentative statement, about the relationships among the variables identified



Observing and Experimenting

 You DESIGN and carry out systematic OBSERVATIONS of the behavior of interest,  These observations are directed at testing your hypothesis 

Refining and Retesting Explanations

 Supported hypotheses are often REFINED and subjected to further exploration  Disconfirmed hypotheses may be reworked and RETESTED

The importance of skepticism

   Ask whether the observed data can be explained another way?

Seek data that disconfirms a theory View all theories as tentative (accepted until disproved) How do I know whether this is true??

How could it be proved false??

Challenges for Scientific Method in Behavioral/Social Research

■ Finding hypotheses that are both testable and that answer your question ■ The feasibility of collecting data to test your hypothesis ■ The ethics of collecting data to test your hypothesis Ex: social impact of parental neglect: experiments carried out on chimpanzees

IV. What is a “research method”?

A strategy of inquiry, which moves from underlying philosophical assumptions to a research design Philosophical assumptions Research strategy/method Research design Philosophical schools regarding research (“epistemology”): positivist – research should seek objective truth interpretive – research should take human experience into account, truth can be subjective critical – research should be aimed at making the world better

Attributes of research methods

•What kinds of questions can be answered?

•What sources of data are utilized?

•How is the data generated/collected?

•How is the data analyzed?

•How do the results of analysis result in answers?

Categories of Empirical Data •Artifacts •Laboratory experiments •Natural observations •Questionnaires •Interviews

Overview of the research methods

■ Quantitative ■ correlational studies ■ surveys ■ experiments ■ quasi-experiments ■ Qualitative ■ ethnography ■ case studies ■ grounded theory ■ Mixed

Classification schemes for research methods

1.

Epistemological – what kind of claims can be made ■ Descriptive/Interpretive ■ Correlational/Relational ■ Causal 2.

Methodological: how we go about it ( process ) 3.

Level of generality ■ Scientific research aims to discover general principles or test a general theory ■ Evaluation research aims to support decision making or test a hypothesis in a specific context

Deploying Empirical Research Methods

Observation Idea Prior Knowledge Research hypothesis or question Research Method What to observe and appropriate measures Research Design Subject Population/ Sampling technique How the data will be analyzed Step-by-step research plan

Steps of Empirical Research

1.

2.

3.

4.

5.

6.

7.

Select a topic or phenomenon to study Perform background investigation/literature review Identify specific research questions or hypotheses Select a research method Develop a detailed research plan/proposal a.

b.

For identifying or recruiting subjects For collecting data Define variables or phenomena to be observed and a coding scheme c.

Determine how the data will be obtained For analyzing data Execute the plan Document the research you performed (ongoing)

V. Purposes and Products

■ Reasons for doing empirical research ■ Disconfirm or support a theory ■ Solve a design or technical problem ■ Evaluate a product or solution ■ Inform practice (develop “best practices”) ■ Understand why people do what they do ■ Predict the future ■ Contribute to a better world ■ Advance personal goals

V. Purposes and Products

■ Products of Empirical Research ■ ■ ■ Evidence of a relationship Disconfirm a theory (or some aspect of it) Comparison of existing products or techniques ■ ■ A new or improved product or technique In-depth study of a particular outcome ■ Explanation of unanticipated outcomes ■ A new theory ■ A new improved research tool or technique ■ A critical analysis (pros and cons)