Research Methods II

Method Design

"When you change how you see the world, your whole world changes.” Aman Motwane

Game Plan

     Designing your research  Quantitative  Qualitative Collecting Data  Primary and Secondary Data Sampling Measurement Qualitative Data Collection  Interviews   Surveys Observation

Method design in the thesis process

Formulate and clarify your research question Chapter 1 Review the Literature/Theory Chapter 2 Formulate your research design Chapter 3 Sampling Answering research question Measurement Analyze your data Chapter 4 Qualitative Methods Quantitative Methods Write your Results and Discussion Chapter 5

The methods process

    Research Design   How are research questions answered?

Ex: A mixed method study investigating the relationship between cultural acuity and practitioner effectiveness Data Collection / Sampling / Setting/   Who did you study and how did you select them?

Ex: Population is all employees of Intel – sample is Finance Department Measurement   How are numbers assigned to concepts? Ex: Quality of Work Life – measured by the QOL index Data Analysis  How are data manipulated and interpreted?

 Ex: Regression Analysis

Alternative Research Designs

Creswell (2003) Quantitative research

Experimental designs Non-experimental designs, such as surveys Predetermined Instrument based questions Performance data, attitude data, observational data and census data Statistical analysis

Qualitative research

Study is exploratory Case studies Ethnographies Emerging methods Open-ended questions Interview data Observation data Document data Audiovisual data Text and image analysis

Mixed methods research

Elaborate and expand findings of one method with another Concurrent – converge quantitative and qualitative data Both predetermined and emerging methods Both open-ended and closed-ended questions Multiple forms of data drawing on all possibilities Statistical and text analysis

Your design will drive your data collection

The Research Onion

Time horizons Techniques and procedures Choices Strategies Approaches Philosophies Data collection and data analysis

Positivism

Realism Interpretivism

Cross Experiment Survey Deductive Case sectional Mono-method study Action Mixed research methods Multi-method

Grounded theory

Longitudinal

Ethnography Objectivism Subjectivism Pragmatism

Archival research

Inductive Functionalist Interpretive

Constructivism

Radical humanist

Positivism versus Constructionism

Constructionism Positivism

    Knowledge is subjective It’s the product of the context in which its constructed (ex. Copernicus) Scientific knowledge is not a “direct representation of the natural world” It’s based on systems of thought that are culturally and historically bound       All knowledge is derived from experience whether of the mind or of the senses. Evidence by what we can see, hear, touch, taste, or smell Is the basis of “The Scientific Method” Emphasizes the importance of testability, reliability, validity, and repeatability Asserts that the researcher is “bias-free” Claims “objectivity”

Your theoretical perspective leads to your research design

Qualitative vs Quantitative Design

      

Quantitative

Tests or verifies theories or explanations Identifies variables to study Relates variables Observes and measures information numerically Uses unbiased approaches Uses statistical procedures Largely deductive reasoning       

Qualitative

Takes place in the natural setting Uses multiple methods that are interactive and humanistic Is emergent Fundamentally interpretive Researcher views social phenomena holistically Researcher reflects on who he or she is in the inquiry and is sensitive to how that shapes the study Largely inductive reasoning

Collecting Primary Data

   Survey Research  Interviews Observation Methods Experimental Research

Collecting Secondary Data

  Consider who would have collected data  Governments, NGOs, Association or interest group etc Good Sources        Statistical Universe Statistical Abstracts Historical Statistics WDI – World Development Indicators IMF – International Financial Statistics United Nations – UN Stat Economic Report of the President Historical tables Secondary data: data that has been collected for some other purpose

Sampling: who AND how many to ask when you can’t study the entire population

    You are conducting a study of professional success factors among CC alumni who are external consultants. What is your best approach?

Two key questions:  Is the sample biased ? Is the size adequate?

Sampling is often a major limitation  Failure to identify the population, sample of convenience, volunteerism, Sample size – bigger is not always better – “it all depends…”

Sampling Techniques

Sampling Probability Simple Random Stratified Random Systematic Cluster Quota Non-Probability Snowball Convenience Purposive Self-Selection

Your goal is to produce valid and credible sample data and statistics that match the precision needed for the study

Probabilistic Sampling

     Simple Random: every member of the population has an equal probability of being sampled (use random # tables) Systematic Sampling: select at intervals (e.g,. Starting at 18, every 50 th listing is selected) Stratified Sampling: grouping the study population into strata and selecting a random sample within each stratum (e.g., dividing the population by education then selecting a random sample) – less than high school, high school, no college diploma, college diploma, Cluster: random selection of groupings from which all members are chosen for the sample (e.g., geographic groupings, intact groups) Multi-stage: select a cluster, then sampling members of the selected clusters

     Quota: ensures that certain characteristics of a population sample will be represented (e.g., need 20 small business owners, 10 minority, etc.) Purposive: (Judgment) experienced individual selects sample based on judgment about particular characteristics (e.g,. Political polls) Snowball: initial respondents are asked who else would be appropriate for the study Self-Selection: participants self-select into the study Convenience: selecting people (units) who are most conveniently available (e.g., college professor using students)

How many interviews, surveys?

 It all depends   Research Question Variance of the population  Survey   Minimum of 30 for most statistical tests More is better – to a point!

 Interviews  Ask until you start to get repetition

Remember – the intent in qualitative research is not to generalize, but to elucidate the particular, the specific

Measurement

: Variable definition

Variable Conceptual Definition (include source) Operational Definition required for measurement (include source where appropriate)

DV: IV: IV: IV:

Measurement: Research questions and variables

  Q1: What is the relationship between faculty sleep deprivation and clarity of faculty communication in a thesis workshop?

 Dependent variable => clarity of communication  Independent variable => sleep deprivation Q2: What is the relationship between clarity of faculty communication and student learning during a thesis workshop?

 Dependent variable => student learning  Independent variable => clarity of communication

Faculty sleep deprivation, communication clarity and student learning

Variable Conceptual Definition (include source) Operational Definition required for measurement (include source where appropriate)

Content analysis of lecture Clarity of faculty communication Sleep deprivation Faculty stays on topic, uses appropriate examples, invites and answers questions – what, how, who, when and where Number of nights over a one week period where sleep falls below 5 hours a night Faculty knowledge of subject matter (moderating variable) Demonstrated level of skill, ability and experience with subject matter # of hours of sleep per night # years experience, Education Student learning Degree to which students demonstrate the ability to apply concepts from the workshop to their project Application of concepts in projects (content analysis)

Measuring Variables: Scale construction

    Scale – any measuring instrument (questionnaire, interview, test) composed of one or more items (

questions, observations, etc…)

that have logical relationships with one another. Standardized measures – don’t reinvent the wheel. Source for standardized measures –  Journal articles, books, other thesis or dissertation projects If you have to create your own  KISS Measure twice and cut once… the value of beta testing your approach

Scale Example: Job Characteristics

Hackman and Oldham, 1975  Measures the degree to which a job is designed to be motivating (the motivating potential of a job)  Five dimensions: skill variety, task identity, task significance, autonomy and feedback      My job requires me to use a variety of skills.

My job allows me to see the finished products of my work.

Doing my job well affects others in some important way.

My job is designed so that I know when I have performed well.

My job allows me freedom to work with minimum supervision.

Scale Examples

  Participation/Involvement Scale     I feel free to suggest new and better ways of doing things.

I am asked how we can improve the way my work group operates. Sufficient effort is made to get the opinions and ideas of people in this work unit.

Suggestions made by unit personnel are implemented in our daily work activities.

Satisfaction     In general, I am satisfied with my job.

I have a sense of personal fulfillment at the end of the day.

I am a valued member of my unit.

I would recommend an assignment in my unit to a friend.

Validity and Reliability

  Validity – does it measure what it is supposed to measure?

 Many types of validity that you need to consider Reliability – is it consistent?

 Increase reliability by:     Sample size Multiple items Multiple raters Multiple measures

Qualitative Data Collection

     Interviews Surveys Documents Audiovisual materials Observations

Surveys / Interviews

    Interviews Telephone Interviews Group Interviews/ Focus Groups Self-Administered Questionnaires Mail, Fax, Internet  Survey Monkey

Interview Question Design: Good Questions

       Sound conversational Use words participants would use Are easy to say Are clear Are short Are open-ended Are one-dimensional

Question Design: Probing Questions

     How do you mean that?

Would you give me an example of what you mean?

Tell me more about that… I don’t understand Anything else?

Example of a Questioning Route

     Opening Questions Introductory Questions Transition Questions Key Questions Ending Questions

Other questioning techniques

    Listing things Rating things Picture sort Drawing a picture

Questioning Pitfalls

     Avoid asking “why”?  Can put respondents on the defensive Be cautious about giving examples Double-barreled questions Abbreviations / Jargon / Slang Double-negatives

Interview template

•Discusses setting •Transcribes interview •Adds reflection notes where appropriate

Designing Questionnaires/Surveys

   Survey Monkey Zoomerang InstantSurvey

CC has a Survey Monkey account. See Library for account name and password.

 Major Decisions in Questionnaire Design  What should be asked?

    How should each question be phrased?

In what sequence should the questions be arranged?

What questionnaire layout will best serve the research objectives?

How should the questionnaire be pre tested?

A survey is only as good as the questions it asks

Phrasing Questions

     Simple-Dichotomy Determinant Choice Frequency determination Attitude rating scale Checklist

Observational Studies

 What can be observed?

     Human behavior and action Verbal behavior Expressive behavior Spatial relations Temporal patterns  Physical objects

Observational Studies

 Content of Field Notes         Space Actor Act Activity Objects Time Goal Feeling

Observation Field notes

Observation Exercise

   Partner with someone in class Decide on what you could observe to answer a question about the CC experience Go to the place for 5 minutes  Observe and take notes For purposes of this exercise, please do not disturb other classes 2 people can go into room 117 or room 128if you would like to observe a classroom

Observation Follow-up

  General comments   Most challenging? awkward? Easy to record?

Impact of observer  How could you manage that?

Documents

   E-mails, Letters, Memos, reports and minutes of committees Annual reports Could ask participants to keep journals (case studies and narrative research)

Administration

  No class tomorrow – Aju and I will be available to meet with you (Julie after 10:30) Research Question presentation Monday. Please bring your presentation on a memory stick.