Measures of Effectiveness in PA

Maureen Taylor, Ph.D.

Gaylord Family Chair of Strategic Communication University of Oklahoma

Training Objectives

• To illustrate how primary and secondary research inform decision making and provide evidence of communication effectiveness;

PA has Evolved into an Applied Science

– Collect and analyze information on the changing knowledge, opinions, and behaviors of key publics – Serve as the central source of information about an organization and as the official channel of communication between the command and its publics – – Communicate significant information, opinions, and interpretations to keep an organization’s publics aware of command policies and actions Coordinate activities that affect command’s relationships with its publics and other groups.

RPIE

• RPIE is a strategic planning tool.

• • • • R = Research P = Planning I = Implement E = Evaluate I will cover the R and E

Foundation of Measuring

• Formative research provides information about the environment BEFORE a public affairs effort. It provides the baseline. • Summative research provides evidence of impact. It is conducted at the END of the PA effort. This data shows change.

• Change is what we want from our efforts.

Research

• What kinds of primary and secondary research have you conducted in your PA role?

Secondary Research

• Don’t reinvent the wheel.

• There is a lot of free data out there.

• Previously published research can be used to inform decision making and develop strategic messages.

Secondary Sources

• • • • • • Census Periodicals Journals Chamber of Commerce Reports Blogs Military Sources

Primary Research

• You may also be asked to collect primary research to know your publics or plan a campaign.

• Primary research is something that you collect and interpret to prove effectiveness.

• Primary research is collected using different methodologies.

Methodology

• Each method has strengths and limitations • Two broad categories of primary research – Qualitative (describing) – Quantitative (measuring)

The Value of Descriptive Research

• Qualitative research provides narratives and anecdotes that are persuasive – The school in Iraq – The family helped by an aid organization – The business that reopened because security improved – The newspaper that could restart printing because it is safer now to disagree with the government

Useful Qualitative Methods

• • • Interviews Focus groups Observations

Interviews and Focus Groups

• Valuable for understanding WHY people think, believe, and act in certain ways.

• They provide anecdotal evidence, narratives, stories, and the words of the people.

• News is narrative.

Using Interviews in Program Evaluation

• RQ: How have USAID political transition grants helped independent media outlets in the former Yugoslavia promote tolerance and democracy?

• Method: Interviews with over 140 media organizations in Bosnia, Croatia, Serbia and Kosovo. (over a 10 year period)

Findings

• Saturation of the Data – Most owners and editors reported that they were able to develop public affairs programming through grants to cover stories that would not normally make the nightly news.

– Allowed for the development of in-depth, pro social stories that did not need advertising support.

Focus Groups

• In Armenia, focus groups with media organizations identified gaps in training, news coverage, and network consolidation.

• Outcome: IREX reconsidered its model for Armenian media.

Observational Methods

• Watching people interact and/or use services.

• This method tells us HOW people are doing certain things.

Example of Observational Research

• Darfur Peace Process – How can we find out which channels of communication are most trusted?

I sat in the corner of a market cafe and watched Darfuris listen to radio (Government and BBC Arabic) (also looked at cell phone usage)

Lessons Learned

• Sometimes we have flawed assumptions about how people behave. • Observations can help us support or refute these assumptions.

• Bus stop ethnography is valuable in formative research.

Limitations of Interviews

• What are some limitations of interviewing for R and E?

• How can you avoid these limitations?

Reality Check

• Is one good story enough evidence for decision makers?

• Selection bias • Social desirability

Quantitative Research

• Numbers talk. • Surveys, content analysis, and experiments can provide these numbers.

Surveys

• Develop specific questions that allow you to measure awareness, attitudes, and behaviors.

• Data is only as good as the survey questions and the sampling method used in collection.

The Best Surveys…

• • • • • Specific goals Easy flow of questions Use interval level questions (likert, frequency) Short format Sampling methods ensure reliable data

The Worst Surveys…

• • • • • Fishing expeditions Questions jump from topic to topic Open ended questions Long and complicated Convenience sample

Collecting Data to Shape Policy

• Surveys of public opinion of media and NGOs after the Bosnian war – Randomly sampled 1200 Bosnians from each of the regions to ascertain which media they trusted when they needed information to make a decision – USAID changed its funding strategy to stop supporting low impact media and NGOs

Surveys in PA

• Learn about internal and external publics • Cost effective to “piggy back” on other surveys • Random sampling allows us to generalize from a few to many • Online surveys are valuable (but must be interpreted cautiously)

Online Surveys

• For minimal cost, you can upload surveys and invite people to participate.

• But, consider: – selection bias (strong opinions) – non random – software may not allow for skipped questions

Experiments

Is there a causal (or correlative) relationship between a strategic communication effort and an outcome?

• Example: Communication campaigns in Malaysia for improved ethnic relations.

Neighborliness Campaign in Malaysia • Quasi experimental design: Compared attitudes and behaviors of those who received the communication campaign to those who did not.

• Findings: People in the campaign neighborhoods were less likely to have positive attitudes about other ethnic groups and had less frequent positive interactions with members of other ethnic groups.

Content Analysis

• One way to measure the impact of our messages. This allows us to count the number of times our PA output has appeared in the media. • But, more importantly, it also allows us to examine the tone, prominence and placement of messages.

Break Out Session

• In the break out session I will take you through the steps to content analyze news for PA messages. • We will learn how to conduct a content analysis and then complete a practical exercise in content analysis.

Break Out Session Objectives

– To provide in depth background about content analysis; – To provide practical training in conducting content analysis; – To assist you in using content analysis to create and evaluate strategic messaging.

A Frequent PA Tool

• Take the media coverage of your organization and systematically analyze it for: • • • • Tone (positive, negative, neutral) Prominence (where is it in the paper, news) Accuracy (did they get it right? Why not?) Other factors

What Can Be Analyzed?

• Newspaper articles, radio stories, and television news stories (and programs) can be content analyzed to examine the occurrence of certain features.

• • Internet Web sites and chat rooms can also be studied. Darfur study uses chat rooms and blogs as proxy data.

Software Programs

• There are a variety of software programs that are used for content analysis. • If you use this software, then your job would be to interpret the findings. • Use the findings to strategize how to address any problems/opportunities.

Best Practices for Human Coding

• Most of you will not have software and will have to task subordinates with the actual coding.

• Your role is interpret the findings, advise the command, and develop strategic messaging.

Communication with Stakeholders

• PA communication is based on the assumption that we have an obligation to inform our stakeholders.

• We also have the right to try to influence their awareness, attitudes and behaviors.

• We must be open to being influenced as well .

Information Subsidies

• Organizations provide content to media organizations.

• Research Findings: – % of content that makes it into news – Direct quotes from news releases

Different Nations = Different Subsidies

• Information subsidies vary across the world. In some nations, media accept large portions of materials from public relations efforts.

• PA materials may be suspect.

Content Analysis as a Social Science Methodology • Social science applies many of the tools of the physical sciences to studying human beings. • Social science means that PA can be studied, measured, and improved using data rather than guess work.

Media Content Analysis

• Method to “count” results of PA activities • Usually, we look quantity vs. quality • Tactic/output based • Success is often defined in terms of placement of strategic messages in media.

Minimizes Subjective Analysis

• Berelson (1952) defined content analysis as “a research technique for the

objective

,

systematic

and

quantitative manifest

description of the content of communication” (p.18).

• It attempts to take the subjectivity out of the evaluation.

Value of Content Analysis

• It allows the researcher to “make inferences by objectively and systematically identifying specified characteristics of messages” (Holsti, 1969, p. 14).

• We can judge news based on previously specified content areas.

Use in Media Analysis

• Newspaper articles, radio stories, and television news stories (and programs) can be content analyzed to examine the occurrence of certain features.

• Internet Web sites and chat rooms can also be studied.

Proxy Research

• When public opinion is difficult to ascertain, we can use content analysis of news and editorials as a proxy of what people are thinking. • Darfur Example

Unit of Analysis

Different Types of Units of Analysis in Content Analysis Examples: Newspaper, News program, Story, Words, Headlines, Photos, Quotes (sources)

Building from simple to complex…

1.

2.

3.

4.

5.

Clip counting Circulation and readership analysis Advertising value equivalence Placement analysis Prominence analysis 6.

7.

Message analysis Tonality analysis 8.

9.

Overall quality of coverage Competitive analysis (“Share of discussion”) (this portion of the lecture is from Dr. Holroyd)

Media Content Analysis

1. Clip counting – “How often did my story appear?” 2. Circulation and readership analysis – Add to clip counting with data gathered from secondary sources – Looking for circulation, readership, demographics of audiences • Helps answer the “Who might have read/heard this?”

Media Content Analysis

3. Advertising value equivalence (AVE) – “If I had to pay for this space, how much would it cost?” •

“I paid $30,000 for this PA effort but got an AVE of $100,000.”

– – Allows for comparison vs. advertising budgets To calculate costs, consider day of publication, location in publication, article/story length, etc.

• Use reluctantly – Apples-oranges comparison since PA ≠ Ads

Media Content Analysis

(Begin assigning values)

4. Simple content analysis – Variety of characteristics to “define” article or story are assigned a value • Front page?

• • • Above the fold?

Picture? Color?

Half (more?) of article before the jump?

– Use computer programs to develop overall scores of “success”

Media Content Analysis

5. Message Analysis – “Were my key command messages in the article?” – More “points” if messages are earlier in the article • Must know what key messages you wanted to convey prior to review

Media Content Analysis

6. Tonality Analysis – Adds subjective assessment of content – “Was the report favorable or unfavorable to my project?” – Variety of ways to score: • • Positive, neutral or negative 0-100 with 50 neutral – Assess each article individually and then add together to get overall impression of impact

Media Content Analysis

7. Prominence Analysis – Considers 6 factors: 1. Publication where article appears 2. Date of appearance -- Sunday readership higher than Monday 3. Overall size of article (column inches) 4. Where in the publication it appears 5. Use of photography or not 6. Size of headlines – Each assigned a code and analyzed

8. Quality of Coverage – Considers a combination of factors such as tonality, prominence and message inclusion – Each factors in to create overall score for article

Media Content Analysis

9. Competitive Analysis – – – Compare what coverage the competition is getting Compare # clips, prominence, Way to assess relative performance in media • Sometimes called “Share of Discussion” – You had 8 great stories. Other side had 29 so-so ones. You lose.

– Not often done but VERY important • Looks at issue from audience’s view rather than what you think you’ve conveyed.

Walter Reed Hospital Crisis

• In 2007, the Washington Post broke a story about poor conditions for wounded soldiers at the Walter Reed Army Medical Center in Washington, DC. • Here is what content analysis can do for the PAO.

Media Tracking

Date Title Reporter, Placement Outlet

2/25 2/24 2/24 2/21 2/20 2/19 2/18 Admin Issues Cited at Walter Reed Steve Vogel, A9 Review at Walter Reed is Ordered Steve Vogel, A1 Army’s Preemptive News Briefing Howard Kurtz – Columnist, C1 Swift Action Promised at Walter Reed Dana Priest, Anne Hull, A8 Army Fixing Patients Housing Dana Priest, Anne Hull, A1 US Army Facility Reported in Poor Shape Robert Siegel (with Dana Priest, Anne Hull) Soldiers Face Neglect, Frustration at Army’s Top Medical Facility Dana Priest, Anne Hull A1 W Post 0 W Post W Post W Post W Post NPR W Post 0 -3 -3 -3 -3 -3

Tone Place.

8 10 6 8 10 6 10 3 3 3 3 3 3 3

Score

Prom.

Msg Anal Total (Notes)

10 8 6 0 0 8 6 21 21 12 16 16 10 A 10 B Green: 20+ Amber: 11-19 B. Letter-to-editor sent Red: 10 under Notes for RED scores: A. Reporter sent corrective memo

Challenges of Content Analysis • • Flaw #1: Not determining message accuracy – Accuracy is based on 4 factors: • • Basic facts included?

Misstatements (“reduce” score) • • Incomplete information Omissions Flaw #2: Not linking messages to objectives – Where is the story in the project’s “lifecycle?

• Awareness…then knowledge…then interest…then intent/behavior

What Can CA Measure?

• • • • Content analysis is a measure of level of effort, activity or output By watching media content change, we can show effectiveness in strategic messages Cannot determine impact on key audiences Focuses on quantity over quality Other methods needed to triangulate findings to make the claim of impact

End May Not Equal Objective • • Your commander says, “We’ve got a great story. Get it on the front page of the

Times

.” – (His request is a tactic) Your job is to ask, “Why the

Times,

sir?” • Your key audiences

may

be reading the

Times

, but research shows they are more likely to read/watch/listen to another outlet. – If you target that 2 nd outlet, you’re more likely to achieve your objective than just accomplish a tactic

Learn More about CA or Start Exercise

• We can either move on to the exercise or continue learning about examples of content analysis.

• Skip to slide 72

Content Analysis as M&E

Content Analysis is also used for

Monitoring and Evaluation

M&E is a way for organizations to continually refine and adjust what they are doing

M&E

• USAID usually requires a M&E component to every contract and grant • M&E proves that the organization is actually accomplishing what it sets out to do.

Can Training Journalists Help to Improve Content

Kosovo has a new legal system. Journalists were getting all of the legal terminology and court levels wrong in their stories. Their stories were undermining faith in the new system.

IREX conducted 5 Saturdays of legal system training for reporters to help explain the system and introduce them to key people in the system.

Measuring Change

• Benchmark of news coverage of legal stories four weeks BEFORE the training.

• Time 1 coverage of news coverage of legal stories four weeks DURING the training.

• Time 2 news coverage of legal stories four weeks AFTER the training.

Human Trafficking

• Benchmark of news coverage of trafficking stories four weeks BEFORE the training.

• Time 1 coverage of news coverage of trafficking stories four weeks DURING the training.

• Time 2 news coverage of trafficking stories four weeks AFTER the training.

Findings

What Went Wrong?

• This analysis can help us to see the limitations of the training.

• The print journalists

did not

incorporate many aspects of the training into their stories. Next time, the training needs to be modified and the editors need to be included in the training.

Inflammatory Language in Iraqi Media • We are coding four stations (Al Iraqiya, Al Hurra, Al Summariya, and Al Bagdadiya) for levels and frequency of inflammatory terms.

• We found that it is the anchor/presenter who most often uses the terms. We also found that they often use sound bites from politicians that have inflammatory terms.

So What?

• In Iraq, 230 journalists have been killed and the Committee to Protect Journalists notes that violence is directed at reporters who use or work at stations that use inflammatory terms in their coverage. • We are bringing anchors and news directors to Beirut in Oct. to train them in content analysis .

Community Radio Stations Liberia

• We are using content analysis to help identify issues before they become conflicts. • We are having the stations log calls and topics to scan for issues before violence occurs. • Dialogue based programming may be able to diffuse rumors and mediate tensions.

How Can PAOs Use Content Analysis?

• It can be used to evaluate output and then create new strategic content • Components of Excellent News

Practical Exercise 1

Please spend a few minutes identifying all of the components of a good news story.

What are the components of a good print story?

What are the components of a good television story?

What are the components of a good radio story?

Developing Categories

• Deductive Category Development – Take the individual generated components and identify/narrow down categories • Inductive Category Development – Identify categories ahead of time and apply them to content

Operationalization

• Defining the category in such detail as to make it clear for any reader/coder • • Examples: Images of Political Candidates How often standing, who standing with, full body, face

Practical Exercise 2

• As a group, we need to define each content category component. • We need to define a category in as much detail as possible so that anyone who reads our operationalization can see the same things we see. Ensures objectivity.

Intercoder Reliability

Scott’s Pi

is calculated based on percentage of agreement between 2 or more coders • Categories need to be

mutually exclusive exhaustive

and • The unit of analysis can only be placed in

one category. Definitions need to be complete.

Types of texts that can be content analyzed

• • • • • •

Newspapers

Magazines Books PSAs News releases Annual reports • • • • Memos Chat rooms Web pages

Transcripts of:

–

Radio news

– –

Nightly news (or listen to)

Practical Exercise 3

Rating Scale Now that we know how to identify each news component, we need to evaluate the strength of its presence.

What does each number mean to you?

Let’s make 5 the number representing highest/best incorporation of the feature. Let’s make 1 the number representing the weakest showing/missing from the story.

Application Practical Application of Content Analysis to News Refining the Categories

Practical Exercise 4

• Take out Practical Exercise 4 and write in the content categories and the group’s definitions. • Please code the Cuba story based on our content analysis criteria.

Let’s See How Close We Are

• Establishing an informal intercoder reliability.

• Each participant will share their coding and rationale.

Revisiting the Categories

• How well do the operationalizations work in representing excellent news?

• How well does the rating system work?

• You may need to do this several times with the coders. Every new coder added to the team must be trained.

Using Content Analysis to Generate News Stories • Although most often viewed as a post news evaluation methodology, content analysis can also be used to help generate news stories.

• This is a great training tool for developing JR PAOs and support staff.

Check List

• After we know the components of an effective news story, we can use the checklist to ensure that all components are in our strategic messaging.

Training Opportunities

• We can also content analyze the written materials of our subordinates and identify areas for improvement. • The checklist allows supervisors to pinpoint areas for improvement.

• Thank you.

Questions?