From impact factor to influence?

Kamran Abbasi Acting editor, BMJ

What I will talk about

 What is impact factor?

 What is wrong with impact factor?

 What are the alternatives?

 What is influence?

 What does all this mean?

Citation rates

 The Institute of Scientific Information records scientific citations/references  The number of times a publication has been cited within a certain period  Published as the Science Citation Index

Impact factor

 This is the mean citation rate of all articles contained in the journal  Published annually in the SCI Journal Citation reports

Definition of impact factor

   The recorded number of citations within a certain year (eg 2003) to the items published in the journal during the preceding two years (eg 2001 and 2002) Denominator: An “item” is a research paper or a review article Numerator: Citations to all types of articles

Calculating the BMJ’s impact factor for 2003

Proper and improper uses of impact factor  Evaluating individual scientists  Awarding higher academic positions  Evaluating research groups  Evaluating institutions  Resource allocation  Evaluating journals

Some examples

   In Japan people walk around boasting of their own individual impact factor In the UK (and elsewhere) university funding is awarded on the basis of impact factors (research assessment exercise) Deans instruct authors to publish in highest impact factors, though may not be the best fit

What I will talk about

 What is impact factor?

 What is wrong with impact factor?

 What are the alternatives?

 What is influence?

 What does all this mean?

1 What is wrong with impact factors?

     IF not statistically representative of of individual journal articles IF correlates poorly with actual citations of individual articles Authors not only use IF when submitting to journals The ISI database is imperfect and contains citations to “non-citable” articles Per Seglen BMJ 1997;314:497

2 What is wrong with impact factors?

 Self citations are allowed  Review articles are heavily cited and inflate the impact factor  Long articles collect many citations  Short publication lag allows short term journal self citations  Same language citations are preferred by authors

3 What is wrong with impact factors?

      Selective journal self citation Coverage of the database is not complete (3,200 out of 126,000 journals) Database has an English language bias Database is dominated by American publications Journals in database may vary from year to year IF is a function of the number of articles in the research field

4 What is wrong with impact factors?

     Research fields with literature that rapidly becomes obsolete are favoured Impact factor depends on expansion or contraction of research field Small research fields lack journals with high impact Relations between fields (eg clinical v basic science) strongly determine IF Citation rate of article determines journal impact, but not vice versa

What I will talk about

 What is impact factor?

 What is wrong with impact factor?

 What are the alternatives?

 What is influence?

 What does all this mean?

“The combined shortfall of university funding has forced deans of medical schools to behave like managers of Premier Division soccer clubs, recruiting potential stars . . .at the expense of teaching and clinical practice” Banatvala et al Lancet 2005;365:458-9

1 What are the alternatives?

    Will papers from high impact factor journals be cited in evidence based guidance?

Yes there is a correlation but papers from journals with low impact factor are also cited frequently Nakayama JAMA 2004;290:755-6 Might citations in evidence based guidelines be a better clinical measure?

2 What are the alternatives?

 A patient citation index  Proposed by Martin Rosser, editor of JNNP, and by Mary Baker and Matthew Menken from BMJ’s patient advisory board  A fuzzy idea but the gist that patients would rate articles of importance to them and hence of greater clinical importance

3 What are the alternatives?

 The Leiden University system  This is a method of measuring academic achievement through monitoring mentions of an article in the media, in parliament, and other such markers of “influence”  It requires a lot of work but may be fairer

4 What are the alternatives—from rapid responses on bmj.com

        User rating of articles Use a longer term base than 2 years Exclude letters and reviews to focus on research “Scope adjusted impact factor” Expert evaluation of best papers Many from Steve Harnad around citations, downloads, and google type page ranks Faculty of 1000 assessment www.f1000.com

Ask clinicians

What I will talk about

 What is impact factor?

 What is wrong with impact factor?

 What are the alternatives?

 What is influence?

 What does all this mean?

Why do we care about influence?

 The mission of the BMJ Publishing Group has two parts  INFLUENCE: To serve the needs of doctors and others, to influence the international debate on health  PROFIT: to make enough money to support the mission of influence

Why do we care about influence?

 Profit is easily measured—down to the last penny  We are not quite sure what influence is, which makes it hard to measure  Yet influence is the first part of our mission and profit the second  We mustn’t allow the important to be displaced by the measurable

Definitions of influence

    “The power of producing an effect, especially unobtrusively” Chambers dictionary (Mark Twain said: “If you don’t mind who gets the credit you can do anything.”) Influence is in some ways a polite word for power.

Influence is also something to do with brand. A stronger brand=more influence.

What is influence?



Level one: something changes because of what we have published

     Doctors change what they do. Ministers change policy. WHO decides to do things differently. Drugs are prescribed more or less. New techniques or methods are adopted. Old ones are abandoned.

What is influence?

    Written information on its own rarely leads to change “All journals do is take in other people’s washing” This sort of influence is probably rare and is hard to identify. Many different factors usually contribute to a particular change: so even if something we published contributed it could not be described as the cause.

Examples of change caused by research articles I

 Photodynamic therapy with a new drug might cause severe burns  Hettiaratchy S, Clarke J, Taubel J, Besa C. Burns after photodynamic therapy. BMJ 2000; 320: 1245

Examples of change caused by research articles II

 The use of albumin in critically ill patients may be dangerous  Cochrane Injuries Group Albumin Reviewers. Human albumin administration in critically ill patients: a systematic review of randomised controlled trials. BMJ 1998; 317: 235-240

Examples of change caused by research articles III

 Minocycline should not be used as the first line treatment of acne  Made the front page of the Daily Mail, which might be Britain’s most influential newspaper  Gough A, Chapman S, Wagstaff K, Emery P, Elias E. Minocycline induced autoimmune hepatitis and systemic lupus erythematosus-like syndrome. BMJ 1996;312:169-72

What is influence?

 

Level two: setting an agenda or legitimising an issue

Examples of where the BMJ might have done this, at least in Britain   Evidence based medicine Inequalities in health    Prison health care Medical error Doctors and the drug industry

What is influence?

 

Level three: leading by example and being folowed

Possible examples  bmj.com is free      open peer review BMJ ethics committee rapid responses on bmj.com

collected resources on bmj.com

BMJ patient advisory group

What is influence?



Level four: being quoted/cited

 “The only thing worse than being talked about is not being talked about.” Oscar Wilde

What is influence?

 Where might you be quoted?

        Other journals (impact factor) Cochrane reviews Guidelines Mass media Parliament (Hansard) Evidence Based Medicine/Journalwatch Important policy documents (for example, Institute of Medicine report) Presentations

What is influence?



Level five: being paid attention to

 Readership (preferably judged by others)  Website hits (in a week or over time)  Sales

What is influence?



Level six: being known about

 Widely known even if not quoted or read among international health professionals, political leaders, the public  If George Bush knows about you that’s more influential than if your mum does—sadly

Scoring influence the Richard Smith way



Level one: creating change

 Several clear cases  One case 5 points 3 points 

Level two: setting agendas and legitimising issues

 Several cases  One case 5 points 3 points

Scoring influence



Level three: leading by example

  Several clear cases One case 5 points 2 points 

Level four: being quoted

 Hundreds of quotes in all outlets     Hundreds of quotes in some outlets points Tens of quotes in all outlets Tens of quotes in some outlets points A few quotes in a few outlets point 5 points 4 3 points 2 1

Scoring influence



Level five: being paid attention to

   tens of thousands of readers, hits on the website and sales 5 points thousands hundreds 4 points 1 point 

Level six: being known about

 All health professionals and world leaders 2 points  Many health professionals 1 point

What does all this mean?

 Impact factor is widely used but flawed  It’s important to try and measure the influence of journals  We might agree on levels of influence; achieving change is the highest level  Measuring influence is difficult  But we need to find an alternative to impact factor

“What matters absolutely is the scientific content of a paper, and nothing will substitute for either knowing or reading it” Sidney Brenner Nature 1995;375:624

“How can a score count for so much when it is understood by so few and its value is so uncertain? In defence, worshippers of impact factors say we have no better alternative. Isn’t it time for the scientific community to find one?”