The Value of NCATE

A Study of Research Institutions and NCATE Accreditation AACTE, Friday, February 8, 2008

Emerson J. Elliott NCATE, February 2008

1

Initial report from a study

 Research institutions that are accredited by NCATE  Interviews of deans, NCATE coordinators and faculty  Perceptions of NCATE accreditation 2

Topics in this session

A.

B.

C.

D.

E.

Setting for the study Who participated Interview questions Response themes Summing up 3

A. Setting for the study

   AACTE Fall 2004 study of deans CADREI fall 2006, and AACTE committee  Streamline     Reduce burden and cost Address time commitment required of BOE and UAB members Dovetail program review and unit review Make more collaborative, less punitive—more like some other specialized accreditors Brought to the NCATE Executive Board 4

A. Continued, Setting

 Executive Board also set goals for NCATE’s management plan (Fall 2006-Spring 2007)  Reduce the burden of accreditation   Improve service to institutions Increase the value and perceived value of the accreditation process to institutions  And NCATE was concerned that:  Research universities are the source of new knowledge in educator preparation. and for NCATE unit and specialized professional standards. . . 5

A. Continued, Setting

  But faculty from research universities are infrequently involved with NCATE  Research university faculty are often not among the participants in standards writing   And—prior to a change in 2007—research university faculty had participated rarely as examiners or on NCATE’s policy boards [3:15 session at AACTE today] Arranged meeting with research university deans, January 2007 6

A. Continued, Setting: NCATE meeting with research universities

 Outcomes     Symbiotic relationship between research on teaching and learning produced by research universities and use of that research to improve educator preparation Continue discussions with CADREI and AACTE Urge more BOE and board candidates from research institutions Streamlining to ensure an efficient and effective system  As one way to support these outcomes, and to ground the “value” goal in the management plan, NCATE decided to ask the research institutions themselves what they think about NCATE accreditation. 7

B. Who participated?

  Five research institutions accredited by NCATE Five interviews in each institution  Dean   Coordinator Three faculty designated by the dean     4 associate deans 4 department chairs 6 program directors and/or SPA coordinators 1 assessment coordinator and doctoral student 8

B. continued, Who participated: Interviews

 All conducted by telephone    All interviewees gave explicit permission to record the interview All interviews followed the same interview questions All interviewees were promised anonymity  All transcribed 9

  

B. Continued, Who participated: Selected institutions

   All in the East of the Mississippi Three are private not-for-profit, two are public Three participated in the NCATE program review process, two did not All five are doctorate granting, four are Carnegie “high” R & D ($10 to $150 million in 2005), one is Carnegie “very high” (over $250 million in 2005) Two have around $20 million in social sciences R & D in 2005 Enrollments for 2004 range from around 12,000 to more than 42,000 10

B. Continued, Who participated: What is not reflected in the participating institutions?

     Institutions that NCATE does not accredit Institutions in the West “All” NCATE accredited research institutions— selected from two recent cohorts of UAB action Sample is small But:  Still found a range of differences across institutions and individuals 11

B. Continued,Who participated: Accreditation experience

    All had accreditation visits in 2004-2006   One was having an initial visit Four were continuing visits Two had all standards met Two required focused visits related to standards 1 and 2 and are now fully accredited One has a coming focused visit related to standards 1 and 2 12

B. Continued, Who participated: Time warp

YEAR Semes ter UNIT PROG RAM Prog Rpts submit ted VISIT UAB action Intervi ews S 2001 2002 2003 F S F S F New Stds 1 st phase, Transition 2 nd phase 3 rd phase Assessment, syllabi, and program evidence S 2004 F Full PR submit PR submit S 2005 F Full S 2006 F Full S 2007 F Full New 1 st cohort PR submit 2 nd cohort PR submit 3 rd cohort 4 th cohort 5 th cohort 6 th cohort 7 th cohort V V UAB UAB V V UAB UAB I

13

C. Interview Questions

 Overall topics were:    Value that respondents associate with NCATE accreditation How NCATE does its job Advice about how accreditation could be improved  Findings from 3 questions and then 6 themes— topics that recurred in the interview responses 14

C. Continued, questions: (1) Overall reactions

 72% (18 respondents) “positive”  Positive, very positive, useful learning experience, important to do  Includes 3 deans  28% (7 respondents) “negative”     Technical and prescriptive requirements Time and resource intensive Too much assessment, compliance, SPA instructions changed Includes 2 deans—different experiences 15

C. Continued, questions: (1) Overall reactions

 Additional observations  Many at our institution think we don’t need national accreditation—we have really high standards    The process allowed for considerable flexibility in applying standards The team were true professionals The self-study aspect is useful 16

C. Continued, questions: (2) Most helpful

MOST helpful  Deans—reflection, self study, faculty collaboration  Coordinators—assessment  Faculty—fitting assessments to standards, collegial activities among the faculty 17

C. Continued, questions: (3) Least helpful

LEAST helpful    Deans—time, cost, burden, team visit Coordinators—mixed: Praxis, changing rules, duplicate NCATE and state requirements, lack of research institution team members Faculty  Assessments—confusion, rubrics for standards, changing requirements  BOE team—composition, logistics, use of web information 18

D. Response “themes”

 Frequently recurring topics that were not explicit interview questions       Research institution self descriptions BOE teams Program review and SPAs Defining evidence for NCATE TEAC references States—contrasting perspectives 19

D. Continued, themes: (1) Research institutions

 19 comments, 12 respondents, all 5 institutions; 24 points    Character of the institutions      Accreditation is not the way they think Candidates arrive at the graduate level and have completed subject content courses Specialization Emphasis on scholarship and research Always under study Professional life of the faculty    Own projects and funding NCATE takes time from grant writing Faculty change courses all the time, and their assessments  Little motivation to volunteer for NCATE work BOE teams often don’t understand their qualities 20

D. Continued, themes: (2) BOE teams

  48 references (largest number), 18 respondents in 5 institutions.

Distinction between one institution that characterized its visit as “disastrous” and all other

Difficult visit All other institutions

Positive comments Negative comments Other comments Recommendations Total 0% 80% 0 20% 100% 43% 21% 11% 25% 100% 21

D. Continued, themes: (2) BOE teams

 Positive statements about BOE teams--12  The team—true professionals, collegial, business-like, outstanding team, chair from a large research university, respected what we do, team understood who we are, all in this together, good team, worked hard 22

D. Continued, themes: (2) BOE teams

 Negative statements about BOE teams  The team in “all other” institutions—6   lack of preparation, did not use web-based exhibits, lack of research university peers, overemphasis on logistics The team in the “difficult visit” institution—16  failed to ask for information, did not listen, did not use provided documentation, tense exit interview, members not prepared, did not make good use of the Sunday evening poster session 23

D. Continued, themes: (2) BOE teams

 Recommendations—11 of the 48 comments    Chair/members from research institutions (8) Train members about research institutions Assure that teams arrive better prepared  To use electronic data   For more collegial interaction To use time not just to “find data, but to clarify and elaborate” 24

D. Continued, themes: (3) Program Review and SPAs

  22 references from 10 respondents in all 5 institutions; total of 51 points made Inconsistencies and changes—20     Across SPAs and between SPAs and NCATE—some would accept GPA or Praxis data, others would not Changes during the accreditation process Certain SPAs are difficult to work with Lack of fit between the standards and graduate level initial preparation 25

D. Continued, themes: (3) Program review and SPAs

 Concurrences with the program review process—14       More consistent use of assessments Agreement on just 6 to 8 assessments Writing for national recognition helped to redirect programs Faculty worked hard “because there’s this element of pride” Our program’s gotten better Comments by reviewers were fair, right on the money 26

D. Continued, themes: (3) Program review and SPAs

  Complaints—9  Praxis data not aligned; can’t get sub scores  Limited feedback   Took too long Just wanted numbers Others—2  Respondent’s own institution made changes  Faculty would rather fight with their professional organization than adopt standards 27

D. Continued, themes: (3) Program review and SPAs; Recommendations—6

   More consistency    Equivalent demands for all SPAs More coordination across SPAs and between SPAs and NCATE Limit the number of assessments Better fit with graduate level preparation   Write standards for initial preparation that recognize prior candidate preparation in subject content Provide more examples of evidence for “other” professional preparation Reconsider way decisions are made about SPA standards  Need more “we’re in this together,” less territoriality, more at-large partners 28

D. Continued, themes: (4) Evidence

   24 references by 10 respondents from 4 institutions 18 comments on evidence generally    NCATE is too prescriptive too much reliance on “numbers and tables” Not a research base to back up some data requirements   Institutions use grades as evidence of accomplishment Assessment data cannot be aggregated meaningfully across different levels of programs 6 comments on evidence for diversity  Numbers fail to inform the goal of “cultural competence” 29

D. Continued, themes: (4) Evidence; Recommendations

   Use a broader definition, not just “numbers on tables” Let institutions decide   More anthropological—look at what we’re doing and how faculty and our publics interpret that Let institutions self-define their mission, and ask for reporting of evidence within broad parameters—limiting the number of assessments Address particular issues   Take on measuring of student learning based on what good teaching and learning are Focus on ethnic diversity and “cultural competence” 30

D. Continued, themes: (5) TEAC

  10 references made by 6 respondents from 4 institutions; total of 13 points 7 points were set in a context of perceived problems with NCATE        NCATE is overly prescriptive Some required pieces of data are not supported by research Differences between SPAs and NCATE frustrate faculty NCATE is “bean counting” NCATE is burdensome, tedious, time intensive, costly Important qualities of a program cannot be captured in numbers Standards don’t align with the way the state is going 31

D. Continued, themes: (5) TEAC

  Three respondents ventured impressions of TEAC or recognized it as an alternative There were 3 recommendations   NCATE should look at what it does “from a business perspective”, but leave room for compromise TEAC and NCATE representatives should “work out a whole new accreditation system. . . Good aspects of both, but neither is a perfect system”  BUT don’t go “so far in another direction that we don’t have a set of standards that people have to step up to meet” 32

D. Continued, themes: (6) States

  13 references by 6 respondents from just 2 institutions Starkly different portraits of states, so state influence is a strong factor in accreditation, at least for some institutions  NCATE parallels the state, should keep coordinated, make state exam pass rate count for NCATE, can defer to our state on state assessments, would not have completed NCATE accreditation if state had not insisted  Negativity about NCATE is really directed at the state; state report is different only in minor ways—a duplicate, unnecessary, requirement; disparity in the standards that “regular” and “alternative” programs are held to but state refuses to acknowledge 33

E. Summing up

 Respondents found much that was good, e.g.:  I like what I’m hearing about web-based submissions  Electronic submission is very helpful to organize data, other information,   Positive that NCATE is interested in what approved programs are saying and thinking Preparing for NCATE made the regional accreditation easy 34

E. Summing up

 They made recommendations that NCATE needs to sift and consider. Particularly,  Evidence     Teams and recruitment Program standards and standards decisions Regional accreditor experiences Continuing accreditation, candidate data only 35