Transcript Assessment and Examinations John Kirby Graduate School Faculty of Medical Sciences

Assessment and Examinations
John Kirby
Graduate School
Faculty of Medical Sciences
University of Newcastle
Prof. John Kirby, the faculty postgraduate tutor, is responsible for the assessment process of the faculty’s postgraduate students,
looking at all of the assessment reports that come into the graduate school.
This presentation will firstly cover assessment and then the examination process.
Assessment
– Why?
• Monitor student progress
• Identify problems with
– Project
– Supervisor(s)-student relationship
• Completion rate
– ‘exit’ strategy for struggling students
– Student re-registration
Assessments are required by funding bodies but even without this requirement it is essential that students progress is monitored.
This allows the identification of problems hopefully at an early stage. This can help identify problems with projects, as although
projects are assessed at the start this is not a guarantee of success. Problems with students and their relationships with their
supervisors can also be identified. A major concern for this and all graduate schools is the completion rate, students submitting a
thesis on time. Late submission (over 4 years for a usual PhD student) is classed as a fail as far as the University is concerned.
The assessment process allows an exit strategy for students who are struggling with the academic rigour of a research degree. In the
instance of a PhD this will allow students with difficulties to write up for an MPhil, though students can at this point leave with nothing
and that could well be the best thing for that person. Few assessors are prepared to tick the assessment boxes that will allow for an
exit, greater use of these boxes may help avoid problems later in the project. This allows for student registration, students register for
a year at a time.
Assessment
• When?
Focussing on the 3 year PhD studentships. The assessment is done at 9 months in order that a positive outcome can be achieved,
before students are registered for their second year. It is necessary to know how a student is shaping up before the end of the first
year.
The second year assessment is less academically rigid, but shows the project is on time, but the final year assessment, ideally at 36
months, should give an idea of how the thesis is shaping. There is, of course the option for a fourth year for writing up.
Assessment
• What?
– First assessment
Concentrating now on the first year assessment, the students are expected to produce a document – as shown, including a table of
contents, a comprehensive literature review (it is probably now will they will discover what their project actually is). The students need
to have a clear idea of the aims – their aims, not the supervisors - a summary of the results to date, which may not be much in the
way of solid output, but they should be showing the potential to create methodologies and to be able to talk coherently about their
research tools. They crucially then need to discuss their strengths and weaknesses of their research and where they are going.
The limit is 7,500 words, which we hope students will be able to do in combination with their day-to-day work.
Assessment
• What?
– Subsequent assessments
–Final assessment to include a detailed thesis plan
There has to be a positive response to each assessment for the student to re-register for the following year.
Subsequent assessments are shorter, only a few pages in length, and should be used to monitor the student and project, ensuring
that everything is developing as expected by student and supervisor.
The final assessment should be a comprehensive thesis plan, giving the scope of results, or some guidance of when the experiments
are likely to be completed, to the assessment panel.
The assessment report form was designed for the
entire University so may not be entirely appropriate
for this faculty, but is recognised throughout.
The student is expected to complete a progress
report, in their own words, and the supervisor is
allowed to make comments on the student’s
progress, but can also indicate how well the student
is doing.
For training, the Graduate School provides the
infrastructure for research skills and training.
Comments can be made on the use of English by
international students.
The progress recommendation is important, as it
informs the Graduate School of the ability of the
student to progress to the next year.
Please use these panels - a tick in a box is poor return
for the student’s effort.
If the student has an unsatisfa
further assessment may be do
the assessment – this 2 month
student needs to register for th
may be necessary to recomme
even a termination of the stude
Please fill in the panels as indi
the student gets relevant feedb
efforts
Using this checklist may not be applicable
to this faculty, but gives some guidance to
the assessors Using this checklist may not
be applicable to this faculty, but gives
some guidance to the assessors
Purpose of an examination
• For the University
– To assess and maintain quality
– To mark ‘completion’ of the degree programme
• For the Student
– Potentially leads to award of a degree
– Is an important and memorable life event
• can be a real emotional roller coaster (for everyone
involved)
The focus here is mainly on PhD examination, but it is very similar with MD or MPhil.
Why an examination? For the university it is an assessment and maintenance of the quality of the research students, and to mark the
completion of the research programme – whether or not the student is going to get the degree they have applied for.
For the student it potentially leads to the award of a degree, and on the human side of things, it is a memorable life event. It can be
an emotional time for both students and supervisors!
Who needs an oral examination?
• All PhD and MD candidates
– Students and full-time RAs need one internal and one
external examiner
– Staff candidates require two external examiners and an
internal ‘moderator’
• Not all MPhil candidates
– Same criteria for examiners as above
– Oral examination held at examiners request
• Not only for weak students
• With a good candidate can be fun for everyone!
Bearing in mind that regulations vary from University to University, rules may differ as internal and external examiners.
All PhD and MD candidates require an oral examination. Students and full time Research Assistants both need one internal and one
external examiner, whereas staff candidates require two external examiners, plus an internal ‘moderator’, who doesn’t need to know
anything about the submitted thesis, but must ensure that the external examiners are familiarised with how Newcastle proceeds with
examinations.
Not all MPhil candidates require an oral examination, the thesis may be submitted alone, but examiners may want to meet the
candidate. An MPhil must not be treated as a failed PhD student – they may have had considerable time restraints and therefore may
be quite a strong candidate.
Appointment of examiners
• See nomination forms on Graduate School
http://medical.faculty.ncl.ac.uk/postgraduate/internal/staff.html
– Completed by supervisor and Head of school/Institute
• Examiners must:
– be cognisant of standards
– have subject knowledge (need CV)
– command authority
– not have played a role in the research
– (if external) not have been a member of Newcastle staff for
at least 3 years
– be able and willing to examine
The nomination forms are available from the Graduate School Office, or at the website http://medical.faculty.ncl.ac.uk/postgraduate/internal/staff.html, and must be completed by the supervisor and Head of School or
Institute.
Examiners must be aware of the standard required of the exam, must have detailed knowledge of the subject i.e. be an expert in their
field, research active, publishing good research. They must command authority, and must not have played a role in the research, or
co-authored papers. External examiners must not have been a member of Newcastle University staff for at least 3 years.
Finally, the examiner must be willing and able to examine – it is not necessary to take on board every thesis.
Internal examiners
• Liaise with supervisor, student (and sometimes external) to
arrange examination
• Ensure the examination complies with Newcastle University
protocol
• Ensure appropriate report forms are completed and submitted in
a timely manner
• Provide balance, fairness and ensure good examination conduct
• Potentially play a role during any appeals process
Internal examiners for a standard PhD have to liaise with supervisors (who play a major role), student, and sometimes the external
examiner, to arrange the examination. They have to ensure that the exam complies with the Newcastle University protocol, and must
ensure that the report forms are completed and submitted, particularly the joint report form. They must ensure that the exam is
completed fairly and in an appropriate manner. They also potentially play a role in any appeals process, so the consequences have to
be considered.
External examiners
• Usually is the scientific specialist
• Is an experienced examiner
– Chicken and egg….
• Often though to have the ‘casting vote’
– But there are procedures for disagreement
• Maintains inter-university quality
– Report provides important feedback to Graduate
School (and QA etc)
• May not be completely ‘up to speed’ with local
regulations!
External examiners are usually the specific scientific specialists, and are generally experienced, typically examining previously as
internal examiner. Almost always the examiners will agree, with the external examiner having perhaps the more important view, but
there are procedures in place for any disagreement.
The external examiner hopefully maintains inter-university standards, and provides feedback on the training that the candidate has
had.
They may not be totally aware of the local regulations of the University, and it is helpful to be aware that each university may have
totally different procedures for examination.
What if 2 external examiners?
• Need to appoint an internal moderator
• This person need not read or understand the thesis
• Role is to provide advice on Newcastle examination process
– May be required to present records of the exam if the result
is questioned (appeal process).
The moderator is appointed where two external examiners are required. They do not need to read or understand the thesis, but are
generally given the role of letting the external examiners know how Newcastle operates its examination policies, and may be required
to provide details of the examination records in the appeals process.
Criteria – all theses
• Should be:
– Authentic
– Scholarly
– Professional
– Well-structured, written and presented
These criteria are from the regulations, and are self-evident.
MPhil candidates
• Should
– Demonstrate advanced knowledge
– Have good knowledge of literature
• Theses need not
– Demonstrate consistent originality
– Be worthy of publication
MPhil candidates should demonstrate an advanced knowledge of their subject, with a good comprehension of literature.
The thesis does not need to demonstrate consistent originality, nor be worthy of publication, although quite often the MPhil thesis will
contain considerable material suitable for submission for publication.
PhD/MD candidates
• Should
– provide evidence of adequate industry
– demonstrate ability for originality
– understand relationship with wider field
– thesis should contain material worthy of publication
PhD/MD candidates should prove that they have worked sufficiently well to be able to provide an in-depth knowledge of their subject,
demonstrating their originality, and it is up to the examiner to find how wide their knowledge is of their given subject, which may not
be apparent in the thesis. The thesis itself should contain material worthy of publication, giving the students a drive to produce
published papers on their way to the end of their PhD, as no examiner can ever say no to that, prompting a drive to the publication of
data.
Types of thesis
• ‘Standard’
– Divided into chapters with results and
interpretations
• By publication
– A review and a series of ~5 related papers
– Can be difficult to examine as papers have already satisfied
external referees!
– Staff candidates only
There are generally two types of thesis, the first of which is ‘standard’, a bound copy divided into chapters with results, etc, and the
second, which is becoming more popular, by publication. Only staff candidates can submit their thesis by publication, which typically
consists of a written review and a succession of around 5 related papers. It should be clear that the student has contributed
significantly to work in the publications - something which the author must make clear. It is very difficult to examine this type of
thesis, as external referees have already been satisfied that the papers are of a good standard, but one thing an examiner can do is
make sure that the student fully understands the contents of the publication.
Useful WWW sites:
•
ncl.ac.uk/spo/examhand.html
– (examiners handbook)
•
ncl.ac.uk/calendar/volume1/…
– (conventions, regulations etc)
Any problems with these links can be resolved by going to the Graduate School Website, where all forms and documents mentioned
can be downloaded.
There is no word limit
for a thesis here at
Newcastle, but the limit
for the abstract should
be around 300 words,
ideally on one page.
Reading a Thesis
• Are you a proof reader or a scientist?
• You will need to provide a list of corrections if you require them
• I usually stick post-it notes to the margin to localise my
questions within the thesis
– Be sensitive though; hundreds of these can look very scary!
Some academics are more critical of the spelling mistakes a student might have made, than of the concepts and opinions in the
work, but clearly if it doesn’t make sense and the English is appalling, it is up to the examiner to recommend a rewrite. Where a
typographical errors are made, it may be necessary to provide a list of corrections. A lot of examiners now use post-its to mark any
points they’d like to raise , but remember it may be daunting for the student if you use a lot!
Preliminary Report
• Regulations vary between institutions -read them!
• Many (but not all) institutions require examiners to
independently produce reports before the examination
– Some need these to be submitted (well) before the
examination to flag up potential problems
The examiners should write a preliminary report before the oral examination. This does not involve ringing the other examiner, as this
is a procedural irregularity- for your own protection, please follow the rules! In Newcastle, the report is requested 2 weeks before the
exam, but it does not always follow – many submit only days before the exam.
How to conduct the examination - 1
• Arrange the room
– Often good to have pencils and paper to draw on
• At the start candidates can be very nervous!
– Put them at their ease if possible with a soft start (but don’t
anticipate the result!)
– Remember to arrange refreshment breaks
– Consider the candidate’s bladder
A lot of advice is given in the guidance notes, but in general it is a good idea to arrange the room and try to put the student at ease. A
good opening question is to ask them what they are most proud of in their thesis – this is probably the piece they are happiest with
and most comfortable to talk about. Do not anticipate the result of the examination. It is also recommended to take breaks for
refreshment or a toilet visit.
How to conduct the examination - 2
• Agree a plan with your co-examiner
• Remember:
– Oral examination of a good candidate can/should
be one of the most pleasurable academic
experiences for all involved
– Examination of a poor thesis/candidate can be
truly awful!
• You need to devise different strategies for both
situations
Before you start, agree a plan with your co-examiner to work out how you are going to proceed. Devise strategies for different
situations, as the candidate may be very nervous. If the candidate gives a good answer, it is not a problem to say so, and if they run
into difficulties it may be wise to have a break to give them a chance to recover their equilibrium.
RESEARCH DEGREE EXAMINATION
JOINT REPORT OF EXAMINERS
Student Candidates for PhD
Name of Candidate:
Title of Thesis:
Name and Address of External Examiner:
Name of Internal Examiner:
Date of Oral Examination:
SECTION A - RECOMMENDATIONS (tick the relevant box to indicate overall recommendation)
The Candidate be admitted to the degree
(a)(i)
That the candidate be admitted immediately to the degree of Doctor of Philosophy.
(a)(ii)
That the candidate be admitted to the degree of Doctor of Philosophy subject to minor
corrections of the text made to the satisfaction of the internal examiner, normally within a
period of one month of receiving formal notification of the corrections to be made.
(a)(iii)
That the candidate be admitted to the degree of Doctor of Philosophy subject to minor
revisions being made to the satisfaction of the internal examiner, normally within a period
of six months of receiving formal notification of the revisions to be made.
The Candidate be permitted to resubmit for the degree
(b)(i)
That the candidate’s thesis be deemed to be of a satisfactory standard, but that the
candidate be adjudged to have failed to satisfy the examiners in the oral examination
and that the candidate therefore be required to attend within six months either for a second
oral examination or for a written examination, as the examiners shall determine in their
written report.
(b)(ii)
That the candidate be adjudged to have failed to satisfy the examiners and the
candidate be permitted to revise and re-submit the thesis within twelve months without a
further oral examination.
(b)(iii)
That the candidate be adjudged to have failed to satisfy the examiners and the
candidate be permitted to revise and re-submit the thesis within twelve months and be
re-examined orally.
The Candidate be recommended for the Masters degree
(c)(i)
That the candidate has reached the standard required for the appropriate Master’s
Degree and should immediately be awarded that degree instead.
(c)(ii)
That the candidate has reached the standard required for the appropriate Master’s
Degree and should be awarded that degree instead subject to minor corrections of the
text made to the satisfaction of the internal examiner, normally within a period of one
month of receiving formal notification of the corrections to be made.
(c)(iii)
That the candidate has reached the standard required for the appropriate Master’s
Degree and should be awarded that degree instead subject to minor revisions being
made to the satisfaction of the internal examiner, normally within a period of six months of
receiving formal notification of the revisions to be made.
The Candidate be permitted to resubmit for the Masters degree
(d)
That the candidate be permitted to revise and re-submit the thesis for the appropriate
Master’s Degree within twelve months and be re-examined orally if the examiners so
require by indication in their written report.
The Candidate be adjudged to have failed to satify the examiners
e)
That no degree be awarded and that the candidate be adjudged to have failed.
This is t
form for
The top
degree i
there ar
correctio
clear fro
minor co
to the sa
internal
minor re
the satis
examine
but it is
is typica
candida
giving a
examina
they we
2 give v
resubmi
University of London
A.
B.
C.
Pass, subject to minor amendments to be
completed and checked by one or both examiners
within 3 months
Not pass, but candidate allowed to re-write the
thesis and resubmit it within 18 months for
examination by the same examiners (oral exam
may not need to be held)
etc….
Different universities have different report forms, as you can see from this, so careful consideration should be given in your
recommendations
At the end (if all has gone well)
•
•
•
•
Ask the candidate to withdraw for a few minutes
REMEMBER you do not award the degree!!
Work out what you wish to say, then invite the candidate back
Tell the candidate what recommendation you will be making to
the higher degrees committee (or other appropriate authority).
• Smile and shake hands
If all goes well, ask the candidate to leave for a few minutes, decide what you wish to say, as you must remember you are not
personally awarding the degree, then invite the candidate back. You can then tell the candidate of your recommendations to the
appropriate authority.
At the end (if it has NOT gone well) - 1
• Make absolutely sure you know what your options are (read the
regulations)
• You are not obliged to tell the candidate anything (although you
will feel some pressure to do so)
– the candidate will receive written confirmation of the outcome in
due course
• You may wish to speak to the supervisor
• You might need clarification of the regulations (Graduate
School)
If the examination has not gone well, make sure you know what your options are. You are not obliged to tell the candidate on the day,
but can let them know that they will receive written information in due course. You may wish to contact the supervisor to find out
where things have gone wrong, and you may need clarification of the regulations from the Graduate School so you know that what
you propose is right.
At the end (if it has NOT gone well) - 2
• You will need to fill in the report form with very
comprehensive details of any changes you require
– This outcome results in much more effort in the future for the
examiners!
• All report forms look different.
– Make sure you know the precise significance of a tick in
every optional box!
– For example, ticking box 3 (pass) at Newcastle can produce
a very different outcome from box 3 in London (fail)!
The report form, which again will be different from university to university, needs to be filled in with very comprehensive information of
any changes you require and both examiners’ preliminary reports will be looked at.
The appeal process
www.ncl.ac.uk/spo/AppealsForm.pdf
If things go particularly badly, the candidate may follow the appeal process. It is very hard to fail a student, as Universities often come
down on the side of the student.
Invitation to examine
• This is not necessarily an honour
• Why you and not someone else?
– Are you a ‘soft touch’?
– Are you the supervisor’s best friend?
• You will/should see the abstract of the thesis at the
time of invitation
• You have the right to REFUSE!
To get back to your invitation to be an examiner, you will have to consider all reasons why you have been chosen, and remember you
are not obliged to agree to the request.
Case study
• X agreed to act as external examiner for Y’s PhD thesis.
• The thesis was of marginal quality but X’s preliminary report suggested
that with modification and a satisfactory oral examination it was likely
that a +ve recommendation could be made.
• Close scrutiny of the thesis immediately before the examination
revealed references to an earlier MD thesis by the same candidate (at
the same university). X contacted the internal to request that a copy of
this thesis should also be available for scrutiny (it is reasonable to see
all reference sources cited in a thesis).
• Before meeting the candidate on the day of the oral, X and the internal
discovered that the earlier thesis reported about 70% of the data
presented in the PhD thesis (about 45 pages of the introduction to both
works was identical).
• X and the internal examiner commenced the examination.
Discuss the potential outcomes?