Review!
What is Evaluation?
Evaluation seeks to assess the impact of a
defined project – a time limited intervention in this case. There
are two clear purposes of evaluating an intervention:
- To measure individual pupil progress (or lack
of it) enabling you to plan the next steps for their
development. (Has their personal objective been achieved? Is
further intervention needed? Should intervention take the same form
?)
- To decide if the intervention is useful –
reflecting on the value of intervention for future use and how it
can be improved. (Should the intervention be used again? What
could be changed to make it more effective?)
Evaluation is important as it enables you to
monitor individual pupils’ progress and make informed decisions
about:
- Whether an intervention is effective – is it
having a positive impact on pupil outcomes?
- What factors made the intervention effective
(or not)?
- Could any changes be made to make an
intervention more effective? For example is the lead person
confident in their skills? Is there a need for staff training? Did
pupils enjoy the intervention? Did pupils think there was a point
to them doing the intervention?
- Should you continue with an intervention or
repeat an intervention with another group?
- Is an intervention is cost effective (in
time or money) – could similar improvements have been made with no
intervention / another intervention? (When interventions are
formally evaluated the progress of a ‘control’ group – a
group of young people with similar needs who are not
receiving the intervention – is also assessed. This enables
you to decide if the intervention group would have made the same
rate of progress even if the intervention was not put in
place.)
What makes an effective evaluation?
An effective evaluation is one that is:
FIT FOR PURPOSE
You need to be clear about why you are evaluating; what the results
are for; who will use the results; and whether you have chosen the
most appropriate assessment tools
VALID
You need to be careful that you are measuring
what you think you are. If you use a complex questionnaire to
get pupils to rate their confidence you may end up with information
which says more about their literacy / language skills than their
confidence!
RELIABLE
Reliability is the extent to which the measure
would get the same result if you repeated it
MANAGEABLE
Is the planned evaluation practical? Do
you have time to collect and interpret the information. There
is no point collecting information if you aren’t going to use
it!
Making an Evaluation FIT FOR PURPOSE – What to measure
This
document contains details of the Tools to assess the impact of
interventions
An intervention’s impact should be measured in
terms of how well it achieved the objectives set for it. For
example, the impact of a reading intervention should be measured in
terms of changes in reading skills.
An intervention may also bring about changes
in other areas (for example children in a reading skills
intervention may gain in confidence) and these are known as
unintentional impacts. However, in the reading example, the
intervention’s effectiveness is not measured by how much pupils’
confidence increases – although this could be an objective for the
reading intervention the next time it is used!
The objectives for an intervention should be
based on the pupils’ identified needs and what you hope the
intervention will achieve. ‘SMART’ or ‘SMARTER’ goals /
targets can be used:
Specific – worded in clear
language
Measurable – written in a way
that can be observed / counted / measured
Agreed – discussed and owned
by all those involved in the intervention (eg staff, parents,
pupils)
Realistic – achievable
Time Limited – a time frame
is set
Evaluated – planned means to
evaluate
Review – Use of evaluation
data for future planning
In group interventions these may be different
for each individual and will be recorded as ‘target competencies’
(document to be added shortly) on the Intervention
Sheet.
Within the provision management software there
are links to potential targets based on the National Strategies
objectives for Reading,
Writing,
Speaking and
Listening and Numeracy
(National Curriculum Levels P Scale 4- Level 5).
For pupils where unwanted behaviours are the
main concern, targets from SEAL may be appropriate where the
behaviours reflect under developed social and emotional
skills. SEAL
Classroom observation checklist here.
Alternatively, it is suggested that schools
work with pupils to identify mutually agreed targets and
interventions are designed / selected to these. Behaviour
Targets for July 2009 can be viewed here.
Your school’s Educational Psychologist and /
or staff from locality based CEIT teams can work with schools
on solution oriented individual target setting with pupils.
Contact details can be found in the guide to
local support services.
How to Measure the Impact of an Intervention
Evaluations need to be planned before an
intervention begins. If we want to measure the impact an
intervention is having we need to know where skill/knowledge levels
were before the intervention so they can be compared to
skill/knowledge levels after the intervention. We also need
to know how we are going to measure skills/ knowledge.
In the reading example, if an
intervention was designed to increase a child’s sight word
vocabulary, we would want to know how many sight words the child
knew before the intervention (the baseline measure) so we could
compare it to how many sight words were known after the
intervention. We would also need to know how we were going to
measure this.
There are a selection of methods that can be
used to measure the impact of an intervention, see ‘Tools to
assess the impact of interventions’. This provides
information about methods of assessments and potential advantages
and limitations of their use.
National Strategies
The National Strategies suggest considering
the following in deciding whether an intervention is effective.
Success criteria for an intervention should
be:
- At least double the usual expected rate of
progress ie ratio gain of 2 or more / Ratio gain = Gain in months
/ Length of intervention in
months
- Progressing at least 1 National Curriculum
sub level every 2 terms
- For behaviour any success criteria must be
negotiated with the child
- It is accepted that for the 1% of pupils
with severe and complex needs, their rate of progress may be
slower.
Questions to ask of data
- What was the average progress for the
intervention?
- Was this equal to/better than/worse than the
success criteria?
- Did any pupils make exceptional
progress?
- If so – why?
- Did any pupils make inadequate progress?
- If so – why?
Other contextual points to consider
- Pupil attendance
- Pupil punctuality
- Pupil attitude/self esteem
- Pupil readiness for intervention is based on
more than entry level
- Other interventions pupil accessing
- Sessions missed due to another school
activity eg (trips)
- TA absence
- TA redirected to other duties
- Quality of TA deliver
- Appropriate teaching or learning styles
- Quality of accommodation
- Timing of sessions
- Time of day
- Is intervention taking the place of core
subject entitlement?
- Timetabling for convenience or
achievement
- Interruptions or distractions during
interventions
- Has the skills/knowledge gained from the
intervention been transferred into the wider curriculum and made an
impact on (a) NC Levels and (b) progress towards predicted levels
as evidenced by school tracking system?
- Is Teacher Assessment secure?
