Monday 15 February 2016

Marginal Gains and the Plateau Effect: The Benefits of Evidence Based Practice (EIP)

The above twitter dialogue is a snippet of a conversation I had with Gary Jones and other educationalists on social media about the role of research in education. After almost 15 years as a teacher, and 7 as a senior leader, it would appear that there is at last a strong focus on building a profession based partly on research, to support experience based practice. But to what extent should teachers and school leaders be influenced by the body of evidence presently circulating? Both Gary Jones and Dylan Wiliam offer these perspectives on evidence based practice/disciplined practice:


and


However, personally I find myself nodding in agreement with Alex Quigley’s comments in his blog post: Just Don’t Call It Research about how teachers can best become evidence based practitioners:

I’m sure ‘disciplined inquiry‘ could be mangled to mean what we want it to mean, but I very much like Paul Hood’s helpful breakdown of the characteristics of disciplined inquiry:
  1. Meaningful topics are addressed
  2. Systematic, clearly described procedures are employed and described so that readers can follow the logic of the study and assess the validity of the study’s conclusion
  3. There is sensitivity to the errors that are associated with the methods employed and efforts are made to control the errors or consider how they influence the results
  4. Empirical verification and sound logic are valued: and
  5. Plausible alternative explanations are considered
I like these characteristics because they encourage teachers to apply their practical wisdom – recommended by Wiliam – but in a more disciplined way that we can typically apply in our busy working week, by using  ‘logic‘ and considering ‘plausible alternatives‘.

I also agree with Alex about the need to avoid the cherry picking of research. Therefore, in my attempts to avoid confirmation bias, I regularly read the work of David Didau and Greg Ashman in support of traditionalists approaches to teaching.  Their writings (Didau’s in particular) help me to question some of my deep (progressive) rooted beliefs about teaching.  Didau’s sustained references to cognitive science, and his style of writing always make it difficult to entirely deny his assertions. Especially as he fills his writings with subtle reminders about the need to question one’s beliefs about teaching:
“If your beliefs don’t bear up under close critical evaluation, then maybe. just maybe, you believe in something silly.”

"But the irrefutable evidence that we are so often wrong should at least give us reason to question our intuition. If the cognitive scientists are right, we could make a profound difference to how well our students learn. If all their empirical evidence turns out to be wrong, no one’s died. It may not be worth betting your life on, but it outweighs the risk of going with a hunch.”

Despite Didau’s reminders, research can never tell the complete story.  Teaching that has resulted in outstanding outcomes for my students year on year is evidence that what I am doing works.   Yet there is always room for improvement; and perhaps (for this teacher) evidence informed practice can lead to aggregated marginal gains and help myself and others avoid the plateau effect:

“…There appear to be important gains in teaching quality in the first year of experience and smaller gains over the next few career years. However, there is little evidence that most teachers continue to improve after the first three years.”  Steven G. Rivkin, Eric A. Hanushek, and John F. Kain in 2005

The particular area of my practice that I have been attempting to improve as an evidence informed practitioner is feedback. Whilst feedback has the potential to have a significant positive impact on learning, as evidenced by Hattie’s effects table below:
effect-size-table-john-hattie.png
http://www.learningandteaching.info/teaching/what_works.htm
It also has the potential to have a damaging impact on learning:

"In their review of feedback studies conducted between 1905 and 1995, Kluger and DeNisi (1996) found that in 38% of well-designed studies, feedback actually made performance worse—one of the most counterintuitive results in all of psychology."

I therefore politely request that as you read the remainder of this blog, please try and answer the following questions on my behalf:

  1. Have my students been actively involved in the feedback process?
  2. Do my students know what to do in order to improve their work?
  3. John Hattie stated; “I don’t think it is right that kids come to our schools to watch us work.”  Is this what is happening in my classroom?
  4. Has learning taken place?
  5. Has evidence informed practice benefited my teaching?

Part 1  - The Learning Environment

Both Dylan and Didau suggests that positive relationships are essential for effective teaching:

"Trust is a better predictor of teacher success than expertise. A teacher may know their subject inside out, but if their pupils don't trust them they won't feel very motivated."  Didau (2015: 344)

and

"Without the relationships all the research in the world won't matter." Dylan Wiliam

So I began, as I always do when joining a new school, with building positive relationships with the students. My series of blogs on creating a classroom culture for learning, has on the whole, been a reflection of this process: We make the weather, Mistakes, Being Human 1, Being Human 2, Along with my class blog Asker International School Grade 7 Blog.

Part 2 - Feedback Strategies

The Task
Grade 7 students were required to retell part of the original story of The Three Little Pigs.  Areas to consider:
  • Audience
  • First Person Perspective: Wolf or the Third Pig or alternative characters
  • Contemporary version
  • Speech
  • Internal Monologue
  • SPaG


In line with the International Baccalaureate's philosophy of backward planning, students were given the rubric at the beginning of the unit.  The language of the rubric was not dumbed down - I am of the opinion that students need to work within the language of the domain they are studying. Students must therefore become accustomed to using this language in the present context, and any future context.  Whilst I am a fan of the use of rubrics for backward planning, I am also conscious that they potentially have a number of pitfalls, pitfalls I am confident can be overcome providing certain step are taken. For example, in an article published by Faculty Focus Maryellen Weimer states:

In the study, research reports written using the rubric were significantly better than those written not using a rubric. It is fair to ask whether use of the rubric improved their research report writing in general or only this one time on this one assignment.”

One must therefore be aware that whilst the use of a rubric appears to benefit short term performance, there is no suggestion from this research that this performance will lead to improved performance within a different context, or a future context that requires the same knowledge and/or skills. However, Daniel Willingham argues that:

“Knowledge tends to be inflexible when it is first learned. As you continue to work with the knowledge, you gain expertise; the knowledge is no longer organized around surface forms, but rather is organized around deep structure...For example, your knowledge of calculating the area of rectangles may have once been relatively inflexible; you knew a limited number of situations in which the formula was applicable, and your understanding of why the formula worked was not all that clear. But with increasing experience, you were able to apply this knowledge more flexibly and you better understood what lay behind it. Similarly, it is probably expecting too much to think that students should immediately grasp the deep structure beneath what we teach them. As students work with the knowledge we teach, their store of knowledge will become larger and increasingly flexible, although not immediately.”

Possible Solution: The use of spacing in conjunction with unit rubrics can ensure that the short term gains made by using rubrics can be sustained over time, and hopefully ensure that the information is stored in long term memory:
“If nothing has changed in long-term memory, nothing has been learned.” Kirschner, Sweller & Clark

Teacher Feedback
My next step was to create opportunities for feedback to the students that matched my philosophy: that feedback is most effective in promoting learning if it involves the students in the process of deciding what the next steps should be, so that they are not passive recipients of the teacher’s judgments of their work. The steps provided below have attempted to stay true to this philosophy.

Firstly, I provided written feedback to students to consider based on the rubric, and the school literacy marking code.  The feedback was in response to students writing a first draft that only contained the first five sentences of their text.  I wanted the students to focus on producing quality rather than quantity to begin with.  Feedback (in all forms) would simply be focused on improving these initial first five sentences.

Secondly, in accordance with Andrew Quigley's advice, I also had a quick chat with the students to ensure they understood the comments/code provided in an attempt to avoid misunderstanding:


The need to talk to my students about the written feedback is particularly important considering that English is the second or third language for many of the students in my class.

Grade 7 Asker International School
Although, I am aware that Didau would suggest that this form of feedback simply implies mimicry rather than learning.

Students Responses To Teacher Feedback
Dylan Wiliam's argues that there is no point giving feedback unless you allow time to use the feedback, and that feedback should be more work for the student than it is for the teacher.  As such, students were given time during the following lesson to write their next draft.

Student A: first language Norwegian, second language English
Student B: first language Chechnyan, second language Norwegian, third language English (started learning 18 months ago)

Preparing Students for Peer Feedback
A recent study that focused on learning in the IB Primary Years had this to say about peer assessment:

Teachers were more equivocal about peer assessment, expressing doubts about its value as a valid assessment tool and cautioning about the risks of using peer assessment or feedback in the absence of strong modelling for students.” IB Research Summary Assessment 2016

In contrast, Hattie's comments regarding peer assessment paint a much more positive picture::


I am of the opinion that as teachers it is our role to ensure that students are equipped with an understanding of how to conduct peers assessment, and that as Dylan Wiliam suggests ensure the feedback is cognitive rather than emotional, which is the benefit of having a rubric to work with.  As such, prior to the peer feedback session students wrote down what they should look for based on their knowledge of the assessment rubric. I also intentionally avoided students placing this information into grade bands due to the potential issue surrounding interpretation of the various verbs that are often used to distinguish one grade band from another:

"A rubric is not like an engineering standard, Sadler explained. In such a standard, we can specify to the millimetre something quite explicit. Instead, a rubric will contain vague words such as ‘coherence’ and ‘flow’ and will have a graduation in performance that will move from ‘demonstrating a good command of sentence structure’ to ‘demonstrating a sophisticated command of sentence structure’ and so on. The danger is obvious; different teachers will interpret these statements differently." Ashman (2016: 95)

Additionally, I did not wish the students to align their marking to a particular grade band, rather to simply offer advice that would help move a fellow student forward in their (apparent) learning. This was to avoid the issue of students simply focusing on/or trying to work out the grade the feedback comments might be aligned to:

An interesting development from this process was that some students brought new criteria in the form of descriptive writing techniques that they referred to as rich vocabulary.  What transpired next was a conversation about the ways in which the students could enrich the language they were using. I therefore had a decision to make: either direct the students to focus only on the original rubric requirements, or to allow the students to make comments on additional elements attached to the notion of good writing.  In this case my response to allow additional information beyond the requirements of the rubrics was influenced by both David Didau's and Greg Ashman's comments on the use of assessment rubrics.  Didau states the following relating to the restrictive nature of rubrics:


Greg Ashman also appears to be of the same opinion when he writes:

"Imagine the following scenario: we have used our expert concept of quality or a sample of papers to derive a set criteria that distinguish between essays of differing levels of quality. These criteria are only a sample of all the aspects that we could notice but these are what we write into our assessment rubric. Teachers then view this rubric as a target and specifically focus on teaching students to demonstrate evidence of meeting these specific criteria, perhaps at the expense of other aspects of the performance." (Ashman 2016: 97)

I am inclined to think, that I had not recently read the work of both of these two educationalists then my response would have been to request that the students remain to the confines of the original rubric (anchoring effect maybe?). Instead, I saw this as an opportunity to tap into student curiosity, which is the bedrock of the inquiry based classroom. This was followed up by a series of lessons on using the senses to invoke descriptive writing similar to the information in the clip below.  It was also a classic example of how as teachers we should ensure that we teach the students and not just the subject.
Peer Feedback
The students did not conduct peer feedback in their usual classroom.  Instead, they moved to a neighbouring classroom.  This was part of my drive this year to adhere to the following advice:

"If you’re after sustained improvement then you want to introduce as much variability into your teaching as possible: change rooms, change seating, change displays, remove the comforting and familiar background to lessons; mix up topics.  These desirable topics will slow down performance but will lead to increased long term retention and transfer of knowledge between contexts.” (Didau 2015: 127)

Grade 7 Asker International School
This clip is on the school blog and in the public domain


Checking Peer Feedback
When you read about research that suggests that 80% of the feedback students get on their work is from each other, and 80% of that is wrong, then alarm bells perhaps start ringing.  But only as a reminder to ensure certain strategies are employed. First I requested that students read out the feedback they had received from their peers:

 
I also made certain that on my travels around the classroom I checked the written feedback for incorrect advice.  And whilst there were some mistakes, I agree with Alex Quiqley that these are opportunities for clearing up misunderstandings.  It would appear from my experience that the potential positive impact of peer assessment definitely outweighs the potential negatives providing teachers are aware of its possible pitfalls.  

Draft 3
Student were requested to write their next draft based on the feedback from the peer assessment session, and the lessons relating to the senses and descriptive writing.


                               
Self Assessment

Unlike peer feedback, self-assessment appears a favourable strategy in IB schools: “Self-assessment was highly valued for its ability to build reflective lifelong learners.” IB Research Summary Assessment 2016


In groups, students created the self assessment sheet (without access to the original rubric):

 

Students then completed the document individually with the addition of two individualised areas for improvement:
self assess 3.jpg


Final Version


Student B: first language Chechnyan, second language Norwegian, third language English (started learning 18 months ago)

Student C: first language Mandarin, second/third language Norwegian/English

Final versions can be found here


Reflection


  1. Have my students been actively involved in the feedback process?
  2. Do my students know what to do in order to improve their work?
  3. John Hattie stated; “I don’t think it is right that kids come to our schools to watch us work.”  Is this what is happening in my classroom?
  4. Has learning taken place?
  5. Has evidence informed practice benefited my teaching?

What Next… Spacing: Redesigning the Grade 6,7 & 8 MYP Language and Literature Curriculum

No comments:

Post a Comment