Near future teaching – notes from reading group

For our latest reading group, following Sian Bayne’s fascinating Near Future Teaching seminar for BILT, we wanted to look in more depth at the project materials and related reading.

Michael read ‘Using learning analytics to scale the provision of personalized feedback,’ a paper by Abelardo Pardo, Jelena Jovanovic, Shane Dawson, Dragan Gasevic and Negin Mirriahi. Responding to the need to be able to provide individual feedback to large classes of students, this study presented and tested a novel system for utilizing learning analytic data generated by student activity within a learning management system in order to deliver what the authors called ‘personalized’ feedback to students. As it was designed, the system allowed instructors to create small, one or two sentence pieces of feedback for each activity within a course. Based on these, each week students would be able to receive a set of ‘personalized’ feedback that responded to their level of participation. In the study, the authors found an improvement in student satisfaction with the feedback they received, but only a marginal improvement in performance, as compared to previous years. There were limits to the methodology — the study only made use of at most three years of student data for comparison — and the author’s definition of ‘personalized feedback’ seemed in practice to be little more than a kind of customized boilerplate feedback, but nevertheless the study did have a few interesting points. First, it was admirable in the way that it sought to use learning analytics techniques to improve feedback in large courses. Second, the authors took the well thought out step to not make the feedback given to be about the content of the course, but instead it focused on providing feedback on student study habits. That is, the feedback might encourage students to make sure they did all the reading that week if they weren’t doing well, or might encourage them to be sure to review the material if they had already reviewed it all once. Third, the article offered an interesting recounting of the history of the concept of feedback as it moved from focusing only on addressing the gap between targets and actual performance to a more wholistic and continuous relationship between mentor and student.

Suzi read Higher education, unbundling, and the end of the university as we know it by Tristran McCowan. This paper starts with a thorough guide to the language of unbundling and the kinds of things that we talk about when we talk about unbundling, followed by an extensive discussion of what this means for higher education. My impression from the article was that “unbundling” may be slightly unhelpful terminology, partly because it covers a very wide range of things, and partly because – if the article is to be believed – it’s a fairly neutral term for activities which seem to include asset-stripping and declawing universities. As an exploration of the (possible) changing face of universities it’s well worth a read. You can decide for yourself whether students are better off buying an album than creating their own educational mixtape.

Roger read “Future practices”.   For world 1 , human led and closed, I was concerned that lots was only available to “higher paying students” and there was no mention at all of collaborative learning. For world 2, human led and open, I liked the the idea of the new field of “compassion analytics”, which would be good to explore further, lots of challenge based learning and open content. World 3, tech led and closed, was appealing in its emphasis on wellbeing in relation to technology, and a move away from traditional assessment, with failure recognised more as an opportunity to learn, and reflection and the ability to analyse and synthesise prioritised. From world 4 I liked the emphasis on lifelong learning and individual flexibility for students eg to choose their own blocks of learning.

Chrysanthi read Future Teaching trends: Science and Technology. The review analyzes 5 trends:

  • datafication – e.g. monitoring students’ attendance, location, engagement, real-time attention levels,
  • artificial intelligence – e.g. AI tutoring, giving feedback, summarizing discussions and scanning for misconceptions, identifying human emotions and generating its own responses rather than relying only on past experience and data,
  • neuroscience and cognitive enhancement – e.g. brain-computer interfaces, enhancement tools like tech that sends currents to the brain to help with reading and memory or drugs that improve creativity and motivation,
  • virtual and augmented realities – e.g. that help to acquire medical skills for high-risk scenarios without real risk, or explore life as someone else to develop empathy, and
  • new forms of value – enabling e.g. the recording and verification of all educational achievements and accumulation of credit over one’s lifetime, or the creation of direct contracts between student-academic.

I liked it because it gave both pros and cons in a concise way. It allows you to understand why these trends would be useful and could be adopted widely, at the same time as you are getting a glimpse of the dystopian learning environment they could create if used before ethical and other implications have been considered.

Suggested reading

Feedback, NSS & TEF – notes from reading group

Chrysanthi read “Thanks, but no-thanks for the feedback”. The paper examines how students’ implicit beliefs about the malleability of their intelligence and abilities influence how they respond to, integrate and deliberately act on the feedback they receive. It does so, based on a set of questionnaires completed by 151 students (113 females and 38 males), mainly from social sciences.

Mindset: There are two kinds of mindsets regarding malleability of one’s personal characteristics; People with a growth mindset believe that their abilities can grow through learning and experience; people with a fixed mindset believe they have a fixed amount of intelligence which cannot be significantly developed. “If intelligence is perceived as unchangeable, the meaning of failure is transformed from an action (i failed) to an identity (i am a failure)” (p851).

Attitudes towards feedbackSeveral factors that influence whether a person accepts a piece of feedback – e.g. how reflective it is of their knowledge and whether it is positive or negative – were measured, as well as 2 outcome measures.

Defence mechanisms: Defence mechanisms are useful in situations we perceive as threatening, as they help us control our anxiety and protect ourselves. But if we are very defensive, we are less able to perceive the information we receive accurately, which can be counterproductive; e.g. a student may focus on who has done worse, to restore their self-esteem, rather than who has done better, which can be a learning opportunity.

The results of the questionnaires measuring the above showed that more students had a fixed mindset (86) than growth (65) and that their mindset indeed affected how they responded to and acted on feedback.

  • Growth mindset students are more likely to challenge themselves and see the feedback giver as someone who can push them out of their comfort zone in a good way that will help them learn. They are more motivated to change their behaviour in response to the received feedback, engage in developmental activities and use the defence mechanisms considered helpful.
  • Fixed mindset students are also motivated to learn, but they are more likely to go about it in an unhelpful way. They make choices that help protect their self-esteem, rather than learn, they are not as good at using the helpful defence mechanisms, they distort the facts of the feedback or think of an experience as all good or all bad. The authors seemed puzzled by the indication that fixed students are motivated to engage with the feedback, but they do so by reshaping reality or dissociating themselves from the thoughts and feelings surrounding said feedback.

Their recommendations?

  • Academics should be careful in how they deliver highly emotive feedback, even if they don’t have the time to make it good and individualised.
  • Lectures & seminars early in students’ studies, teaching them about feedback’s goal and related theory and practice, as well as action action-orientated interventions (eg coaching), so they learn how to recognize any self-sabotaging behaviours and manage them intelligently.
  • Strategies to help students become more willing to experience – and stay with – the emotional experience of failure. Eg, enhance the curriculum with opportunities for students to take risks, so they become comfortable with both “possibility” and “failure”.

I think trying to change students’ beliefs about the malleability of their intelligence would go a long way. If one believes their abilities are fixed and therefore if they don’t do well, they are a failure, a negative response to feedback is hardly surprising. That said, the responsibility of managing feedback should not fall entirely on the student; it still needs to be constructive, helpful and given in an appropriate manner.

Suzi read: An outsider’s view of subject level TEFA beginner’s guide to the Teaching Excellence FrameworkPolicy Watch: Subject TEF year 2 by the end of which she was not convinced anyone knows what the TEF is or how it will work.

Some useful quotes about TEF 1

Each institution is presented with six metrics, two in each of three categories: Teaching QualityLearning Environment and Student Outcomes and Learning Gain. For each of these measures, they are deemed to be performing well, or less well, against a benchmarked expectation for their student intake.

… and …

Right now, the metrics in TEF are in three categories. Student satisfaction looks at how positive students are with their course, as measured by teaching quality and assessment and feedback responses to the NSS. Continuation includes the proportion of students that continue their studies from year to year, as measured by data collected by the Higher Education Statistics Agency (HESA). And employment outcomes measures what students do (and then earn) after they graduate, as measured by responses to the Destination of Leavers from Higher Education survey – which will soon morph into Graduate Outcomes.

Points of interest re TEF 2

  • Teaching intensity (contact hours) won’t be in the next TEF
  • All subjects will be assessed (at all institutions), with results available in 2021
  • Insufficient data for a subject at an institution could lead to “no award” (so you won’t fail for being too small to measure)
  • Resources will be assessed
  • More focus on longitudinal educational outcomes, not (binary) employment on graduation
  • It takes into account the incoming qualifications of the students (so it does something like the “value add” thing that school rankings do) but some people have expressed concern that it will disincentivise admitting candidates from non-traditional backgrounds.
  • There will be a statutory review of the TEF during 2019 (reporting at the end of the year) which could change anything (including the gold / silver / bronze rankings)

Suzi also read Don’t students deserve a TEF of their own which talks about giving students a way in to play with the data so that, for example, if you’re more interested in graduate career destinations than in assessment & feedback you can pick on that basis (not on the aggregated data). It’s an interesting idea and may well happen but as a prospective student I can’t say I understood myself — or the experience of being at university — well enough for that to be useful. There’s also a good response talking about the kind of things (the library is badly designed, lectures are at hours that don’t make sense because rooms are at a premium, no real module choice) you might find out too late about a university that would not be covered by statistics.

Roger read “How to do well in the National Student Survey (NSS)” an article from Wonkhe,  written in March 2018. The author, Adrian Burgess, Professor of Psychology at Aston University, offers some reflections based on an analysis of NSS results from 2007 to 2016.

Whilst many universities have placed great emphasis on improving assessment and feedback, this has “brought relatively modest rewards in terms of student satisfaction” and remains the area with the lowest satisfaction.

Burgess’ analysis found that the strongest predictors of overall satisfaction were “organisation and management” closely followed by “teaching quality”.

Amy read Feedback is a two-way street. So why does the NSS only look one way?, an article by Naomi Winstone and Edd Pitt. This piece highlighted the issue that the NSS questions on feedback are framed as if feedback should be a passive experience – that students should be given their feedback. In 2017, the question was changed from “I have received detailed comments” to “I have received useful comments”. Both the old and new question frames feedback as something that is received, a ‘transmission-focussed mindset’, whereas Winstone and Pitt argue that feedback should be a two-way relationship – with the student working with the feedback and their tutor to develop.
The authors do not believe that changing the NSS question will solve all of the problems with students perception of feedback (though it will definitely help!) but they do believe that by promoting feedback as something that individuals work with, have responsibility for and seek out if they feel they need to develop in a certain area, that gradually the mindset will change and become a more sustainable form of learning for students.

Suggested reading

From WonkHE

From the last time we did assessment & feedback, which was July 2017 (I’ve left in who read what then)