On Thursday 2nd February I attended an event at the University of Bath entitled “Assessment and Feedback: Transforming Curricula and Assessment in HE.” There were many interesting sessions , of which the following were some of my personal highlights.

Dr Alex Buckley from the University of Strathclyde spoke about their use of the TESTA  approach to reviewing assessment at a Programme level. Funded by the HEA, the project, standing for Transforming the Experience of Students Through Assessment,  originally involved four partner institutions, Bath Spa, Chichester, Winchester and Worcester.  It is now used in over 50 universities in an attempt to address deep challenges of assessment which Alex stressed need to be considered at Programme level. 

The TESTA website contains further information and resources on this approach, including a manual on how to implement it. Alex explained that it involves triangulating data from a programme audit, an assessment experience questionnaire for students and focus groups.  At Strathclyde those programmes who have engaged with TESTA have found it an extremely useful diagnostic tool as well as helping colleagues to think differently about assessment. After the process, programme teams have a workshop with educational developers where they consider practical changes that can be made which address the TESTA findings.   The TESTA website contains case studies and best practice guides with concrete suggestions. An example is reducing reliance on formal documentation to communicate standards, and putting greater effort into providing exemplars in order make explicit, and open to discussion, the meaning of assessment criteria and enable to students internalise these through marking exercises and self and peer assessment in relation to the exemplars.

Kay Sambell from Edinburgh Napier University expanded in the afternoon on Alex’s point that we need to facilitate student engagement with feedback rather than simply flagging up when feedback is being provided. However both Kay and Jane Rand recognised that this can be easier said than done. Literature provided evidence of the effectiveness of this a decade ago, Jane said, but much practice hasn’t changed.  

Kay went on to show some practical strategies that can be used. Her work is based on the Assessment Standards Knowledge Exchange at Oxford Brookes,  which provides a range of useful resources. She also referred to the work of Winstone, Nash, Parker & Rowntree (2016) entitled Supporting Learners’ Agentic Engagement With Feedback: A Systematic Review and a Taxonomy of Recipience Processes, which emphasises the importance of “proactive recipience” of feedback.

Kay talked about dialogic use of exemplars, which can take different forms; complete or part of an assignment, authentic or re-created, annotated with feedback or not. She went on to give an example of a peer review workshop from her own practice, the process for which is outlined in the photo on the right. Students really valued this opportunity for extended dialogue around assessment criteria.

Kay also referred to the work of Nicol, Thomas and Breslin on feedback production being recognised as just as valuable for learning as receipt of feedback. She recognised that students are sometimes reluctant to engage with engagement activities (such as peer review)! However, when they do engage they find them extremely useful, and she has found that exemplar assignments are highly effective as “vicarious peer assessment”. Kay mentioned the work of Carless and Chan on managing dialogic use of exemplars. This contains analysis of how teachers can orchestrate dialogue around exemplars. They suggest in the paper that ” the dialogic use of exemplars should be a core aspect of teachers’ repertoire of assessment for learning strategies, in that the development of student skills in making academic judgements is fundamental to the university experience.”  This is a point often made by D Royce Sadler, well known for his work on conditions necessary for students to benefit from feedback (Sadler, 1989) . In his own teaching Sadler makes use of a version of exemplars in the peer review of formative writing his students do. He puts his own attempt at the writing task in with his students’ which are distributed and peer reviewed. Sadler describes this in more detail in “‘Opening up feedback: Teaching learners to see“.

References

David Carless & Kennedy Kam Ho Chan (2016): Managing dialogic use of exemplars, Assessment & Evaluation in Higher Education

David Nicol, Avril Thomson & Caroline Breslin (2014) Rethinking feedback practices in higher education: a peer review perspective, Assessment & Evaluation in Higher Education, 39:1, 102-122

Sadler, D. R. (2013) Opening up feedback: Teaching learners to see” ( In Merry, S., Price, M., Carless, D., & Taras, M. (Eds.) Reconceptualising Feedback in Higher Education: developing dialogue with students. (Ch. 5, 54-63). London: Routledge.

Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18, 119-144

Naomi E. Winstone, Robert A. Nash, Michael Parker & James Rowntree (2016): Supporting Learners’ Agentic Engagement With Feedback: A Systematic Review and a Taxonomy of Recipience Processes, Educational Psychologist

On Wednesday 6th July I attended my first meeting of the Learning Design Cross Institutional Network, kindly organised by Lisette Toetenel of the Learning Design team at the Open University, and hosted by UCL.  The network has only been going for a year, but with an ever growing focus on systematic enhancement of teaching and the student experience across the HE sector, learning design is becoming a field of great interest to many institutions, and this was reflected in the attendance at the event, which was double the number of the previous meeting. Participants came from a range of organisations, including Liverpool, Edinburgh, UCL, Oxford, Greenwich, Northampton and the Open University.

So what exactly is learning design and why is it of such interest?

Fiona Hale from the University of Edinburgh has recently carried out a review of relevant literature and resources as well as interviews with practitioners across the sector, and opened the day with a summary of her work which is informing developments of the Edinburgh Learning Design roadmap (ELDeR) at her institution.  Fiona acknowledges the importance of learning design research but prioritises learning design practice which she defines as

the process of designing learning experiences (planning, structuring, sequencing) through facilitated activities that are pedagogically informed, explicit and make better use of technologies in teaching

(Learning Design Scoping: Final Report)

It is a creative process and distinct from development of courses, although it may involve prototyping of activities and getting feedback on these.

Why is this sort of process seen as so valuable? Firstly, research by Fiona and others has shown that traditionally course design often “just happens”. When there is an explicit design process this tends to be focussed more on content than thinking about the activities students will be undertaking in their  learning. Learning design foregrounds activity as opposed to content, on the basis that it is the activities which help students to engage with content. Content alone is not learning. 

Secondly traditional design practice often happens in silos, with individual academics working in isolation. Sometimes the person designing is not necessarily the person teaching, which can lead to a lack of coordination. The facilitated process Fiona and others use is a collaborative endeavour, involving as many colleagues from the unit and/or programme team as possible working together with library staff and learning technologists. Collaborative design has been shown to be effective, for example through the success of Gilly Salmon’s long-established Carpe Diem approach. (http://www.gillysalmon.com/carpe-diem.html )

Thirdly use of a structured process is easier to evaluate and scale as well as being easily shareable.  In Fiona’s case her team have been facilitating the workshops initially, but because there is a standard model and resources, they are now being cascaded down to facilitators in Schools, which has increased capacity and spread effective practice.

So what sort of approaches are other institutions taking?

At Edinburgh the process involves a workshop which takes place over two days, similar to the Northampton CaIeRO approach, both of which are based on Gilly Salmon’s Carpe Diem model.  Others use shorter interventions such as UCL’s one and a half hour ABC workshops. These do not provide an opportunity for any actual activity prototyping  though, in contrast to the approaches used at Edinburgh and Northampton.

All of the approaches are very active, with participants able to make use of visual resources such as activity cards to try out different sequences of types of activities in their designs. This hands on approach is quite different from other purely discursive review processes, and feedback from participants illustrates that they find this very effective in helping engagement and productive outcomes. 

Learning design as product

In addition to the process, learning design can generate a product, for which Dalziel uses the analogy of musical notation.  In the same way that musical notation attempts to capture the energy and complexity of music, a learning design representation tries to capture “the great ideas of teaching to be shared so that we can build that community knowledge”.

( “Learning Design – conceptualising a framework for teaching and learning online” (2016) ed Dalziel 2016 )

Different attempts have been made to represent learning designs, including through use of tools like LAMS and the Learning Designer.   Helen Walmsley-Smith from Staffordshire University is one of those who are looking at challenges of categorising learning designs. In order to demonstrate this point Helen invited participants to have a go at analysing and categorising a series of descriptions  of learning activities. This revealed that the different ways that colleagues talk about teaching and learning can present challenges, as can the way teachers describe activities in their instructions for students. 

One of the frameworks that has proven widely useful in helping to categorise learning activities and thereby help with achieving balanced learning design is Diana Laurillard’s:

(Adapted from: Laurillard, D. (2012) Teaching as a design science: building pedagogical patterns for learning and technology )

We have used both this and the eight learning events model in a number of contexts including workshops and the TEL essentials online course, as ways of helping colleagues think about different ways in which TEL can effectively support learning.

(Leclerc and Poumay (2005) The 8 Learning Events Model and its principles  http://www.labset.net/media/prod/8LEM.pdf)

Diana’s framework is used in the Learning Designer tool, which allows colleagues to visualise through a pie chart the proportion of each different type of  activity in their course. This can then prompt reflection, for example if there appears to be a dominance of “Acquisition” type activity , this may lead to consideration of balancing this more with discussion or practice. 

Future of learning design

Diana provided the final presentation of the day, which looked to the future of learning design. She sees it as an essential to enable scaling up of engagement with learning technology, and to help good pedagogies to cross to different subject areas and contexts, for example between online and blended courses. She ended by suggesting some ways in which MOOCs could help, for example in sharing ideas and designs, and taking advantage of analytics to further research evidence of effectiveness.

(Slides and audio from presentation by Diana Laurillard)

This was an extremely useful day, and provided a great opportunity to meet colleagues who are both starting out on the learning design journey and others who have significant experience and for whom this process is really gaining traction and leading to better designed courses.  Many thanks to all of the contributors. 

We are very keen to trial one or more of these approaches at Bristol, and would love to hear from any colleagues who may be interested.  To find out more please email:  roger.gardner@bristol.ac.uk

Last week I attended a workshop about Faculty Learning Communities (FLCs), organised by colleagues in Academic Staff Development . This was an extremely interesting day led by Milton Cox from Miami University (which Milton was keen to emphasise is not in Miami Florida, surprising many!)

In the morning we looked at evidence for the effectiveness of FLCs and recommendations for how to design and run them successfully.  Milton referred to a number of useful resources, many of which can be found on the FLCs website.  The afternoon started by considering scholarship of teaching and learning and how it can be supported through FLCs. Milton talked through examples, some of which had been presented at the Lilly conference on evidence-based teaching and learning. The day closed with a discussion of important factors when leading and facilitating FLCs.

Discussion and questions were varied during the day, including concerns about time,  opportunities afforded by cross disciplinary collaboration in FLCs, and ethical considerations in educational research. I was particularly interested in Milt’s repeated emphasis on the effectiveness of “people talking to people over time” in effecting changes in pedagogic practice. He spoke at length about evidence from implementation science, and particularly from the National Implementation Research Centre on the effectiveness of FLCs in delivering educational development.

Many thanks to ASD for organising the event. Milton was an engaging lead and the day provided plenty of food for thought for us at Bristol, as we embark on the new University  Strategy, which aims to nurture and grow our community of innovators and scholars in teaching and learning.

Blackboard have now released all of the presentations from this year’s Blackboard Teaching & Learning Conference, including our own about the work at Bristol to create packages to support the online submission and marking workflow. The slides are on Slideshare.

Congratulations to Blackboard on this conference, which was by some way the best I have been to, both in terms of the wonderful location at the University of Groningen in the Netherlands, the wide range of extremely useful presentations and the networking opportunities especially with other Blackboard users from around Europe and beyond. There were many interesting sessions which I could write about, but here are the top 5 things to interest and/or inspire me!

Students taking charge of Higher Education

Wednesday’s highlight for me was undoubtedly the host institution’s session entitled “Sharing Best Practice at the University of Groningen: A Student Centric Approach. This  covered two main areas, the development of a new student Portal and the role that students have in support for TEL.  The second of these was extremely impressive. There is a team of 24 students who provide first line support for a range of systems including Blackboard, which at Groningen is called Nestor. They have a thorough training and induction programme lasting 6 months, and then are typically employed, for up to 12 hours a week, for up to two and a half years.  Students manage the service, and they have recently developed a MOOC called “Students taking charge of higher education”, (with Futurelearn), which covers for example how students demonstrate professional  behaviours.

Jon Hummel from the Nestor support team talks about how they help develop professional behaviours in their student led team

Exemplary course design

With the TELED team’s recent focus on course design, I was interested to hear Lloyd Stock and Alan Mason talk about the Blackboard exemplary course programme (ECP). Although already familiar with the rubric used, I found the examples they showed useful (including the YouTube playlist of course tours by winners), together with their ideas around promoting good course design through an awards programme, whether that be submitting courses to Blackboard’s own ECP or internally within the University, as for example done at the University of Aberystwyth.  This theme was picked up several times during the conference including by Danny Monaghan, and Pete Mella from the University of Sheffield, who talked about their institution’s experience of improving the quality of Blackboard courses through an exemplary course program, including academic colleagues using these as evidence for assignments in their equivalent of the CREATE programme.

Natalie Thorne from the Distance Learning Unit at Leeds Beckett University gave some insights into how they use Blackboard to support distance learning programmes in a very effective way. Natalie demonstrated some excellent visual design of courses including activity and page layouts , engaging learning activities using both native Blackboard and external tools,  as well as practical tips including how to reduce clicks by linking directly to learning modules from a course menu. Natalie’s session really reinforced the point that Blackboard can be an effective  environment for online distance learning, as long as courses are well-designed.

Blackboard developments

In the Blackboard roadmap session there was an emphasis on an updated look and feel and responsive theme for our version, 9.1. A  new system and course theme is due out in Summer 2016, and responsive design and mobile optimisation including for submission of assignments in the Autumn.

Blackboard demonstrated the new responsive theme for 9.1

TELfest

Farzana Latif stimulated much interest with her account of the University of Sheffield’s TELfest event. The week long festival takes place annually, and has had a significant impact in raising awareness of and interest in TEL. In the first year they had 175 colleagues attending at least one of the sessions, and in the second this went up to 280. There are a mixture of sessions, for the more and less experienced, some run by the TEL team, some by academics and others such as the Library. It has helped Farzana and her team both promote certain themes and/or new opportunities, but is also a valuable opportunity for them to listen to staff views, needs and concerns, for example in their “Blackboard listening session”, where representatives from the company have attended. They have found that the events attract staff who had not engaged with TEL before, and have helped new champions to emerge.

Assessment and feedback

Large scale online exams, electronic management of coursework, and implementation of the Blackboard Grades Journey were recurring themes.  A number of universities in Europe are successfully doing large scale computer based exams using Blackboard (as well as other systems)  with Groningen itself being an excellent example. I was extremely impressed with the photos they showed of their 600 seater exam hall, which has flexible desk space so it can be used for handwritten or computer-based exams, including typed essay style exams using an adapted version of the Blackboard text editor. They reported that their online exams on Blackboard managed hosting are going very well.

Dr Lisette Bakalis from the University of Groningen talks about the digital exams they run.

There were some useful accounts of implementing the Grades Journey, which Joe Gliddon attended as this will be something he will be involved with in his secondment to the SLSP programme.  Last but not least, Joe and I had to wait until Friday morning to run our session “Submit work here” which looks at the work we have been doing here at Bristol on the use of Blackboard  packages to provide a scalable workflow for coursework assessment and feedback online. Our presentation was well-attended, over 30 participants almost filling the small room,  and there was plenty of discussion.

Participants in the Submit Work Here session in discussion

A number of colleagues from other institutions approached us with questions and comments, for example around ideas for other uses of packages, such as to provide learning activity templates.

Photo of our session posted in the Conference app

Overall the conference provided plenty of insights and ideas for us to consider.  For the Ed Dev team the interest shown in exemplary course design and how this can be given recognition both internally and externally was particularly inspiring and timely. 

By Roger Gardner

This was a thoroughly enjoyable event organised by the University of Bristol Graduate School of Education. Coinciding with the School’s centenary celebrations it aimed to look ahead to potential developments and changes in education over the next 100 years. All of the speakers were excellent and thought-provoking, but here are a few personal highlights.

Dr Richard Harris kicked off by suggesting that the current face-to-face University experience will become the exception rather than the norm in future with the majority of learning in HE being “pay as you go” from large online universities, backed by a mixture of philanthropy and commercial interest.

Professor Sri Subramanian outlined some of his work on brain computer interfaces and gestural interfaces, as well as morphees, (“self-actuated flexible mobile devices adapting their shapes on their own to the context of use in order to offer better affordances”).

Professor Mike Fraser reassured all those teachers present that the “Robot teacher” was not coming any time soon, stressing the importance of physicality and co-presence in learning environments and highlighting the gap and the nuances separating best and mechanical practice.

After a delicious lunch (as promised!) we re-convened to vote on some of the predictions, discussing whether they were likely to happen in 10, 20, 50, 100 years or never. Opinions were quite varied on many of the statements we considered (most are available on Google Moderator.)  There was quite a bit of discussion on the subject of so-called “smart drugs”, whether use will increase and and to what extent consumption of these can be considered “cheating” when other stimulating drugs such as caffeine are commonplace.

One emerging theme of interest seemed to be the area of genetics and education, for example speculation and concerns around genetic enhancement of learning ability. Another was wearable devices (highlighted in the Horizon Report 2013 shortlist), including the possibility of student learning being monitored through use of implants or wearable devices.

So plenty of food for thought, and a stimulating range of perspectives from the invited speakers.  I particularly liked the conversational approach of the event, in which Paul Howard Jones chatted with each panel member for ten minutes before inviting questions from the audience.

By Roger Gardner

As part of the Education Excellence Seminar series organised by Phil Langton in the Faculty of Medical and Veterinary Sciences, I presented recently on ideas for using technology to enhance marking and feedback.

The aim of the session was to raise awareness of a range of options in this area. In between some demonstrations and consideration of benefits and issues, there was plenty of discussion and questions. We looked at some examples from our collection of case studies on the TEL website, and there was interest from some colleagues attending in trying out some of the approaches in the coming year, including screencast feedback and Turnitin Grademark.