Notes from the recent GW4 meeting at Cardiff University

Posted by

Last Friday, Han, Mike and I attended a GW4 event in Cardiff, where the main topics on the agenda were students as collaborators, and shared projects that we can embark on together.

The day started with brief updates from each team:

Cardiff…

  • Have a new space for learning and teaching experimentation
  • Are working on a Curriculum Design Toolkit, as part of which they are looking at unbundling content to work in different ways for different markets
  • Have a Learning Hub Showcase (http://www.cardiff.ac.uk/learning-hub)
  • Have funding that students can bid for, for teaching projects
  • Ran a Summer Intern Project – one of which focused on advice in how to use lecture capture
  • Had a blank course rollover with a new minimum standard

Bath…

  • Have a major curriculum design project upcoming
  • Are moving towards programme-level assessment rather than modular
  • Have new funding for staff and students to work together
  • Are championing a flipped learning approach
  • Have a placement student
  • Are working on a ‘Litebox project’ (http://blogs.bath.ac.uk/litebox/) where students create an environment where the whole University can learn about new and existing technologies for use in learning and teaching, and share their experiences of them
  • Are expanding their distance learning postgraduate numbers


During the second part of the day, we talked about students as producers. Splitting into small groups, we shared our experiences, challenges and tips for working with students on both accredited and unaccredited courses. It was widely accepted by the group that collaboration with students is mutually beneficial. Students are able to move from being passive consumers of knowledge to genuine partners in their education, and we as professionals have a lot to gain from the expertise, connections with other students, and knowledge of life at the university that students can offer us.

The experience of working with students a Bristol, Bath and Cardiff has been positive but limited. All three universities have hired student interns in the past, but would like to do more in terms of making ‘students as producers’ a key underpinning concept in accredited courses. The expectations of ‘what university learning will be like’ puts a dent in the willingness of students to engage with accredited collaborative projects. We discussed how students may see universities as institutions of teaching rather than of learning, particularly as  tuition fees have risen, and expect more teacher-to-student time for their money. Our group talked about introducing the idea of innovative learning techniques earlier in students’ degree programmes, and even on pre-university open days, in order to change the expectations of students from traditional lecture-based learning to problem-based modules and more.

For the last part of the day, we talked about projects that the GW4 could collaborate on, and contributed to this padlet board. We all shared ideas, then each of us cast three votes for the projects we’d like to see most. A common theme was the sharing of knowledge and expertise in areas like FutureLearn, ePortfolios and case studies. We also talked about working together to put pressure on companies or to bid for shared funding in order to improve practice in ways that wouldn’t be possible for a single institution.

Bath have volunteered to host the next meeting in February or March, in which we’ll talk about ePortfolios and assessment.

Programme level assessment – notes from the reading group

Posted by

Suzi read Handbook from UMass – PROGRAM-Based Review and Assessment: Tools and Techniques for Program Improvement

A really clear and useful guide to the process of setting up programme level assessment. The guide contains well-pitched explanations, along with activities, worksheets, and concrete examples for each stage of the process: understanding assessment, defining programme goals and objectives, designing the assessment, selecting assessment methods, analysing and reporting. Even the “how to use this guide” section struck me as helpful, which is unheard of.

The proviso is that your understanding of what assessment is for would need to align with theirs, or you would need to be mindful of where it doesn’t. As others do, they talk about assessment to improve, to inform, and to prove and they do also nod to external requirements (QAA, TEF, etc in our context). However, their focus is on assessment as part of the project of continual (action) research into, and improvement of, education in the context of the department’s broader mission. This is a more holistic approach that might bring in a wide range of measures including student evaluations of the units, data about attendance, and input from employers. I like this focus but it might not be what people are expecting.

During the group we discussed the idea of combining some of the ideas from this, and the approach Suzanne read about (see below). A central team would collaborate with academic staff within the department in what is essentially a research project, supporting conversations between staff on a project, bringing in the student voice and leaving them with the evidence-base and tools to drive conversations about education in their context – empowering staff.

(Side note – on reflection I’m pretty sure this is the reason this particular reading appealed to me.)

Chrysanthi read Characterising programme‐level assessment environments that support learning by Graham Gibbs & Harriet Dunbar‐Goddet.

The authors propose a methodology for characterising programme-level assessment environments, so that they can later be studied along with the students’ learning.

In a nutshell, they selected 9 characteristics that are considered important either in quality assessment or for learning (e.g. variety and volume of assessment). Some of these were similar to the TESTA methodology Suzanne described. They selected 3 institutions that were different in terms of structure (e.g. more or less fixed, with less or more choice of modules, traditional or variety in assessment methods etc. They selected 3 subject areas, the same in all institutions. They then collected data about the assessment in these and coded each characteristic so there would be 3 categories: low, medium, high. Finally, they classified each characteristic for each subject in each institution according to this coding. They found that the characteristics were generally consistent within institution, showing a cultural approach to assessment, rather than a subject- related one. They also identified patterns, e.g. that assessment aligned well with goals correlates with variety in methods. While the methodology is useful, their coding of characteristics as low-medium-high is arbitrary and their sample small, so the stated quantities in the 3 categories are not necessarily good guidelines.

Chrysanthi also watched a video from the same author Suzanne read about: Tansy Jessop: Improving student learning from assessment and feedback – a programme-level view (video, 30 mins).

There was a comparison of 2 contradictory case studies, 1 that seemed like a “model” assessment environment, but where the students did not put in much effort and were unclear about the goals and unhappy, and 1 that seemed problematic in terms of assessment but where students knew the goals and were satisfied. The conclusion was that rather than having a teacher plan a course perfectly and transmit large amounts of feedback to each student, it might be worth encouraging students to construct it themselves in a “messy” context, expanding constructivism to assessment as well.

Additionally, as students are more motivated by summative assessment, have a staged assessment where students are required to complete some formative assessment that feeds into their summative assessment. Amy & Chris suggested that this has already started happening in some courses.

Finally, the speaker noted that making the formative assessment publicly available, such as in blog posts, motivates the students, that it would be better if assessment encouraged working steadily throughout the term, rather than mainly at peak times around examinations and that feedback is important for goal clarity and overall satisfaction.

Both paper and video emphasised the wide variety in assessment characteristics between different programs. In the paper’s authors’ words, “one wonders what the variation might have been in the absence of a quality assurance system”.

The discussion went into the marking system and the importance students give to the numbers, even when they are often irrelevant to the big picture and their future job.

Amy summarised a summary she had created after attending a Chris Rust Assessment Workshop at the University. The workshop focussed on the benefits of programme-level assessment, looking at the current problems with assessment in universities and offering practical solutions and advice on creating programme-level assessments. The workshop started by looking at curriculum sequencing – it’s benefits and drawbacks, and illustrated this with examples where it had been successful.

Chris then discussed ‘capstone and cornerstone’ modules as a model for programme-level assessment, and explain where it had been a success in other universities. He discussed the pseudo-currency of marks and looked at ways we can alter our marking systems to improve student’s attitude to assessments and feedback. He ended the session by looking at the ways you can engage students with feedback effectively, and workshop attendees shared their advice with colleagues on how they engage their students with feedback. You can find the summary here.

Suzanne read Transforming assessment through the TESTA project by Tansy Jessop (who will be the next Education Excellence speaker) and Yaz El Hakim, which briefly describes the TESTA project, the methods they use and the outcomes they have noted so far. There are also references within the text to more detailed publications on specific areas of the methods, or on specific outcomes, if you want to find out more detail.

In brief, the TESTA project started in 2009, and has now expanded to 20 universities in the UK, Australia and the Netherlands, with 70 programmes having used TESTA to develop their assessment. The article begins by giving a pretty comprehensive overview of the reasons why programme assessment is so high on the agenda, including the recognition that assessment affects student study behaviours, and that assessment demonstrates what we value in learning, so we should make sure it really is focused on the right things. There was also a discussion about how the ‘modularisation’ of university study has left us with a situation of very separated assessments, which make it difficult to really see the impact of assessment practices across a programme, particularly for students who take a slower approach to learning. Ultimately the TESTA project is about getting people to talk about their practices on a ‘big picture’ level, identity areas which could be improved, and then work from a base of evidence to make those improvements. There is a detailed system of auditing current courses, including sampling, interviews with teaching s and programme directors, student questionnaires, and focus groups. the information from this is then used as a catalyst for discussion and change, which will manifest differently in each different programme and context.

The final paragraph of the report sums it up quite well: “The value of TESTA seems to lie in getting whole programmes to discuss evidence and work together at addressing assessment and feedback issues as a team, with their disciplinary knowledge, experience of students, and understanding of resource implications. The voice of students, corroborated by statistics and programme evidence has a powerful and particular effect on programme teams, especially as discussion usually raises awareness of how students learn best.”

Suggested reading

Games and Simulation enhanced Learning (GSeL) Conference

Posted by

GSeL Logo - 8bit style graphics.Last Thursday I caught the 8.44am cross country to Plymouth to attend the first Games and Simulation enhanced Learning (GSeL) Conference. GSeL is a newly formed interdisciplinary research theme group, part of Plymouth University’s Pedagogic Research Institute and Observatory (PedRIO).

VR Hackathon

Plymouth University 2nd November 2017

The main event was on Friday the 3rd, but a session billed for the previous day – ‘Hackathon: VR for Non-programmers’ sounded promising. So, I ventured down a day early to channel my inner geek. I’ve got a basic (but rusty) understanding of coding so hoped that the ‘non-programmers’ tagline was true. Turns out the session was well designed for those with little to no experience. Michael Straeubig expertly guided around 15 attendees with differing skills through the process of creating a simple VR equivalent of ‘Hello World’ over the course of 2 hours.

The Hackathon was a hands on workshop running through downloading a-frame framework template project files from Michael’s github, installing open source software atom.io for editing/coding.

Sounds complicated? Yeah, sort of – but Michael’s laid-back-whilst-enthusiastic delivery helped fill in the gaps and moved at a steady pace we could all keep up with. He guided us through creating our first scene, adding in various 3-dimensional objects, altering their size and colour. Setting up a local server on our laptops via atom.io, we were able to move beyond viewing the 3D space we’d programmed and view it on a pair of budget VR goggles (Google Cardboard) on our smartphones.

It was a great primer for dipping toes/feet/legs into creating simple VR spaces from scratch using free tools. The a-frame project files supplied had additional examples of how to extend and develop. I don’t mind admitting I spent a large part of the rest of the day tinkering. A Michael drily observed during the session, we’d become ‘cool coders’.

Games and Simulation enhanced Learning (GSeL) Conference

Plymouth University 3rd November 2017

A day of talks and workshops based around the use of Games and Simulations, both real-life and digital. Some personal highlights for me included:

Professor Nicola Whitton’s keynote ‘Play matters: exploring the pedagogic value of games and simulation’ which tapped eloquently into themes like Failing without Consequence and motivation/engagement through playing games.

Matthew Barr (University of Glasgow) ‘Playing games at University: the role of video games in higher education and beyond’ – a great talk about his work with ‘gaming groups’ and the benefits cooperative video game playing brought students. “If I ruled the world, every student would play Portal 2”.

James Moss (Imperial College) ‘Gamification: leveraging elements of game design in medical education’ – some brilliant examples of using scenario based games in medical education. ‘Stabed to Stable’ involved scenario/persona based learning, a horizontal whiteboard, post-it notes and pens, with students clustered around trying to map out processes (checks/actions) they needed to go through, whilst James periodically helped guide or threw related spanners into the works. An overhead time-lapse video showed a dynamic session in action. A second game involved teams becoming the ‘medical officer’ helping a team of characters climb Everest. This simulation included mountain noise recordings (incrementally getting louder), random wildcards presenting challenges, lighting changes and James squirting participants in the face with water.

Michael Parsons (University of South Wales) ‘Keeping it Real: Integrating Practitioners in a Public Relations Crisis Simulation’ – shared his experience running a real-time simulation for PR students. Students attempted to handle a recreation of the infamous Carnival Triumph ‘Poop Cruise’ in the University’s Hydra Minerva Suite. The simulation used news report recordings, archived social media posts and live interaction with actors via telephones over several ‘acts’ to simulate a PR teams attempts to handle a particularly disastrous voyage. It all went well till the passengers were close enough to land to get mobile phone reception (and access to social networks).

The conference presented a feast of examples of using games and simulations in teaching and learning. From creating crosswords to utilising digital badges to recognise achievements to data visualisation in Virtual Reality, the place was abuzz with ideas. The focus on the potential of play and gaming to engage students meant the event had something for everyone, whether die-hard techy or strictly analogue.

Role and future of universities – notes from the reading group

Posted by

Maggie read Artificial intelligence will transform universities. Here’s how – World Economic Forum
The article presents the idea that Universities have created a need to innovate and evolve to meet the changing needs caused by the upsurge in AI. Already, the marking of student papers is becoming a thing of the past as AI is able to assess and even ” flag up” issues with ethics.Students are less able to distinguish between teacher marking and that of a “bot”. Teaching is additionally being impacted as students are able to undertake statistics courses using AI, massively reducing learning (and human teacher) time, with apparently equal learning and application outcomes. The author argues that Universities will need to up their game regarding employability and indeed attractive employment (remuneration). The paper is an easy-to-read item and clearly outlines the range of benefits and subsequent issues in relation to AI. All pertinent.

Suzi read three short opinion pieces: What are universities for and how do they work? by Keith Devlin, Everything must be measured: how mimicking business taints universities by Jonathan Wolff, and Universities are broke. So let’s cut the pointless admin and get back to teaching by André Spicer.

Devlin focused largely on the role of research within maths departments. The most interesting part, for me, came at the end when he talked about universities as communities and learning occurring “primarily by way of interpersonal interaction in a community”. Even without thinking about research outputs, there is value then in having a rich and varied community with faculty who have deep love and enthusiasm for their subject.

Wolf provided a clear and compelling dissection of how current educational policy is creating adverse incentives to community-mindedness (both within and between universities). Something detrimental to the education sector, which is such a significant part of the UK economy.

Spicer provides an insight into how this feels as an academic. He talks about how “In the UK, two thirds of universities now have more administrators than they do faculty staff.” and describes academics are “drowning in shit” (pointless admin).

For me, Spicer’s solutions for what universities could do to change this weren’t so compelling. If I could change one thing I would look at how we cost (or fail to cost) academic staff time. Academics can feel that they are expected to just do any amount of work they are given, or at least they often have no clear divide between work and not-work and have to constantly negotiate their time.

Amy didn’t read, but watched Why mayors should rule the world – Benjamin Barber – and would highly recommend it. Our modern democracy revolves around ancient institutions – we elect leaders most of us never meet and feel like we have very little input into the democratic process. This isn’t the case in cities – the leaders of cities, mayors, are seldom from anywhere other than the city they look after. They went to the local schools, they use the public transport and hospitals – they’ve watched their city grow. They have a vested interest in improving it. Positive changes towards existential issues such as climate change and terrorism are happening in cities (he gives an example of the LA port, which after an initiative to clear up, reduced the city’s overall emissions by 20%), and something can be learnt from the way that they operate. There are networks of mayors across the world, with a sense of competitiveness between them as to who can be the best city. Mayors from different cities meet up and share their practices, helping other cities implement changes using best practice, without the bureaucracy of central government slowing change down.

Suzanne watched  What are universities for? the RSA talk by Professor Stefan Collini and  Professor Paul O’Prey. The second half of the video was more focused on the way that higher tuition fees have changed the nature of the relationship between universities and students, but the introduction to the talk was much more on the topic we were discussing today. Stefan Collini began by saying that he believes universities are partially protected spaces which prioritise ‘deepening human understanding’, and that there are few if any other places where this happens. He compared them to other organisations which do research, such as R&D departments in industry, or teams working in politics, but said the difference was that universities were able to follow second order enquiries, and look at the boundaries of topics and knowledge, as they didn’t have a primary purpose of furthering one particular thing or ideal. So, although there are many benefits, such as increased GDP, from the kind of enquiry universities do (the ‘deepening of human understanding’ he started out with), that isn’t their aim or goal. He also went on to say that although we tend to see universities as primarily for the benefit of the individual students (furthering their careers, developing their own skills and knowledge) they should be seen as providing public good as well, for the reasons outlined above. In the group we discussed his basic premise, that universities are ‘protected spaces’, and decided that we aren’t sure that is really the case (especially with so much research being funded by grants from industry). However, it did lead to an interesting discussion about what we feel universities are actually for, if they aren’t what Collini outlined.

Suggested reading

Tool for creating ‘Deep Links’ to a Blackboard Course/Organisation

Posted by

Blackboard Deep Link Generator

A frequent question we’re asked at this time of year is ‘How can I link directly to a Blackboard Course?’

It’s something that can be done, but simply sending out a courses URL will only work if the recipient is logged in/authenticated. Often the use case for sending out direct links means that it’s really unlikely the user will be logged in, so they need to be prompted. Sending out an email that says ‘log into Blackboard and then come back and click this link’ is obviously a nonsense.

This is where ‘deep linking’ comes into play. By amending the URL and adding a bit of extra text at the beginning you can create a link that forces the user to sign in/authenticate and then redirect them to the desired course, bypassing the Blackboard Home page. Unfortunately processing a Bb URL in this way can feel like you’re dabbling in the dark arts.

We were asked about this yesterday, so I’ve re-purposed one of the tools I used to use to do it all for you. Click the link above to launch.

Note: This will only work for University of Bristol URLs, so external visitors will need to view source and amend themselves – you should only need to change the ‘prefix’ variable to your own institutions login pages. 

Designing and Evaluating Accessible Learning Experiences 

Posted by

I attended an Accessibility in Education workshop in London last Wednesday (24th May).  Lisa Taylor-Sayles and Dr Eric Jensen presented two strands – design and evaluation.

Lisa’s presentations covered designing for accessibility and inclusivity. Her focus was on Universal Design for Learning (UDL).  Designing learning experiences fit for everyone, regardless of their needs.

She began with a potted history, WW2 veterans returning with disabling injuries. This led to changes in approach to town planning, infrastructure, and assistive technology. Later the principles of Universal Design developed into the Three Principles for UDL.

Read more about the Seven Principles for Universal Design here.

Read more about the Three Principles for Universal Design for Learning

Lisa gave practical advice on small changes that build to improve learning experiences. Simple steps such as using a tool to check colours benefit everyone. Rather than assuming you’ll retrofit for accessibility if needed, you increase inclusivity.

This approach dovetailed well with Eric’s strand which focussed on effective evaluation. He covered Formative evaluation, surveys and qualitative methods such as Empowerment Evaluation. Eric gave real life examples of where he’s used these approaches in his work (and what works and doesn’t).

Sessions and workshops alternated between the two strands, keeping it fresh. For me this also helped cement the need for evaluation needs to be core to any learning design. Would definitely recommend the event if it’s repeated.

https://www.methodsforchange.org/designing-evaluating-accessible-learning-experiences/

Video – notes from the reading group

Posted by

Hannah read ‘Motivation and Cognitive Strategies in the Choice to Attend Lectures or Watch them Online‘ by John N Bassilli. It was quite a in depth study but the main points were:

  • The notion of watching lectures online has a positive reaction from those who enjoy the course and find it important, but also from those who don’t want to learn in interaction with peers and aren’t inclined to monitor their learning.
  • From the above groups, the first group is likely to watch lectures online in addition to attending them face-to-face, whereas the second group are likely to replace face-to-face interaction with online study.
  • The attitude towards watching lectures online is related to motivation (ie. those who are motivated to do the course anyway are enthusiastic about extra learning opportunities), whereas the actual choice to watch them is related to cognitive strategies.
  • There is no demonstrable relation between online lecture capture and exam performance, but often the locus of control felt by students is marginally higher if they have the option to access lectures online.

Amy recommended Lifesaver (Flash required) as an amazing example of how interactive video can be used to teach.

Suzi read three short items which lead me to think about what video is good for. Themes that came up repeatedly were:

  • People look to video to provide something more like personal interaction and (maybe for that reason) to motivate students.
  • Videos cannot be skimmed – an important (and overlooked) difference compared to text.

The first two items were case studies in the use of video to boost learning, both in the proceedings of ASCILITE 2016.

Learning through video production – an instructional strategy for promoting active learning in a biology course, Jinlu Wu, National University of Singapore. Aim: enhance intrinsic motivation by ensuring autonomy, competence, and relatedness. Student video project in (theoretically) cross-disciplinary teams. Select a cell / aspect of a cell, build a 3D model, make a short video using the model and other materials, write a report on the scientific content and rationale for the video production. Students did well, enjoyed it, felt they were learning and seem to have learn more. Interesting points:

  • Students spent much longer on it than they were required to
  • Nearly 400 students on the module (I would like to have heard more about how they handled the marking)

Video-based feedback: path toward student-centred learning, Cedomir Gladovic, Holmesglen Institute. Aim: increase the student’s own motivation and enhance the possibility for self-assessment and reflection. They want to promote the idea of feedback as a conversation. Tutor talking over students online submission (main image) with webcam (corner image). Students like it but a drawback is that they can’t skim feedback. Interesting points:

  • How would tutors feel about this?
  • Has anyone compared webcam / no webcam?
  • Suggested video length <5 mins if viewed on smartphone, <10 mins if viewed on monitor

Here’s a simple way to boost your learning from videos: the “prequestion” looks at the effect of testing whether students remember more about specific questions and more generally when they are given prequestions on a short video. Answer seems to be yes on both counts. They thought that prequestions were particularly useful for short videos because students can’t easily skim through to just those topics.

Roger read “Using video in pedagogy”, an article from the Columbia University Center for Teaching and learning.

The article primarily focuses on the use of video as a tool for teacher reflection. The lecturer in question teaches Russian and was being observed. As she teaches in the target language which her observer didn’t speak her original motivation was to make the recording then talk the observer through what was happening. In actual fact she discovered additional benefits she had not envisaged. For example she was able to quantify how much time she was speaking compared to the students (as an important objective is to get students speaking as much as possible in the target language, and the teacher less). Secondly she could analyse and reflect on student use of the vocabulary and structures they had been taught. Thirdly it helped her to reflect on her own “quirks and mannerisms” and how these affected students. Finally the video provided evidence that actually contradicted her impressions of how an activity had gone . At the time she had felt it didn’t go well, but on reviewing the video afterwards she actually saw that it had been effective.

Suggested reading

Evidence – notes from the reading group

Posted by

Suzi read Real geek: Measuring indirect beneficiaries – attempting to square the circle? From the Oxfam Policy & Practice blog. I was interested in the parallels with our work:

  • They seek to measure indirect beneficiaries of our work
  • Evaluation is used to improve programme quality (rather than organisational accountability)
  • In both cases there’s a pressure for “vanity metrics”
  • The approaches they talk about sound like an application of “agile” to a fundamentally non-technological processes

The paper is written at an early point in the process of redesigning their measurement and evaluation of influencing. Their aim is to improve the measurement of indirect beneficiaries at different stages of the chain, adjust plans, “test our theory of change and the assumptions we make”. Evaluation is different when you are a direct service provider than when you are a “convenor, broker or catalyst”. They are designing an evaluation approach that will be integrated into day to day running of any initiative – there’s a balance between rigor and amount of work to make it happen.

The approach they are looking at – which is something that came up in a number of the papers other people read – is sampling: identifying groups of people who they expect their intervention to benefit and evaluating it for them.

Linked to from this paper was Adopt adapt expand respond – a framework for managing and measuring systemic change processes. This paper presents a set of reflection questions (and gives some suggested measures) which I can see being adapted for an educational perspective:

  • Adopt – If you left now, would partners return to their previous way of working?
  • Adapt – If you left now, would partners build upon the changes they’ve adopted without us?
  • Expand – If you left now, would pro-poor outcomes depend on too few people, firms, or organisations?
  • Respond – If you left now, would the system be supportive of the changes introduced (allowing them to be upheld, grow, and evolve)?

Roger read “Technology and the TEF” from the 2017 Higher Education Policy Institute (HEPI)  report “Rebooting learning for the digital age: What next for technology-enhanced higher education?”.

This looks at how TEL can support the three TEF components, which evidence teaching excellence.

For the first TEF component, teaching quality, the report highlights the potential of TEL in increasing active learning, employability especially digital capabilities development, formative assessment, different forms of feedback and EMA generally, and personalisation. In terms of evidence for knowing how TEL is making an impact in these areas HEPI emphasises the role of learning analytics.

For the second component, learning environment, the report focusses on access to online resources, the role of digital technologies in disciplinary research-informed teaching, and again learning analytics as a means to provide targeted and timely support for learning. In terms of how to gather reliable evidence it mentions the JISC student digital experience tracker, a survey which is currently being used by 45 HE institutions.

For the third component, student outcomes and learning gain, the report once again highlights student digital capabilities development whilst emphasising the need to support development of digitally skilled staff to enable this. It also mentions the potential of TEL in developing authentic learning experiences, linking and networking with employers and showcasing student skills.

The final part of this section of the report covers innovation in relation to the TEF.  It warns that “It would be a disaster” if the TEF stifled innovation and increased risk-averse approaches in institutions. It welcomes the inclusion of ’impact and effectiveness of innovative approaches, new technology or educational research’ in the list of possible examples of additional evidence as a “welcome step.” (see Year 2 TEF specification Table 8)

Mike read  Sue Watling – TEL-ing tales, where is the evidence of impact and In defence of technology by Kerry Pinny. These blog posts reflect on an email thread started by Sue Watling in which she asked for evidence of the effectiveness of TEL. The evidence is needed if we are to persuade academics of the need to change practice.  In response, she received lots of discussion, including and what she perceived to be some highly defensive posts.  The responses contained very little by way of well- researched evidence. Watling, after Jenkins, ascribes ‘Cinderella Status’ to TEL research, which I take to mean based on stories, rather than fact.  She acknowledges the challenges of reward, time and space for academics engaged with TEL. She nevertheless  makes a pleas that we are reflective in our practice and look to gather a body of evidence we can use in support of the impact of TEL. Watling describes some fairly defensive responses to her original post (including the blog post from James Clay that Hannah read for this reading group). By contrast. Kerry Pinny’s post responds to some of the defensiveness, agreeing with Watling – if we can’t defend what we do with evidence, then this in itself is evidence that something is wrong.

The problem is clear, how we get the evidence is less clear. One point from Watling that I think is pertinent is that it is not just TEL research, but HE pedagogic research as a whole, that lacks evidence and has ‘Cinderella status’. Is it then surprising that TEL HE research, as a  subset of  HE pedagogic research, reflects the lack of proof and rigour? This may in part be down to the lack of research funding. As Pinny points out, it is often the school or academic has little time to evaluate their work with rigour.  I think it also relates to the nature of TEL as a  set of tools or enablers of pedagogy, rather than a singular approach or set of approaches. You can use TEL to support a range of pedagogies, both effective and non-effective, and a variety of factors will affect its impact.  Additionally, I think it relates to the way Higher Education works – the practice there is and what evidence results tends to be very localised, for example to a course, teacher or school. Drawing broader conclusions is much, much harder.  A lot of the evidence is at best anecdotal. That said, in my experience, anecdotes (particularly form peers) can be as persuasive as research evidence in persuading colleagues to change practice (though I have no rigorous research to prove that).

Suzanne read Mandernach, J. 2015, ” Assessment of Student Engagement in Higher  Education: A Synthesis of Literature and Assessment Tools“, International Journal of Learning, Teaching and Educational Research Vol. 12, No. 2, pp. 1-14, June 2015

This text was slightly tangential, as it didn’t discuss the ideas behind evidence in TEL specifically, but was a good example of an area in which we often find it difficult to find or produce meaningful evidence to support practice. The paper begins by recognising the difficulties in gauging, monitoring and assessing engagement as part of the overall learning experience, despite the fact that engagement is often discussed within HE. Mandernach goes back to the idea of ‘cognitive’ ‘behavioural’ and ‘affective’ criteria for assessing engagement, particularly related to Bowen’s ideas that engagement happens with the leaning process, the object of study, the context of study, and the human condition (or service learning). Interestingly for our current context of building MOOC-based courses, a lot of the suggestions for how these engagement types can be assessed is mainly classroom based – for example the teacher noticing the preparedness of the student at the start of a lesson, or the investment they put into their learning. On a MOOC platform, where there is little meaningful interaction on an individual level between the ‘educator’ and the learner, this clearly becomes more difficult to monitor, and self-reporting becomes increasingly important. In terms of how to go about measuring and assessing engagement, student surveys are discussed – such as the Student Engagement Questionnaire and the Student Course Engagement Questionnaire. The idea of experience sampling – where a selection of students are asked at intervals to rate their engagement at that specific time – is also discussed as a way of measuring overall flow of engagement across a course, which may also be an interesting idea to discuss for our context.

Suggested reading

Horizon Report 2017 – notes from the reading group

Posted by

This time we looked at the NMC Horizon Report 2017 and related documents.

Amy read ‘Redesigning learning spaces’ from the 2017 Horizon report and ‘Makerspaces’ from the ELI ‘Things you should know about’ series. The key points were:

  • Educational institutions are increasingly adopting flexible and inclusive learning design and this is extending to physical environments.
  • Flexible workspaces, with access to peers from other disciplines, experts and equipment, reflect real-world work and create social environments that foster cross-discipline problem-solving.
  • For projects created in flexible environments to be successful, the facilitator allow the learners to shape the experience – much of the value of a makerspace lies in its informal nature, with learning being shaped by the participants rather than the facilitator.
  • There are endless opportunities for collaboration with makerspaces, but investment – both financial and strategic – is essential for successful projects across faculties.

Roger read blended learning designs. This is listed as a short term key trend, driving ed tech adoption in HE for the next 1 to 2 years. It claims that the potential of blended learning is now well understood, that blended approaches are widely used, and that the focus has moved to evaluating impact on learners. It suggests that the most effective uses of blended learning are for contexts where students can do something which they would not otherwise be able to, for example via VR. In spite of the highlighting this change in focus it provides little detailed evidence of impact in the examples mentioned.

Suzi read the sections on Managing Knowledge Obsolescence (which seemed to be around how we in education can make the most of / cope with rapidly changing technology) and Rethinking the Role of Educators. Interesting points were:

  • Educators as guides / curators / facilitators of learning experiences
  • Educators need time, money & space to experiment with new technology (and gather evidence), as well as people with the skills and time to support them
  • HE leaders need to engage with the developing technology landscape and build infrastructure that supports technology transitions

Nothing very new, and I wasn’t sure about the rather business-led examples of how the role of university might change, but still a good provocation for discussion.

Hannah read ‘Achievement Gap’ from the 2017 Horizon Report. It aimed to talk about the disparity in enrolment and performance between student groups, as defined by socioeconomic status, race, ethnicity and gender, but only really tackled some of these issues. The main points were:

  • Overwhelming tuition costs and a ‘one size fits all’ approach of Higher Education is a problem, with more flexible degree plans being needed. The challenge here is catering to all learners’ needs, as well as aligning programmes with deeper learning outcomes and 21st century problems.
  • A degree is becoming increasingly vital for liveable wages across the world, with even manufacturing jobs increasingly requiring post secondary training and skills.
  • There has been a growth in non-traditional students, with online or blended offerings and personalise and adaptive learning strategies being implemented as a retention solution.
  • Some Universities across the world have taken steps towards developing more inclusive offerings: Western Governors University are offering competency based education where students develop concrete skills relating to specific career goals; Norway, Germany and Slovenia offer free post secondary education; under the Obama administration, it was made so that students can secure financial aid 3 months earlier to help them make smarter enrolment decisions; in Scandinavian countries, there is a lot of flexibility in transferring to different subjects, something that isn’t widely accepted in the UK but could help to limit the drop-out rate.
  • Some countries are offering different routes to enrolment in higher education. An example of this is Australia’s Fast Forward programme provides early information to prospective students about alternative pathways to tertiary education, even if they have not performed well in high school. Some of these alternative pathways include online courses to bridge gaps in knowledge, as well as the submission of e-portfolios to demonstrate skills gained through non-formal learning.
  • One thing I thought the article didn’t touch on was the issue of home learning spaces for students. Some students will share rooms and IT equipment, or may not have access to the same facilities as others.

Learning Design Cross Institutional Network #5

Posted by

On Friday 24th February, my colleague Hannah and I ventured up to Northampton University to attend the fifth Learning Design Cross Institutional Network (LDCIN) event. The LDCIN was formed in 2015, with colleagues from a number of institutions across the world taking place in discussions about learning design in education.

The event began with an introduction from Simon Walker, who heads up the Educational Development team at the University of Greenwich. He discussed the future of learning design; the increased interest with the introduction of the TEF, and the impact big data will have on how we design our courses, briefly touching on the report the Open University have recently published on data analytics and learning design (see below for more information).

Participants who had offered to give a ten-minute overview of their work were then invited to deliver. This session started with Natasa Petrovic, from UCL, who discussed her ABC model for learning design – a process my colleague Hannah had successfully used the day before for her Bristol Futures enrichment course! This model is becoming widely adopted as a method to develop course design, with participants only having to attend a 90-minute to 2-hour session for a complete overhaul of their module. More on this method can be found here.

We then had three further presentations from colleagues across the country. Fiona Hale from the University of Edinburgh presented their new model for learning design, which (she admits) very closely resembles the CAIeRO model, created at Northampton. Adele Gordon from Falmouth discussed their development as a learning design team, and how their focus was on employability above anything else – a method that will hopefully be increasingly adopted across the sector.

Finally, Tom Olney and Jitse van Ameijde from the Open University talked about their work on data analytics and retention-satisfaction. They have created a model for designing activities that ensure high retention and success (the ‘ICEBERG’ model) and have discovered some interesting trends. For example, students have higher satisfaction on courses where there are fewer collaborative activities, yet their ‘success’ (in terms of retention, meeting learning outcomes and grades) is lower. Similarly, more collaborative activities meant lower student satisfaction, yet much higher success rates. The report on designing for retention can be found here. This report will have the biggest impact where universities offer online-only courses, where retention is higher than on traditional courses.

These sessions were followed by a tea break (no biscuits provided!) and then a session from Jisc’s Ruth Drysdale, who posed the ‘wicked’ question of how to evidence the impact of TEL – a question that was best answered by Jitse van Ameijide, who simply said ‘You can’t – and shouldn’t.’ The impact of TEL should only be measured by learning success as a whole, rather than how technology has impacted on learning. This focus on successful learning rather than the impact of various technologies was a key theme throughout the morning, and potentially the focus of the next LDCIN meeting.

Next on the agenda was a session from our hosts, which asked us to answer the question ‘How do you solve a problem like Waterside?’ Waterside is a new university campus being built in the heart of Northampton but, unlike a traditional campus, Waterside will have no lecture theatres – teaching will take place online and via small-group or one-to-one tutorials. All course programmes (over 2,000 of them!) have to be redesigned to fit the new teaching style, which also means that the minds of all academics will have to be won over to face this new and radical change. We were tasked with deciding on the best way to motivate staff to engage with this strategy, thinking about five key areas: grassroots campaigns, community, strategic, faculty-level and research-based.

Our group created three models to engage staff with the new teaching strategy. The first, and least desirable, was a ‘top-down’ model, where senior management forced staff to engage with workshops to redesign their courses. However, this would not be a positive change, and would leave staff feeling demotivated and uninspired. The second was a ‘hand-holding’ approach, where a great deal of resource was added to the learning design team to ensure each academic had a bespoke session to redesign their course, with a number of community groups set up to support staff and provide on-hand advice whenever it was needed. The final approach was champion-led, where each faculty had a self-elected ‘champion’ of learning design, who could create a buzz inside their faculty and be available to support staff at short notice. Technology Enhanced Learning has been engaged with most in departments that have appointed learning technologists, and we believe this model is key to success, especially when it comes to changing culture and mindset.

Unfortunately, we had to leave after this session, but judging from the Twitter feed the afternoon was also a success, with a workshop from Edinburgh’s Fiona Hale on mapping learning activities and anther session from the Learning Design team at Northampton on evaluating learning design support.

In an ever-changing sector, it is essential colleagues working towards similar goals come together to share their experiences, methods and thoughts. I was especially inspired by colleagues at Northampton, who are leading the way in terms of a blended approach to education – I can’t wait to see how the challenge of Waterside works out. The LDCIN will be meeting again in the summer to discuss this and other projects taking place across the network. To keep up to date with the latest LDCIN updates, click here.