Miscellany – notes from the reading group

No theme this month – just free choice. Here’s what we read (full notes below):

Naomi read Stakeholders perspectives on graphical tools for visualising student assessment and feedback data.

This paper from the University of Plymouth looks at the development and progression of learning analytics within Higher Education. Luciana Dalla Valle, Julian Stander, Karen Gretsey, John Eales, and Yinghui Wei all contributed.  It covers how four graphical visualisation methods can be used by different stakeholders to interpret assessment and feedback data. The different stakeholders being made up of external examiners, learning developers, industrialists (employers), academics and students.

The paper discusses how there is often difficulty pulling information from assessments and feedback as there can be a lot of data to cover. Having graphic visualisations means information can be shared and disseminated quickly, as there is one focal point to concentrate on. Its mentioned that some can include ‘too much information that can be difficult for teachers to analyse when limited time is available.’ But it is also discussed how it is important then to evaluate the visualisations from the point of view of the different stakeholder who may be using them.

The paper looks at how learning analytics can be seen as a way to optimise learning and allow stakeholders to fully understand and take on board the information that they are provided with. For students it was seen as a way to get the most out of their learning whilst also flagging student’s facing difficulties. The paper also talks about how it brings many benefits to students who are described as the ‘overlooked middle’. Students are able to easily compare their assessments, attainment, and feedback to see their progression. Student’s agreed that the visualisations could assist with study organisation and module choice, and it’s also suggested taking these analytics into account can improve social and cultural skills. For external examiners, analytics was seen as a real step forward in their learning and development. For them it was a quick way to assimilate information and improve their ‘knowledge, skills and judgement in Higher Education Assessment. Having to judge and compare academics standards over a diverse range of assessment types is difficult and visual graphics bring some certain simplicity to this. For learning developers too, using these images and graphics are suggested to help in ‘disseminating good practice.

The paper goes on to explain how it does improve each of the stakeholder’s evaluation of assessment. It goes into a lot of detail of the different visualisations suggested, commenting on their benefits and drawbacks of each set of data, which is worth taking a more detailed look at. It should also be noted that the paper suggested there could be confidential or data protection issues involved with sharing or releasing data such as this as in most cases this certain data is only seen at faculty or school level. Student demoralisation is also mentioned near the end of the paper as being a contributing factor to why these graphics may not always work in the best ways. It finishes by suggesting how it would be interesting to study student’s confidence and self-esteem changes due to assessment data sharing. It an interesting idea that needs to be carefully thought out and analysed to ensure it produces a positive and constructive result for all involved.

Suzanne read: Social media as a student response system: new evidence on learning impact

This paper begins with the assertion that social media is potentially a “powerful tool in higher education” due to its ubiquitous nature in today’s society. However, also recognising that to date the role of social media in education has been a difficult one to pin down. There have been studies showing that it can both enhance learning and teaching and be a severe distraction for students in the classroom.

The study sets out to answer these two questions:

  • What encourages students to actively utilise social media in their learning process?
  • What pedagogical advantages are offered by social media in enhancing students’ learning experiences?

To look at these questions, the researchers used Twitter in a lecture-based setting with 150 accounting undergraduates at an Australian university. In the lectures, Twitter could be used in two ways: as a ‘backchannel’ during the lecture, and as a quiz tool. As a quiz tool, the students used a specific hashtag to Tweet their answers to questions posed by the lecturer in regular intervals during the session, related to the content that had just been covered. These lectures were also recorded, and a proportion of the students only watched the recorded lecture as they were unable to attend in person. Twitter was used for two main reasons. First, the researchers assumed that many students would already be familiar and comfortable with Twitter. Secondly, using Twitter wouldn’t need any additional tools, such as clickers, or software (assuming that students already had it on their devices).

Relatively early on, several drawbacks to using Twitter were noted. There was an immediate (and perhaps not surprising?) tension between the students and lecturers public and private personas on Twitter. Some students weren’t comfortable Tweeting from their own personal accounts, and the researchers actually recommended that lecturers made new accounts to keep their ‘teaching’ life separate from their private lives. There was also a concern about the unpredictability of tapping into students social media, in that the lecturer had no control over what the students wrote, in such a public setting. It also turned out (again, perhaps not unsurprisingly?) that not all students liked or used Twitter, and some were quite against it. Finally, it was noted that once students were on Twitter, it was extremely easy for them to get distracted.

In short, the main findings were that the students on the whole liked and used Twitter for the quiz breaks during the lecture. Students self-reported being more focused, and that the quiz breaks made the lecture more active and helped with their learning as they could check their understanding as they went. This was true for students who actively used Twitter in the lecture, those who didn’t use Twitter but were still in the lecture in person, and those who watched the online recording only. During the study, very few students used Twitter as a backchannel tool, instead preferring to ask questions by raising a hand, or in breaks or after the lecture.

Overall, I feel that this supports the idea that active learning in lectures is enhanced when students are able to interact with the material presented and the lecturer. Breaking up content and allowing students to check their understanding is a well-known and pedagogically sound approach. However, this study doesn’t really provide any benefit in using Twitter, or social media, specifically. The fact that students saw the same benefit regardless of whether they used Twitter to participate, or were just watching the recording (where they paused the recording to answer the questions themselves before continuing to the answers), seems to back this up. In fact, in not using Twitter in any kind of ‘social’ way, and trying to hive off a private space for lecturers and students to interact in such a public environment seems to be missing the point of social media altogether. For me, the initial research questions therefore remain unanswered!

Suzi read Getting things done in large organisations

I ended up with a lot to say about this so I’ve put it in a separate blog post: What can an ed techie learn from the US civil service?. Key points for me were:

  • “Influence without authority as a job description”
  • Having more of a personal agenda, and preparing for what I would say if I got 15 minutes with the VC.
  • Various pieces of good advice for working effectively with other people.

Chrysanthi read Gamification in e-mental health: Development of a digital intervention addressing severe mental illness and metabolic syndrome (2017). This paper talks about the design of a gamified mobile app that aims to help people with severe chronic mental illness in combination with metabolic syndrome. While the target group is quite niche, I love the fact that gamification is used in a context that considers the complexity of the wellbeing domain and the interaction between mental and physical wellbeing. The resulting application, MetaMood, is essentially the digital version of an existing 8-week long paper-based program with the addition of game elements. The gamification aims to increase participation, motivation and engagement with the intervention. It is designed to be used as part of a blended care approach, combined with face to face consultations. The game elements include a storyline, a helpful character, achievements, coins and a chat room, for the social element. Gamification techniques (tutorial, quest, action) were mapped to traditional techniques (lesson, task, question) to create the app.

The specific needs of the target group needed the contributions of an interdisciplinary team, as well as relevant game features; eg the chat room includes not only profanity filter, but also automatic intervention when keywords like suicide are used (informing the player of various resources available to help in these cases). Scenarios, situations and names were evaluated for their potential to trigger patients, and changes were made accordingly; eg the religious sounding name of a village was changed, as it could have triggered delusions.

The 4 clinicians that reviewed the app said it can proceed to clinical trial with no requirement for further revision. Most would recommend it to at least some of their clients. The content was viewed as acceptable and targeted by most, the app interesting, fun & easy to use. I wish there had been results of the clinical trial, but it looks like this is the next step.

Roger read “Analytics for learning design: A layered framework and tools”, an article from the British Journal of Educational Technology.

This paper explores the role analytics can play in supporting learning design. The authors propose a framework called the “Analytics layers for learning design (AL4LD)”, which has three layers: learner, design and community analytics.

Types of learner metrics include engagement, progression and student satisfaction while experiencing a learning design. Examples of data sources are VLEs or other digital learning environments, student information systems, sensor based information collected from physical spaces, and “Institutional student information and evaluation (assessment and satisfaction) systems”. The article doesn’t go into detail about the latter, for example to explore and address the generic nature of many evaluations eg NSS, which is unlikely to provide meaningful data about impact of specific learning designs.

Design metrics capture design decisions prior to the implementation of the design. Examples of data include learning outcomes, activities and tools used to support these. The article emphasises that “Data collection in this layer is greatly simplified when the design tools are software systems”. I would go further and suggest that it is pretty much impossible to collect this data without such a system, not least as it requires practitioners to be explicit about these decisions, which otherwise often remain hidden.

Community metrics are around “patterns of design activity within a community of teachers and related stakeholders”, which could be within or across institutions. Examples of data include types of learning design tools used and popular designs in certain contexts. These may be shared in virtual or physical spaces to raise awareness and encourage reflection.

The layers inter-connect eg learning analytics could contribute to community analytics by providing evidence for the effectiveness of a design. The article goes on to describe four examples. I was particularly interested in the third third one which describes the “experimental Educational Design Studio” from the University of Technology Sydney. It is a physical space where teachers can go to explore and make designs, ie it also addresses the community analytics layer in a shared physical space.

This was an interesting read, but in general I think the main challenge is collection of data in the design and community aspects. For example Diana Laurillard has been working on systems to do this for many years, but there seems to have been little traction. eg The learning design support environment  and the Pedagogical Patterns Collector.

Amy read: Addressing cheating in e-assessment using student authentication and authorship checking systems: teachers’ perspectives. Student authentication and authorship systems are becoming increasingly well-used in universities across the world, with many believing that cheating is on the rise across a range of assessments. This paper looks at two universities (University A in Turkey and University B in Bulgaria) who have implemented the TeSLA system (an Adaptive Trust-based eAssessment System for Learning). The paper doesn’t review the effectiveness of the TeSLA system, but rather the views of the teachers on whether the system will affect the amount of cheating taking place.

The research’s main aim is to explore the basic rationale for the use of student authentication and authorship systems, and within that, to look at four specific issues:

  1. How concerned are teaching about the issue of cheating and plagiarism in their courses?
  2. What cheating and plagiarism have teachers observed?
  3. If eAssessment were introduced in their courses, what impact do the teaching think it might have on cheating and plagiarism?
  4. How do teachers view the possible use of student authentication and authorship checking systems, and how well would such systems fit with their present and potential future assessment practises?

Data was collected across three different teaching environments: face-to-face teaching, distance learning and blended learning. Data was collected via questionnaires and interviews with staff and students.

The findings, for the most part, were not hugely surprising: the main type of cheating that took place at both universities was plagiarism, followed by ghost-writing (or the use of ‘essay mills’). These were the most common methods of cheating in both exam rooms and online. The difference between the reasons staff believed students cheated and why students cheated varied widely too. Both teachers and students believed that:

  • Students wanted to get higher grades
  • The internet encourages cheating and plagiarism, and makes it easy to do so
  • There would not be any serious consequences if cheating and plagiarism was discovered

However, teachers also believed that students were lazy and wanted to take the easy way out, whereas students blamed pressure from their parents and the fact they had jobs as well as studying for reasons.

Overall, staff were concerned with cheating, and believed it was a widespread and serious problem. The most common and widespread problem was plagiarism and ghost writing, followed by copying and communicating with others during assessments. When asked about ways of preventing cheating and plagiarism, teaching were most likely to recommend changes to educational approaches, followed by assessment design, technology and sanctions. Teachers across the different teaching environments (face-to-face, blended and distance learning) were all concerned with the increase in cheating that may take place with online/ eAssessments. This was especially the case for staff who taught on distance learning courses, where students currently take an exam under strict conditions. Finally, all staff believed that the use of student authentication and authorship tools enabled greater flexibility in access for those who found it difficult to travel, as well as in forms of assessment. However, staff believed that cheating could still take place regardless of these systems, but that technology could be used in conjunction with other tools and methods to reduce cheating in online assessments.

Wellbeing – notes from the reading group

Roger read: Curriculum design for wellbeing.

This is part of an online professional development course for academics, produced by a project run by a number of Australian Universities co-ordinated by the University of Melbourne. It aims  to build the capacity of University educators to design curriculum and create teaching and learning environments that enhance student mental wellbeing. There are 5 modules: on student wellbeing, curriculum design, teaching strategies, difficult conversations and your wellbeing.

I focussed on module 2 which is on curriculum design. It starts by stressing the importance of students, through the curriculum, experiencing autonomous motivation, a sense of belonging, positive relationships,  feelings of autonomy and competence (M-BRAC). All of these are aspects of good practice in curriculum design.

It goes on to consider how elements of curriculum design support student mental wellbeing, covering alignment, organisation and sequencing of content, engaging learning activities and a focus on assessment for learning.

For example, aligning ILOS with assessment and learning activities helps student autonomy as they understand how what they are doing contributes to their goals and they develop their knowledge and skills, including self-regulation, to achieve the ILOS. Assessment for learning plays a key role here. Clear organisation and sequencing of content contribute to effective learning. Both alignment and structure help to build students’ sense of competence.  Engaging and meaningful learning activities can increase student motivation and encourage peer interaction, which can contribute to building relationships and a sense of belonging.

It suggests that when reviewing the curriculum one should ask:

  • How will the curriculum be experienced by my (diverse) students eg international students, mature students, “first in family” students?
  • Will the curriculum foster or thwart experiences of M-BRAC

The module then has a number of FAQs you are asked to consider, with suggested answers. These were really useful as they tease out some of the complexities, for example “Is setting group assignments in the first year a good way of helping students develop positive peer relationships, and feel a sense of connectedness or belonging?”  Here they recognise that if not well-designed or if students are not supported to develop group work skills it can have a negative impact.

The module ends with a set of case studies illustrating how curricula have been re-designed to better support M-BRAC.

Amy read: Approach your laptop mindfully to avoid digital overload.

This was a short article that recognised the ever-increasing belief that we are being constantly bombarded with masses of new information which, in turn, means that many are suffering with stress-related diseases, anxiety and depression. The reliance on digital devices to provide constant streams of information in the form of news articles, social media feeds and messages mean that without these devices we feel lost without them. A full digital detox is suggested at the beginning of this article, though this may be a short-term solution and often an impractical one.

The authors of this article suggest introducing the practice of mindfulness into our lives to combat this. They describe mindfulness as ‘a moment-to-moment attention to present experience with a stance of open curiosity’. Mindfulness has been studied extensively by the medical community and has shown to help with stress, anxiety and depression in individuals. One can ‘reprogramme’ their mind to deal with stresses more easily by training it to be more present. The authors suggest two key ways they suggest to introduce mindfulness into our use of digital devices to reduce the pressures they can put on us.

One of the methods they suggest is ‘mindful emailing’, which includes practices such as taking three breaths before responding to a stressful email and also considering the psychological effect that the email will have to the recipients.

The second method they suggest is the mindful use of social media, citing ‘checking our intentions before uploading a feed (post?), being authentic in our communications and  choosing the time we spend on social media, rather than falling into it.’

If you haven’t tried mindfulness before take a look at these tips and short audio meditations.

Chrysanthi read: Designing a product with mental health issues in mind,

This article – true to its title – talks about including technological features that aim to help the vulnerable users. While the examples given are taken from a banking application context, the suggestions can be applied to other contexts. More specifically, the article mentions positive friction and pre-emptive action. Positive friction goes against developers’ usual desire to make everything easier and faster and aims to put some necessary obstacles in the way instead, for users that need it. The example used is allowing certain users with somewhat “impulsive” behaviour to check their recent purchases and confirm that they indeed want them. This would help eg bipolar disorder sufferers, who overspend in their manic phase, often at night, and slip into depression in the morning because of their irreversible mistake. In the specific app, this is still a speculative feature.  Pre-emptive action aims to prevent trouble by anticipating certain events and acting on them, eg perceiving a halt in income and sending a well timed notification to start a conversation so the person doesn’t end in debt (and therefore create more stress for themselves). Also, allowing vulnerable customers to choose their preferred time and form of communication (eg phone might be anxiety inducing or email might seem complex).

In an education context, positive friction could be relevant in cases where students are repeatedly doing things they no longer need to do. This would help when – under the illusion they are still learning – students are focusing on redoing exercises they already know how to do – which might help them feel accomplished but doesn’t add value from some point on – or on consuming more and more content, even when they haven’t actually digested what they have learned so far. It isn’t very clear how this could be applied during exam period, though. Pre-emptive action is perhaps easier to translate in an educational context. Any action (or inaction) that is either outside the student’s usual pattern or outside a successful pattern, might be a conversation starting point, or a trigger for suggestions for alternative ways to handle their studies. Also, allowing them different options to learn and communicate with their professors and peers.

Chrysanthi also read: E-mail at work: A cause for concern? The implications of the new communication technologies for health, wellbeing and productivity at work.

This paper explores the potential negative implications of using email at work. The email features they consider as potentially problematic are: speed, multiple addressability, recordability, processing & routing. Essentially, email as a message that can be instantly transmitted to several people at once, automatically stored, easily altered and different versions of it sent to various people, not all of which necessarily visible on the recipient list. According to the authors, emails may increase stress by increasing workload and interruptions, adding difficulty to interpretation of the message and tone, increasing misunderstandings and groupthink, reducing helpful argumentation and social support, increasing isolation and surveillance – which increases discontent, offering a new ground for bullying and harassment, or hindering the processing of negative feelings. Having established these potential negative implications, the authors point out that more research is needed to understand the optimal ways to use email at work, for effective communication and humane workplaces.

Naomi read: Did a five-day camp without digital devices really boost children’s interpersonal skills?

This article was about a brief study led by Yalda Uhls in Southern California. It studied two groups of pupils ‘who on average spent 4.5 hours a day ‘texting watching TV, and video gaming’. Half of the children were sent on a five-day educational camp in the country side where all technical devices were banned. The other half stayed at school as usual. Emotional and psychological tests were carried out on the students before and after the 5 days were completed.

There was a small amount of evidence to suggest the children who had spent time away from devices improved psychologically over the five days. However, because there were several small problems with the study no firm answers can be taken from it. Its suggested the children who went away for the five days only looked like they improved on the tests because they started at a lower level then the children who stayed at school. It was also suggested that the children who stayed at school tests deteriorated because they were tired from doing a week’s work.

As it suggests in the article, the results of this study weren’t particularly hard hitting but it does raise the question of how much the younger generations are using their devices throughout the day. Uhls does admit in the article that there were shortcomings to the study, but they suggest that these findings relate to the ‘wider context of technology fears’ and hope their paper will be ‘a call to action for research that thoroughly and systematically examines the effects of digital media on children’s social development.’ Although the study needed a more comprehensive approach the ideas behind it are interesting and relate to several issues that we see in everyday life – is it good for us to spend so much time on our devices, or is it integral to how we live now?

Suzi read: Learning in the age of digital distraction

This interview with Adam Gazzaley, a neurologist, is a plug for a book called The Distracted Mind in which he and and research psychologist Larry D Rosen talk about the way our brains are wired influences how we use technology.

They suggest that information-foraging is a development of our evolved food-foraging mechanisms, and so is to some extent driven by our very basic drive to survive. Because of this it is hard to prevent it from distracting us from our ability to set and pursue higher-level goals.

Information-foraging doesn’t just impact on people’s ability to focus, it can cause anxiety and stress, and affect mood.

Suggestions for possible ways to combat this include:

  • accepting that we need to (re-)learn to focus (they are also developing brain-training video games)
  • using play in education (but this was only very briefly mentioned I wasn’t clear if this was playfulness or gamification)
  • physical exercise
  • mindfulness
  • sitting down to dinner as a family, or otherwise building in device-free interaction time

Suggested reading

Playful learning – notes from the reading group

Suzi read Playful learning: tools, techniques, and tactics by Nicola Whitton

This is a useful scene-setting article, suggesting ways of framing discussions on playful learning and pointing the way to unexplored territory suitable for future research.

There are three ideas about how to talk about playful learning:

  • The magic circle – a socially constructed space in which play can happen
  • A mapping of aspects of games onto playful learning: surface structures of playful learning <> mechanics of games, deep structures <> activities of play, implicit structure <> philosophy of playfulness
  • Tools / techniques / tactics of playful learning: objects artefacts & technologies / approaches / mechanics and attributes – these could serve as prompts for getting playfulness into teaching

Whitton suggests three characteristics of the magic circle which make it pedagogically useful: “the positive construction of failure; support for learners to immerse themselves in the spirit of play; and the development of intrinsic motivation to engage with learning activities.” In playful activities, failure will be framed positively, participants suspend disbelief which can encourage creativity, and participation is voluntary so there is intrinsic motivation (the difficulty of this last in particular in a formal education setting is acknowledged).

There’s a lot of acknowledgement that playfulness may not be an easy fit in higher education. Obstacles include: the inescapably of real world power relationships, confusing gamification with true playfulness, the need for things to be mandatory and assessed, existing attitudes to failure, prejudice about play being for children, lack of time, confidence and social capital.

I wasn’t certain about the point about play being a privilege. While certain types of play might attract a relatively slender demographic (escape rooms, real world games) and it’s important not to assume that everyone would want to engage in these, adults seem to play to learn in a range of contexts. I thought about the kinds of spaces where you would see playful learning: cooking, Karaoke, parkour, getting dressed up, new social media platforms (when FB started and everyone was poking each other and and biting each other and throwing bananas, hashtags came from playing with the way Twitter worked), and of other adult pursuits. There is playfulness in higher education too, although it’s often not explicitly described as such. Maybe there is a danger of rarefying play and almost making it by definition a domain for geeks alone, not recognising play that has not been made explicit.

This got me thinking about why we play, and why we might want to play in HE, and about one of my favourite quotes:

“The things we want are transformative, and we don’t know or only think we know what is on the other side of that transformation. Love, wisdom, grace, inspiration – how do you go about finding these things that are in some ways about extending the boundaries of the self into unknown territory, about becoming someone else?”

— Rebecca Solnit, A field guide to getting lost

This sums up a lot of what university could and should be. Playfulness has to have a very key role in that: place to play with possible selves, both academic and social.

Chrysanthi read Gamifying education: what is known, what is believed and what remains uncertain: a critical review by Christo Dichev and Darina Dicheva.

This is a review aiming to find what is known about gamification in educational contexts based on empirical evidence, rather than beliefs. The authors seem to find that much more is believed or uncertain, rather than known. For example, their main findings are that a. gamification has started being used at a pace much faster than researchers pace at understanding how it works, b. there is very little knowledge about how to effectively apply it in specific contexts and c. there is not enough evidence about its benefit long term.

While the understanding of how to engage, motivate and aid learning through gamification is inadequate, researchers are still praising the practice, thus inflating expectations about its effectiveness. The frequent use of performance-centric game elements like points, levels, badges and leaderboards is noteworthy; in absence of justification from the researchers implementing them, the authors hypothesize that this happens because they are similar to traditional classroom practices and easy to implement. But this leaves other major game elements out; authors note – among others – role play, narrative, choice, low risk failure. These tiny elements are then expected to affect broad concepts like motivation, with researchers often concluding that that is the case, without enough evidence to claim it is so.

This implies a somewhat blind application of the easiest-to-implement elements of gamification, with the belief that it will be enough to motivate students to perform better. But how are points different to marks and levels different to grades and chapters?

Perhaps gamification can’t be a canned, one-size-fits-all-learning-contexts solutions. Perhaps researchers and practitioners need to put in time and at least a bare minimum of imagination to create something that will be engaging enough for students, for the evidence supporting it to not be stamped “inconclusive” when under scrutiny.

David read Playful learning in Higher Education: developing a signature pedagogy by Rikke Toft Nørgård, Claus Toft-Nielsen & Nicola Whitton (2017)

This paper starts off with a bit of a rant about the commercialisation of higher education and the focus on metrics to measure performance and how this creates an assessment-driven environment focused on goal-oriented behaviours characterised by avoidance of risk and fear of failure. The authors see recent gameful approaches in higher education as a response to this but warn that while gamification may increase motivation, games often focus on extrinsic motivational drivers and the results may be short-lived. They also cite research which points to issues around perceived appropriateness and students manipulating points-based incentive systems (and my colleagues and I have encountered examples of this in out teaching).

In contrast to gamification, they regard playful learning as something which encourages intrinsic and longer-term motivation by offering the chance to explore and experiment without fear of being judged for failure and therefore being able to learn from it. They use the ideas of the ‘magic circle’ and ‘lusory attitude’ to describe the environment in which this can occur. The concept of the magic circle is used a lot in gaming and is a metaphor for the ‘space’ we enter into when we fully engage with a game, accepting the different norms and codes of practice (or actively constructing them with other ‘players’). This can refer to physical (e.g. sports) or virtual/imaginary (e.g. computer games) or a combination of both (e.g. a child’s tea party). For this to work, we need to assume an ‘lusory attitude’. This gives participants a shared mindset in which they are free to play, experiment and fail in a safe place.

The Magic Circle – How Games Transport Us to New Worlds

The authors then turn to the question of how to implement such an approach. Using the results of two studies about what students report (a) makes their learning enjoyable and (b) disengages them, they develop a ‘signature pedagogy’ for playful learning in higher education. The notion of ‘signature pedagogy’ they assume is split into three levels:

  • The foundation is formed by Implicit (playful) structures, which are the necessary assumptions and attitudes (values, habits, ethics)
    • Lusory attitude
    • Democratic values and openness
    • Acceptance of risk-taking and failure
    • Intrinsic motivation
  • Deep (play) structures represent the nature of the activities which the implicit structures facilitate
    • Active and physical engagement
    • Collaboration with diverse others
    • Imagining possibilities
    • Novelty and surprise
  • Surface (game) structures are the ‘mechanics’ of an activity, including the materials, tools and actions involved
    • Ease of entry and explicit progression
    • Appropriate and flexible levels of challenge
    • Engaging game mechanics
    • Physical or digital artefacts

The authors see the implicit (playful) structures as the necessary starting point for their ‘signature pedagogy’ but do not say how students get to this point. Indeed they acknowledge the inherent paradox in their model:

“…for many students to view learning as valuable then it must be valued by the system (assessed), yet it is simultaneously this assessment that makes learning stressful and undermines the creation of a safe and comfortable environment.”

For me, then, this article leaves three interrelated questions to be discussed:

  1. For playful learning to be successful, do students need to have the implicit structures already in place or can students acquire these through the playful activity itself?
  2. If these implicit structures are prerequisite, how do we get students to acquire them?
  3. As this involves a change in students’ attitudes which the authors argue are reinforced by the current assessment-driven environment, does this pedagogical approach have any chance of success without change at the programme or institutional level?

Suzanne read Unhappy families: using tabletop games as a technology to understand play in education by John Lean, Sam Illingworth, Paul Wake, published in the ALT Journal special issue

In this article, the authors decide to take a step back when considering the ‘future’ of digital technologies in relation to playful learning by considering traditional table top games as a form of technology. They aimed to better understand the affordances of digital game tools by looking at table top games as an analogue, in order to reflect critically on the pedagogical uses of games and playful learning. Their hypothesis was that table top games (see the article for a full definition of how they classify a game as ‘table top’), are successful because: 1) they provide an immediate and accessible shared space, which is also social; 2) this space and the game are both easily modified by players and educators; and 3) they provide a tactile, sensory experience. So, in essence, that they are social, modifiable and tactile, which are all things that could be transferred into digital games in education, but which are often overlooked.

To explore this hypothesis, they used a specific game, that of  ‘Gloom‘, which was played by participants at the 2017 ALT Playful Learning Conference. In relation to the first hypothesis, they found that the game encouraged a lot of social interaction. Firstly, the game encourages players to talk about their recent lived experiences as a means of deciding who gets to go first (ie, who has had the worst day thus far). Additionally, there is a storytelling element of the game, which also encourages an empathetic interaction between the game and the players, as well as between players.

Regarding how modifiable the game is, the players found it was easy to change and adapt the game, even during play. They also explored the ways of playing around with and stretching the rules, to create different rules or games within the game play. The authors note that this is often not as easily achieved in digital games, where rules can be more fixed and more difficult to circumvent. Thirdly, the players did undoubtedly find the game tactile, as the cards provide a physical element, further enhanced by the way the cards are played. The cards themselves have transparent elements, so as you stack cards you create different versions of them, allowing for the storytelling element.

In conclusion, the authors used this game play experience to revisit some preconceptions about what ‘play’ or ‘playfulness’ is in a game context. They felt that the ‘true’ play seemed to happen when the players had modified the rules to the point where the game itself was almost no longer required. The players were exploring and testing their new game playfully, in the way that they were interacting with each other and the environment. In terms of education, they felt that this playfulness had great potential for learning. The process of negotiating the play, and working out how to play with others who might have different ideas to you (for example, either wanting to stick to the rules or wanting to break them) is potentially a powerful social learning opportunity.

However, they also noted that this very character of playful learning – that it is negotiated and created by the context and participants – makes it extremely difficult to categorise or understand pedagogically. If we need to allow for such variety of outcomes in playful learning, it can be difficult to work out how we can situate it within other educational structures, like lesson plans or learning objectives.

Suggested reading

Digital and physical spaces – notes from the reading group

Amy read – Institutional to Individual: realising the postdigital VLE? by Lawrie Phipps.

Lawrie starts this article by quoting himself – ‘Digital is about people’. He believes that learning is effective when we are connection in conversations and in groups – this is been proven many times over – but that these conversations should not be confined. The ‘confinement’ he talks of is the attempt by unnamed institutions to restrict their teaching staff by controlling the access and provision of alternative tools, which, Lawrie argues, don’t often align with their everyday activities. He mentions two projects that are taking places at universities – the Personalised User Learning & Social Environments (PULSE) project at Leeds Beckett (difficult to find anything about this online) and the Aula team, who have created a ‘conversational layer’ to run alongside a VLE and provide an ‘ecosystem for a range of other tools’.

The article moves on to discuss the emerging trend of disaggregation as being an indicator of ‘post digital academic practice’… I’d be interest to know what he means by this but the article does not shed any light on this. If I were to guess at its meaning I would think that the digital is becoming so integrated into our lives that it can no longer be considered a practice – it is seamless, and therefore doesn’t need to be recognised. He reminds us to be mindful of the other emerging themes of digital spaces; control, surveillance and ‘weaponised’ metrics used by corporate bodies and the use of algorithms to control our feeds.

Lawrie finishes by letting us know that ‘the report’ (I presume the ‘Next Generation Digital Learning Environments’, mentioned earlier in the article) is coming together nicely, and urges the reader get in touch if they have any relevant cases of disaggregation for practical purposes.

 

Chrysanthi read Digital sanctuary and anonymity on campus by Sian Bayne.

The article is trying to make a case for anonymity in online social exchange in the context of higher education.

The author points out that as part of the point of higher education is to help students own and defend their knowledge, anonymity is not usual, barring exceptions like peer review. But this works better for those with privileges than those without and it doesn’t work for every topic that a student would be interested in. In their view, anonymity offers 1. social value and 2. a way to resist digital surveillance. By looking at the use of an anonymous social media app called Yik Yak – which was popular for 2-3 years, but then removed the anonymity and then closed – they realised that it was a tool often used to facilitate anonymous peer support, which was very helpful to students concerning topics like social difficulties or isolation, relationships, health (sexual and emotional) or teaching-related issues.

Anonymity also serves to resist the ubiquitous surveillance that occurs in large part through social media, that record everything individuals do and like for their financial benefit. But there can be online social networks where students don’t need to hand over their data to be able to use them.

They argue that the absence of an app like this reduces students’ opportunities to a support group and that the counter argument usually put forward – that anonymous spaces facilitate abuse – is weak, considering abuse can and does happen everywhere, including non anonymous social media like Facebook. They are concerned about where the supportive conversations that people would previously have anonymously are happening now, for topics like mental health or relationships.  Overall, they believe universities need such anonymous spaces and should figure out how to implement them balancing data, trust & safety.

I think the author makes good points. Regarding where the conversations are happening now, I am assuming

1. other anonymous but not higher education specific spaces, such as reddit, which means people will get support, although from a broader population that is not coming from the same context, with all the challenges this implies.

2. non anonymous spaces, like Facebook, which means people are essentially broadcasting their issues on platforms that a. may use this information for their benefit and the student’s detriment, b. store and display the data with the user’s name for a long time, with no guarantees for who can/ cannot see it. This makes abuse easier, as well as enabling people looking up the individual (e.g. future employers) to see information they should otherwise not have access to.

3. they are not getting support, which could lead to isolation.

Overall, I do see the point of universities implementing anonymous digital spaces for their students.

 

Naomi read: The SCALE-UP Project: A Student-Centred Active Learning Environment for Undergraduate Programmes by Robert J. Beichner.

The author starts by describing these scale-up areas as places where ‘student teams are given interesting things to investigate, while their instructor roams.’ Although this is one of the short areas we hear about how the actual space is designed to improve learning and collaboration.The purpose of these teaching spaces is to encourage discussion between student’s and their peers. By working in small groups on separate tables within the classroom student’s can work on separate activities and use a shared laptop or whiteboard to research or make note of their findings. Thay can then discuss with other groups.

The main point of the paper revolves around the idea of social interaction between students and their teachers being the ‘active ingredient’ in making this approach to teaching work. Beichner talks about how student’s in these classes gain a better conceptual understanding then the student’s taking traditional lecture-based classes. Studies saw a high rise in student’s confidence, their problem-solving skills, as well as teamanship and communication. There is some concern about whether this approach is meaning less content is being delivered to the students, but Beichner argues the content is being developed and created by the student’s themselves.

Discussion led learning is always going to be popular, but we need to think about the physical space too and whether it is needed or not. The size of these classes needs to be considered too – what can be classed as too big? Beichner’s study was interesting, but not surprising, and it would have been good to know how the design or the space and tables aiding the learning too.

 

Suzi read The Educause NDingle and an API of one’s own by Michale Feldstein (which is a rebuttal to a rebuttal of the Educause Next Generation Digital Learning Environment report)

This is a clear and interesting artical discussing where learning management systems (LMSs) could/should go – as digital spaces for learning. The perspective on this is relatively technical, discussing the underlying architecture of the system but the key ideas are very approachable:

LMSs could move from being one application that tries to do everything, to being more like an oporating system on a mobile phone – hosting apps and managing the ways they can communicate with each other

Lego is also used as a metaphor for this more adaptable LMS, but Feldstein discusses the tension between having fairly generic blocks that don’t build anything in particular but allow you to be very creative (Lego from my childhood), and having sets which are intended to build a particular thing but which are then less adaptable (more typical of modern Lego). I found this a harder idea to apply, though I can appreciate that just because something comes in blocks and can be taken apart, doesn’t mean it is genuinely flexible and adaptable.

Personal ownership of data is discussed – the idea of students even hosting their own work and having a personal API via which they grant the institution’s LMS (and hence teachers) access to read and provide feedback on their work (“an API of one’s own”). This seems to me an attractive idea, in a purist origins-of-the-web way. People have suggested similar approaches in various domains, social media in particular, and I don’t know of any that have worked.

 

Suzanne read Semiotic Social Spaces and Affinity Spaces From The Age of Mythology to Today’s Schools by James Paul Gee. The premise of this text is to reconsider the idea of a community of practice, to think about it as related to the space in which people interact (and in what way), rather than membership of the community (particularly membership given to people by others, or through arbitrary groupings). Gee argues that thinking about community in this way is more useful, as membership means so many different things to different people, so trying to decide who is ‘in’ or ‘out’ of a group is problematic. He explains his ‘alternative’ to thinking about a ‘community of practice’ as an ‘affinity space’ in quite a lot of detail, using the analogy of a real-time computer game as an example, which here I won’t try to explain fully. However, some key ideas around what makes an ‘affinity space’ are that there needs to be some kind of content, generated by the community around a common endeavour. The people who interact with this content do so with an agreed set of ‘signs‘ with their own particular ‘grammar‘ or rules. This grammar can be internal (signs decided on within the group), or external (eg the way that people’s beliefs and identities are formed around these signs, and their relationship with them), and the external grammar can influence the internal grammar. Another interesting aspect is the idea of portals. An affinity space will have a number of ways that people can interact with it. To take the game example, the game itself could be a portal, but so could a website about game strategy, or a forum discussing the game. Importantly, the content, signs and grammar of the space can be changed by those interacting through those portals, so the content is not fixed. The final points are that people interacting in the space are both ‘expert’ and ‘novice’, and both intensive and extensive knowledge is valued. Individuals with specific skills or who a great amount of knowledge about a specific thing are as valued by the space as those who work to build a more distributed community of knowledge, and there are many different ways people can participate. Gee’s text presents quite an in depth concept, which seems quite theoretical. However, thinking about something like the Bristol Futures themes (Global Citizenship, Innovation and Enterprise or Sustainable Futures), we discussed how it might be applied, and how it might help us to think about things like reward and recognition, or success measures, in a very different way.

Suggested reading

Programme level assessment – notes from the reading group

Suzi read Handbook from UMass – PROGRAM-Based Review and Assessment: Tools and Techniques for Program Improvement

A really clear and useful guide to the process of setting up programme level assessment. The guide contains well-pitched explanations, along with activities, worksheets, and concrete examples for each stage of the process: understanding assessment, defining programme goals and objectives, designing the assessment, selecting assessment methods, analysing and reporting. Even the “how to use this guide” section struck me as helpful, which is unheard of.

The proviso is that your understanding of what assessment is for would need to align with theirs, or you would need to be mindful of where it doesn’t. As others do, they talk about assessment to improve, to inform, and to prove and they do also nod to external requirements (QAA, TEF, etc in our context). However, their focus is on assessment as part of the project of continual (action) research into, and improvement of, education in the context of the department’s broader mission. This is a more holistic approach that might bring in a wide range of measures including student evaluations of the units, data about attendance, and input from employers. I like this focus but it might not be what people are expecting.

During the group we discussed the idea of combining some of the ideas from this, and the approach Suzanne read about (see below). A central team would collaborate with academic staff within the department in what is essentially a research project, supporting conversations between staff on a project, bringing in the student voice and leaving them with the evidence-base and tools to drive conversations about education in their context – empowering staff.

(Side note – on reflection I’m pretty sure this is the reason this particular reading appealed to me.)

Chrysanthi read Characterising programme‐level assessment environments that support learning by Graham Gibbs & Harriet Dunbar‐Goddet.

The authors propose a methodology for characterising programme-level assessment environments, so that they can later be studied along with the students’ learning.

In a nutshell, they selected 9 characteristics that are considered important either in quality assessment or for learning (e.g. variety and volume of assessment). Some of these were similar to the TESTA methodology Suzanne described. They selected 3 institutions that were different in terms of structure (e.g. more or less fixed, with less or more choice of modules, traditional or variety in assessment methods etc. They selected 3 subject areas, the same in all institutions. They then collected data about the assessment in these and coded each characteristic so there would be 3 categories: low, medium, high. Finally, they classified each characteristic for each subject in each institution according to this coding. They found that the characteristics were generally consistent within institution, showing a cultural approach to assessment, rather than a subject- related one. They also identified patterns, e.g. that assessment aligned well with goals correlates with variety in methods. While the methodology is useful, their coding of characteristics as low-medium-high is arbitrary and their sample small, so the stated quantities in the 3 categories are not necessarily good guidelines.

Chrysanthi also watched a video from the same author Suzanne read about: Tansy Jessop: Improving student learning from assessment and feedback – a programme-level view (video, 30 mins).

There was a comparison of 2 contradictory case studies, 1 that seemed like a “model” assessment environment, but where the students did not put in much effort and were unclear about the goals and unhappy, and 1 that seemed problematic in terms of assessment but where students knew the goals and were satisfied. The conclusion was that rather than having a teacher plan a course perfectly and transmit large amounts of feedback to each student, it might be worth encouraging students to construct it themselves in a “messy” context, expanding constructivism to assessment as well.

Additionally, as students are more motivated by summative assessment, have a staged assessment where students are required to complete some formative assessment that feeds into their summative assessment. Amy & Chris suggested that this has already started happening in some courses.

Finally, the speaker noted that making the formative assessment publicly available, such as in blog posts, motivates the students, that it would be better if assessment encouraged working steadily throughout the term, rather than mainly at peak times around examinations and that feedback is important for goal clarity and overall satisfaction.

Both paper and video emphasised the wide variety in assessment characteristics between different programs. In the paper’s authors’ words, “one wonders what the variation might have been in the absence of a quality assurance system”.

The discussion went into the marking system and the importance students give to the numbers, even when they are often irrelevant to the big picture and their future job.

Amy summarised a summary she had created after attending a Chris Rust Assessment Workshop at the University. The workshop focussed on the benefits of programme-level assessment, looking at the current problems with assessment in universities and offering practical solutions and advice on creating programme-level assessments. The workshop started by looking at curriculum sequencing – it’s benefits and drawbacks, and illustrated this with examples where it had been successful.

Chris then discussed ‘capstone and cornerstone’ modules as a model for programme-level assessment, and explain where it had been a success in other universities. He discussed the pseudo-currency of marks and looked at ways we can alter our marking systems to improve student’s attitude to assessments and feedback. He ended the session by looking at the ways you can engage students with feedback effectively, and workshop attendees shared their advice with colleagues on how they engage their students with feedback. You can find the summary here.

Suzanne read Transforming assessment through the TESTA project by Tansy Jessop (who will be the next Education Excellence speaker) and Yaz El Hakim, which briefly describes the TESTA project, the methods they use and the outcomes they have noted so far. There are also references within the text to more detailed publications on specific areas of the methods, or on specific outcomes, if you want to find out more detail.

In brief, the TESTA project started in 2009, and has now expanded to 20 universities in the UK, Australia and the Netherlands, with 70 programmes having used TESTA to develop their assessment. The article begins by giving a pretty comprehensive overview of the reasons why programme assessment is so high on the agenda, including the recognition that assessment affects student study behaviours, and that assessment demonstrates what we value in learning, so we should make sure it really is focused on the right things. There was also a discussion about how the ‘modularisation’ of university study has left us with a situation of very separated assessments, which make it difficult to really see the impact of assessment practices across a programme, particularly for students who take a slower approach to learning. Ultimately the TESTA project is about getting people to talk about their practices on a ‘big picture’ level, identity areas which could be improved, and then work from a base of evidence to make those improvements. There is a detailed system of auditing current courses, including sampling, interviews with teaching s and programme directors, student questionnaires, and focus groups. the information from this is then used as a catalyst for discussion and change, which will manifest differently in each different programme and context.

The final paragraph of the report sums it up quite well: “The value of TESTA seems to lie in getting whole programmes to discuss evidence and work together at addressing assessment and feedback issues as a team, with their disciplinary knowledge, experience of students, and understanding of resource implications. The voice of students, corroborated by statistics and programme evidence has a powerful and particular effect on programme teams, especially as discussion usually raises awareness of how students learn best.”

Suggested reading

Role and future of universities – notes from the reading group

Maggie read Artificial intelligence will transform universities. Here’s how – World Economic Forum
The article presents the idea that Universities have created a need to innovate and evolve to meet the changing needs caused by the upsurge in AI. Already, the marking of student papers is becoming a thing of the past as AI is able to assess and even ” flag up” issues with ethics.Students are less able to distinguish between teacher marking and that of a “bot”. Teaching is additionally being impacted as students are able to undertake statistics courses using AI, massively reducing learning (and human teacher) time, with apparently equal learning and application outcomes. The author argues that Universities will need to up their game regarding employability and indeed attractive employment (remuneration). The paper is an easy-to-read item and clearly outlines the range of benefits and subsequent issues in relation to AI. All pertinent.

Suzi read three short opinion pieces: What are universities for and how do they work? by Keith Devlin, Everything must be measured: how mimicking business taints universities by Jonathan Wolff, and Universities are broke. So let’s cut the pointless admin and get back to teaching by André Spicer.

Devlin focused largely on the role of research within maths departments. The most interesting part, for me, came at the end when he talked about universities as communities and learning occurring “primarily by way of interpersonal interaction in a community”. Even without thinking about research outputs, there is value then in having a rich and varied community with faculty who have deep love and enthusiasm for their subject.

Wolf provided a clear and compelling dissection of how current educational policy is creating adverse incentives to community-mindedness (both within and between universities). Something detrimental to the education sector, which is such a significant part of the UK economy.

Spicer provides an insight into how this feels as an academic. He talks about how “In the UK, two thirds of universities now have more administrators than they do faculty staff.” and describes academics are “drowning in shit” (pointless admin).

For me, Spicer’s solutions for what universities could do to change this weren’t so compelling. If I could change one thing I would look at how we cost (or fail to cost) academic staff time. Academics can feel that they are expected to just do any amount of work they are given, or at least they often have no clear divide between work and not-work and have to constantly negotiate their time.

Amy didn’t read, but watched Why mayors should rule the world – Benjamin Barber – and would highly recommend it. Our modern democracy revolves around ancient institutions – we elect leaders most of us never meet and feel like we have very little input into the democratic process. This isn’t the case in cities – the leaders of cities, mayors, are seldom from anywhere other than the city they look after. They went to the local schools, they use the public transport and hospitals – they’ve watched their city grow. They have a vested interest in improving it. Positive changes towards existential issues such as climate change and terrorism are happening in cities (he gives an example of the LA port, which after an initiative to clear up, reduced the city’s overall emissions by 20%), and something can be learnt from the way that they operate. There are networks of mayors across the world, with a sense of competitiveness between them as to who can be the best city. Mayors from different cities meet up and share their practices, helping other cities implement changes using best practice, without the bureaucracy of central government slowing change down.

Suzanne watched  What are universities for? the RSA talk by Professor Stefan Collini and  Professor Paul O’Prey. The second half of the video was more focused on the way that higher tuition fees have changed the nature of the relationship between universities and students, but the introduction to the talk was much more on the topic we were discussing today. Stefan Collini began by saying that he believes universities are partially protected spaces which prioritise ‘deepening human understanding’, and that there are few if any other places where this happens. He compared them to other organisations which do research, such as R&D departments in industry, or teams working in politics, but said the difference was that universities were able to follow second order enquiries, and look at the boundaries of topics and knowledge, as they didn’t have a primary purpose of furthering one particular thing or ideal. So, although there are many benefits, such as increased GDP, from the kind of enquiry universities do (the ‘deepening of human understanding’ he started out with), that isn’t their aim or goal. He also went on to say that although we tend to see universities as primarily for the benefit of the individual students (furthering their careers, developing their own skills and knowledge) they should be seen as providing public good as well, for the reasons outlined above. In the group we discussed his basic premise, that universities are ‘protected spaces’, and decided that we aren’t sure that is really the case (especially with so much research being funded by grants from industry). However, it did lead to an interesting discussion about what we feel universities are actually for, if they aren’t what Collini outlined.

Suggested reading

Video – notes from the reading group

Hannah read ‘Motivation and Cognitive Strategies in the Choice to Attend Lectures or Watch them Online‘ by John N Bassilli. It was quite a in depth study but the main points were:

  • The notion of watching lectures online has a positive reaction from those who enjoy the course and find it important, but also from those who don’t want to learn in interaction with peers and aren’t inclined to monitor their learning.
  • From the above groups, the first group is likely to watch lectures online in addition to attending them face-to-face, whereas the second group are likely to replace face-to-face interaction with online study.
  • The attitude towards watching lectures online is related to motivation (ie. those who are motivated to do the course anyway are enthusiastic about extra learning opportunities), whereas the actual choice to watch them is related to cognitive strategies.
  • There is no demonstrable relation between online lecture capture and exam performance, but often the locus of control felt by students is marginally higher if they have the option to access lectures online.

Amy recommended Lifesaver (Flash required) as an amazing example of how interactive video can be used to teach.

Suzi read three short items which lead me to think about what video is good for. Themes that came up repeatedly were:

  • People look to video to provide something more like personal interaction and (maybe for that reason) to motivate students.
  • Videos cannot be skimmed – an important (and overlooked) difference compared to text.

The first two items were case studies in the use of video to boost learning, both in the proceedings of ASCILITE 2016.

Learning through video production – an instructional strategy for promoting active learning in a biology course, Jinlu Wu, National University of Singapore. Aim: enhance intrinsic motivation by ensuring autonomy, competence, and relatedness. Student video project in (theoretically) cross-disciplinary teams. Select a cell / aspect of a cell, build a 3D model, make a short video using the model and other materials, write a report on the scientific content and rationale for the video production. Students did well, enjoyed it, felt they were learning and seem to have learn more. Interesting points:

  • Students spent much longer on it than they were required to
  • Nearly 400 students on the module (I would like to have heard more about how they handled the marking)

Video-based feedback: path toward student-centred learning, Cedomir Gladovic, Holmesglen Institute. Aim: increase the student’s own motivation and enhance the possibility for self-assessment and reflection. They want to promote the idea of feedback as a conversation. Tutor talking over students online submission (main image) with webcam (corner image). Students like it but a drawback is that they can’t skim feedback. Interesting points:

  • How would tutors feel about this?
  • Has anyone compared webcam / no webcam?
  • Suggested video length <5 mins if viewed on smartphone, <10 mins if viewed on monitor

Here’s a simple way to boost your learning from videos: the “prequestion” looks at the effect of testing whether students remember more about specific questions and more generally when they are given prequestions on a short video. Answer seems to be yes on both counts. They thought that prequestions were particularly useful for short videos because students can’t easily skim through to just those topics.

Roger read “Using video in pedagogy”, an article from the Columbia University Center for Teaching and learning.

The article primarily focuses on the use of video as a tool for teacher reflection. The lecturer in question teaches Russian and was being observed. As she teaches in the target language which her observer didn’t speak her original motivation was to make the recording then talk the observer through what was happening. In actual fact she discovered additional benefits she had not envisaged. For example she was able to quantify how much time she was speaking compared to the students (as an important objective is to get students speaking as much as possible in the target language, and the teacher less). Secondly she could analyse and reflect on student use of the vocabulary and structures they had been taught. Thirdly it helped her to reflect on her own “quirks and mannerisms” and how these affected students. Finally the video provided evidence that actually contradicted her impressions of how an activity had gone . At the time she had felt it didn’t go well, but on reviewing the video afterwards she actually saw that it had been effective.

Suggested reading

Evidence – notes from the reading group

Suzi read Real geek: Measuring indirect beneficiaries – attempting to square the circle? From the Oxfam Policy & Practice blog. I was interested in the parallels with our work:

  • They seek to measure indirect beneficiaries of our work
  • Evaluation is used to improve programme quality (rather than organisational accountability)
  • In both cases there’s a pressure for “vanity metrics”
  • The approaches they talk about sound like an application of “agile” to a fundamentally non-technological processes

The paper is written at an early point in the process of redesigning their measurement and evaluation of influencing. Their aim is to improve the measurement of indirect beneficiaries at different stages of the chain, adjust plans, “test our theory of change and the assumptions we make”. Evaluation is different when you are a direct service provider than when you are a “convenor, broker or catalyst”. They are designing an evaluation approach that will be integrated into day to day running of any initiative – there’s a balance between rigor and amount of work to make it happen.

The approach they are looking at – which is something that came up in a number of the papers other people read – is sampling: identifying groups of people who they expect their intervention to benefit and evaluating it for them.

Linked to from this paper was Adopt adapt expand respond – a framework for managing and measuring systemic change processes. This paper presents a set of reflection questions (and gives some suggested measures) which I can see being adapted for an educational perspective:

  • Adopt – If you left now, would partners return to their previous way of working?
  • Adapt – If you left now, would partners build upon the changes they’ve adopted without us?
  • Expand – If you left now, would pro-poor outcomes depend on too few people, firms, or organisations?
  • Respond – If you left now, would the system be supportive of the changes introduced (allowing them to be upheld, grow, and evolve)?

Roger read “Technology and the TEF” from the 2017 Higher Education Policy Institute (HEPI)  report “Rebooting learning for the digital age: What next for technology-enhanced higher education?”.

This looks at how TEL can support the three TEF components, which evidence teaching excellence.

For the first TEF component, teaching quality, the report highlights the potential of TEL in increasing active learning, employability especially digital capabilities development, formative assessment, different forms of feedback and EMA generally, and personalisation. In terms of evidence for knowing how TEL is making an impact in these areas HEPI emphasises the role of learning analytics.

For the second component, learning environment, the report focusses on access to online resources, the role of digital technologies in disciplinary research-informed teaching, and again learning analytics as a means to provide targeted and timely support for learning. In terms of how to gather reliable evidence it mentions the JISC student digital experience tracker, a survey which is currently being used by 45 HE institutions.

For the third component, student outcomes and learning gain, the report once again highlights student digital capabilities development whilst emphasising the need to support development of digitally skilled staff to enable this. It also mentions the potential of TEL in developing authentic learning experiences, linking and networking with employers and showcasing student skills.

The final part of this section of the report covers innovation in relation to the TEF.  It warns that “It would be a disaster” if the TEF stifled innovation and increased risk-averse approaches in institutions. It welcomes the inclusion of ’impact and effectiveness of innovative approaches, new technology or educational research’ in the list of possible examples of additional evidence as a “welcome step.” (see Year 2 TEF specification Table 8)

Mike read  Sue Watling – TEL-ing tales, where is the evidence of impact and In defence of technology by Kerry Pinny. These blog posts reflect on an email thread started by Sue Watling in which she asked for evidence of the effectiveness of TEL. The evidence is needed if we are to persuade academics of the need to change practice.  In response, she received lots of discussion, including and what she perceived to be some highly defensive posts.  The responses contained very little by way of well- researched evidence. Watling, after Jenkins, ascribes ‘Cinderella Status’ to TEL research, which I take to mean based on stories, rather than fact.  She acknowledges the challenges of reward, time and space for academics engaged with TEL. She nevertheless  makes a pleas that we are reflective in our practice and look to gather a body of evidence we can use in support of the impact of TEL. Watling describes some fairly defensive responses to her original post (including the blog post from James Clay that Hannah read for this reading group). By contrast. Kerry Pinny’s post responds to some of the defensiveness, agreeing with Watling – if we can’t defend what we do with evidence, then this in itself is evidence that something is wrong.

The problem is clear, how we get the evidence is less clear. One point from Watling that I think is pertinent is that it is not just TEL research, but HE pedagogic research as a whole, that lacks evidence and has ‘Cinderella status’. Is it then surprising that TEL HE research, as a  subset of  HE pedagogic research, reflects the lack of proof and rigour? This may in part be down to the lack of research funding. As Pinny points out, it is often the school or academic has little time to evaluate their work with rigour.  I think it also relates to the nature of TEL as a  set of tools or enablers of pedagogy, rather than a singular approach or set of approaches. You can use TEL to support a range of pedagogies, both effective and non-effective, and a variety of factors will affect its impact.  Additionally, I think it relates to the way Higher Education works – the practice there is and what evidence results tends to be very localised, for example to a course, teacher or school. Drawing broader conclusions is much, much harder.  A lot of the evidence is at best anecdotal. That said, in my experience, anecdotes (particularly form peers) can be as persuasive as research evidence in persuading colleagues to change practice (though I have no rigorous research to prove that).

Suzanne read Mandernach, J. 2015, ” Assessment of Student Engagement in Higher  Education: A Synthesis of Literature and Assessment Tools“, International Journal of Learning, Teaching and Educational Research Vol. 12, No. 2, pp. 1-14, June 2015

This text was slightly tangential, as it didn’t discuss the ideas behind evidence in TEL specifically, but was a good example of an area in which we often find it difficult to find or produce meaningful evidence to support practice. The paper begins by recognising the difficulties in gauging, monitoring and assessing engagement as part of the overall learning experience, despite the fact that engagement is often discussed within HE. Mandernach goes back to the idea of ‘cognitive’ ‘behavioural’ and ‘affective’ criteria for assessing engagement, particularly related to Bowen’s ideas that engagement happens with the leaning process, the object of study, the context of study, and the human condition (or service learning). Interestingly for our current context of building MOOC-based courses, a lot of the suggestions for how these engagement types can be assessed is mainly classroom based – for example the teacher noticing the preparedness of the student at the start of a lesson, or the investment they put into their learning. On a MOOC platform, where there is little meaningful interaction on an individual level between the ‘educator’ and the learner, this clearly becomes more difficult to monitor, and self-reporting becomes increasingly important. In terms of how to go about measuring and assessing engagement, student surveys are discussed – such as the Student Engagement Questionnaire and the Student Course Engagement Questionnaire. The idea of experience sampling – where a selection of students are asked at intervals to rate their engagement at that specific time – is also discussed as a way of measuring overall flow of engagement across a course, which may also be an interesting idea to discuss for our context.

Suggested reading

Horizon Report 2017 – notes from the reading group

This time we looked at the NMC Horizon Report 2017 and related documents.

Amy read ‘Redesigning learning spaces’ from the 2017 Horizon report and ‘Makerspaces’ from the ELI ‘Things you should know about’ series. The key points were:

  • Educational institutions are increasingly adopting flexible and inclusive learning design and this is extending to physical environments.
  • Flexible workspaces, with access to peers from other disciplines, experts and equipment, reflect real-world work and create social environments that foster cross-discipline problem-solving.
  • For projects created in flexible environments to be successful, the facilitator allow the learners to shape the experience – much of the value of a makerspace lies in its informal nature, with learning being shaped by the participants rather than the facilitator.
  • There are endless opportunities for collaboration with makerspaces, but investment – both financial and strategic – is essential for successful projects across faculties.

Roger read blended learning designs. This is listed as a short term key trend, driving ed tech adoption in HE for the next 1 to 2 years. It claims that the potential of blended learning is now well understood, that blended approaches are widely used, and that the focus has moved to evaluating impact on learners. It suggests that the most effective uses of blended learning are for contexts where students can do something which they would not otherwise be able to, for example via VR. In spite of the highlighting this change in focus it provides little detailed evidence of impact in the examples mentioned.

Suzi read the sections on Managing Knowledge Obsolescence (which seemed to be around how we in education can make the most of / cope with rapidly changing technology) and Rethinking the Role of Educators. Interesting points were:

  • Educators as guides / curators / facilitators of learning experiences
  • Educators need time, money & space to experiment with new technology (and gather evidence), as well as people with the skills and time to support them
  • HE leaders need to engage with the developing technology landscape and build infrastructure that supports technology transitions

Nothing very new, and I wasn’t sure about the rather business-led examples of how the role of university might change, but still a good provocation for discussion.

Hannah read ‘Achievement Gap’ from the 2017 Horizon Report. It aimed to talk about the disparity in enrolment and performance between student groups, as defined by socioeconomic status, race, ethnicity and gender, but only really tackled some of these issues. The main points were:

  • Overwhelming tuition costs and a ‘one size fits all’ approach of Higher Education is a problem, with more flexible degree plans being needed. The challenge here is catering to all learners’ needs, as well as aligning programmes with deeper learning outcomes and 21st century problems.
  • A degree is becoming increasingly vital for liveable wages across the world, with even manufacturing jobs increasingly requiring post secondary training and skills.
  • There has been a growth in non-traditional students, with online or blended offerings and personalise and adaptive learning strategies being implemented as a retention solution.
  • Some Universities across the world have taken steps towards developing more inclusive offerings: Western Governors University are offering competency based education where students develop concrete skills relating to specific career goals; Norway, Germany and Slovenia offer free post secondary education; under the Obama administration, it was made so that students can secure financial aid 3 months earlier to help them make smarter enrolment decisions; in Scandinavian countries, there is a lot of flexibility in transferring to different subjects, something that isn’t widely accepted in the UK but could help to limit the drop-out rate.
  • Some countries are offering different routes to enrolment in higher education. An example of this is Australia’s Fast Forward programme provides early information to prospective students about alternative pathways to tertiary education, even if they have not performed well in high school. Some of these alternative pathways include online courses to bridge gaps in knowledge, as well as the submission of e-portfolios to demonstrate skills gained through non-formal learning.
  • One thing I thought the article didn’t touch on was the issue of home learning spaces for students. Some students will share rooms and IT equipment, or may not have access to the same facilities as others.

Flexible and inclusive learning – notes from reading group

Amy read: Why are we still using LMSs, which discusses the reasons LMS systems have not advanced dramatically since they came onto the market. The key points were:

  • There are five core features that all major LMS systems have: they’re convenient; they offer a one-stop-shop for all University materials, assessments and grades; they have many accessibility features built in; they’re well integrated into other institutional systems and there is a great deal of training available for them.
  • Until a new system with all these features comes onto the market, the status quo with regard to LMS systems will prevail.
  • Instructors should look to use their current LMS system in a more creative way.

Mike read: Flexible pedagogies: technology-enhanced learning HEA report

This paper provided a useful overview of flexible learning, including explanations of what it might mean, dilemmas and challenges for HE. The paper is interesting to consider alongside Bristol’s Flexible and Inclusive learning paper. For the authors, Flexible learning gives students choice in the pace, place and mode of their learning. This is achieved through application of pedagogical practice, with TEL positioned as an enable or way of enhancing this practice. Pace is about schedules (faster or slower), or allowing students to work at their own pace. Place is about  physical location and distance. Mode includes notions of distance and blended learning.

Pedagogies covered include personalised learning, flexible learning – (suggesting it is similar to adaptive learning in which materials adapt to individual progress), gamification, fully online and blended approaches. The paper considers the implications of offering choice to students for example, over what kind of assessment. An idealised form would offer a very individualised choice of learning pathway, but with huge implications on stakeholders.

In the reading, group, we had an interesting discussion as to whether students are always best equipped to understand and make such choices. We also wondered how we would resource the provision of numerous pathways.  Other  risks include potential for information overload for students, ensuring systems and approaches work with quality assurance processes. Barriers include interpretations of KIS data which favours contact time.

We would have a long way to go in achieving the idealised model set out here. Would a first step be to change the overall diet of learning approaches across a programme, rather than offering choice at each stage? Could we then introduce some elements of flexibility in certain areas of programmes, perhaps a bit like the Medical School’s Self Selected Components, giving students choice in a more manageable space within the curriculum?

Suzanne read:  Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. The main points were:

  • Self-regulated learning is something which happens naturally in HE, as students will assess their own work and give themselves feedback internally. This paper suggests this should be harnessed and built on in feedback strategies in HE.
  • Shift in focus to see students having a proactive rather than reactive role in feedback practices, particularly focused on deciphering, negotiating and acting on feedback.
  • The paper suggests 7 principles for good feedback practice, which encourages this self-regulation: 1. clarifying what good performance is; 2. facilitating self-assessment; 3. delivering high quality feedback information; 4. encouraging dialogue; encouraging self-esteem and motivation; 6. giving opportunities to close the gap between where the student is now and where they need/want to be; 7. using feedback to improve teaching.
  • For our context, this gives some food for thought in terms of the limitations of a MOOC environment for establishing effective feedback practices (dialogue with every student is difficult if not impossible, for example), and emphasises the importance of scaffolding or training effective peer and self-assessment, to give students the confidence and ability to ‘close the gap’ for themselves.

Suzanne also read: Professional Development Through MOOCs in Higher Education Institutions: Challenges and Opportunities for PhD Students Working as Mentors

This paper reports on a small-scale (20 participants), qualitative study into the challenges and opportunities for PhD students acting as mentors in the FutureLearn MOOC environment. As a follow-on from the above reading, using mentors can be a way to help students with the peer and self-assessment practices, which is why I decided to read it in parallel. However, it also focuses on the learning experiences of the PhD student themselves as they perform the mentor role, also giving these students a different (potentially more flexible and inclusive) platform to develop skills.

Overall, the paper is positive about the experiences of PhD MOOC mentors, claiming that they can develop skills in various areas, including:

  • confidence in sharing their knowledge and interacting with people outside their own field (especially for early career researchers, who may not yet have established themselves as ‘expert’ in their field);
  • teaching skills, particularly related to online communication, the need for empathy and patience, and tailoring the message to a diverse audience of learners. It’s noteworthy here that many of these mentors had little or no teaching experience, so this is also about giving them teaching experience generally, not teaching in MOOCs specifically;
  • subject knowledge, as having to discuss with the diverse learning community (of expert and not expert learners) helped them consolidate their understanding, and in some cases pushed them to find answers to questions they had not previously considered.

Roger read Authentic and Differentiated Assessments

This is a guide aimed at School teachers. Differentiated assessment involves students being active in setting goals, including the topic, how and when they want to be evaluated. It also involves teachers continuously assessing student readiness in order to provide support and evaluate when students are ready to move on in the curriculum.

The first part of the article describes authentic assessment, which it defines as asking students to apply knowledge and skills to real world settings, which can be a powerful motivator for them. A four stage process to design authentic assessment is outlined.

The second part of the article focuses on differentiated assessment. We all have different strengths and weaknesses in how we best demonstrate our learning, and multiple and varied assessments can help accommodate these. The article stresses that choice is key, including of learning activity as well as assessment. Project and problem based learning are particularly useful.  Learning activities should always consider multiple intelligences and the range of students’ preferred ways of learning, and there should be opportunities for individual and group tasks as some students will perform better in one or the other.

Hannah read: Research into digital inclusion and learning helps empower people to make the best choices, a blog by the Association for Learning and Teaching about bridging the gap between digital inclusion and learning technology. The main points were:

  • Britain is failing to exploit opportunities to give everyone fair and equal access to learning technology through not doing enough research into identifying the best way to tackle the problem of digital exclusion
  • Learning technology will become much more inclusive a way of learning once the digital divide is addressed
  • More must be done to ensure effective intervention; lack of human support and lack of access to digital technology are cited as two main barriers to using learning technology in a meaningful way
  • We need to broaden understanding of the opportunities for inclusion, look into how to overcome obstacles, develop a better understanding of the experiences felt by the excluded and understand why technological opportunities are often not taken up

Suzi read:  Disabled Students in higher education: Experiences and outcomes which discusses the experience of disabled students, based on surveys, analysis of results, interviews, and case studies at four, relatively varied, UK universities. Key points for me were:

  • Disability covers a wide range of types and severity of issues but adjustments tend to be formulaic, particularly for assessment (25% extra time in exams)
  • Disability is a problematic label, not all students who could do will choose to identify as disabled
  • Universal design is the approach they would advocate where possible

Suzi also read: Creating Better Tests for Everyone Through Universally Designed Assessments a paper written for the context of large-scale tests for US school students, which nonetheless contains interesting background and advice useful (if not earth-shattering). The key messages are:

  • Be clear about what you want to assess
  • Only assess that – be careful not to include barriers (cognitive, sensory, emotional, or physical) in the assessment that mean other things are being measured
  • Apply basic good design and writing approaches – clear instructions, legible fonts, plain language