Suzi read Why “what works” won’t work: evidence-based practice and the democratic deficit in educational research, Biesta, G 2007 and a chapter by Alberto Masala from the forthcoming book From Personality to Virtue: Essays in the Philosophy of Character, ed Alberto Masala and Jonathan Webber, OUP, 2015

Biesta gives what is broadly an argument against deprofessionalisation in the context of government literacy and numeracy initiatives at primary school level. I found the main argument somewhat unclear. It was most convincing talking about the difficulty in defining what education is for, making it difficult to test whether an intervention has worked. Talks at length about John Dewey and his description of education as a moral practice and learning as reflective experimental problem solving.

“A democratic society is precisely one in which the purpose of education is not given but is a constant topic for discussion and deliberation.”

Masala’s paper is on virtue/character education but is of wider interest as it talks very clearly about educational theory. I found particularly useful in this context the distinction between skill as a competence (defined by performance, so easily testable) and skill as mastery (defined by a search for superior understanding and less easily tested), and the danger of emphasising competence.

Hilary read Version Two: Revising a MOOC on Undergraduate STEM Teaching, which briefly outlined some key approaches and intended developments in a Coursera MOOC aimed at STEM graduates and post docs interested in developing their teaching.

The author of the blog post is Derek Bruff (director of the Vanderbilt University Center for Teaching, and senior lecturer in the Vanderbilt Department of Mathematics with interests in agile learning, social media and SRS – amongst other things: see http://derekbruff.org/)

Two key points:

1. MOOC centred learning communities – the MOOC adopted a facilitated blended approach, building on the physical groupings of graduate student participants by facilitating 42 learning communities across the US, UK and Australia to use face to face activities to augment the course materials, and improve completion rates.
2. Red Pill: Blue Pill – adopting the metaphor used by George Siemens in the Data, Learning and Analytics MOOC to give two ways to complete the course – either an instructor-led approach which was more didactic and focussed on the ability to understand and apply a broad spectrum of knowledge OR a student-directed approach which used peer graded assignments and gave the students the opportunity to pick the materials which most interested them, and so focus on gaining a deeper but less comprehensive understanding of the topic.

Final take away – networked learning is hard, as would be the logistics of offering staff / student development opportunities as online and face-to-face modules, with different pathways through the materials, but interesting …

Steve read Building evidence into education, 2013 report by Ben Goldacre for the UK government

Very accessible summary of the case for evidence-based pedagogy in the form of large-scale randomised controlled trials. Compares current ‘anecdote/authority’ edu research with past medical work – lots of interesting analogies. Focused on primary/secondary education but some ideas can transfer to higher – although would be more challenging.

Presents counterarguments to a number of common arguments against the RCT approach – it IS ethical if comparing methods where you don’t know which is best (and if you do know, why bother trialling?!). Difficulty in measuring is not a reason to discount, RCTs are a way to remove noise. Talks about importance of being aware of context and applicability. Uses some good medical examples to illustrate points.

Sketches out an initial framework – teachers don’t need to be research experts (doctors aren’t), should be research-focused team leading and guiding with stats/trials experts etc.

Got me thinking – definitely worth a read.

Roger read “Using technology for teaching and learning in higher education: a critical review of the role of evidence in informing practice“, (2014) by Price and Kirkwood

This study explores the extent to which evidence informs teachers’ use of TEL in Higher Education. It involved a literature review, online questionnaire and focus groups. The authors found that there are differing views on what constitutes evidence which reflect differing views on learning and may be characteristic of particular disciplines. As an example they suggest a preference for large-scale quantitative studies in medical education.
In general evidence is under-used by teachers in HE, with staff influenced more by their colleagues and more concerned about what works rather than why. Educational development teams have an important role as mediators of evidence.

This was a very readable and engaging piece, although the conclusions didn’t come as much of a surprise!  The evidence framework they used (page 6) was interesting, with impact categorised as micro (e.g. individual teacher), meso (e.g. within a department) or Macro (across multiple institutions).

Mike read Evidence-based education: is it really that straightforward?, 2013, Marc Smith, Guardian Education response to Ben Goldacre

This is a thoughtful and well argued response to Goldacre’s call for educational research to learn from medical research, particularly in the form of randomised controlled trials. Smith is not against RCTs, but suggests they are not a silver bullet.

Smith applauds the idea that we need teachers to drive the research agenda and that we do need more evidence. His argument that it will be challenging to change the culture of teaching to achieve this, seems valid, but is not necessarily a reason not to try. The thrust of his argument is that  RCTs, whilst effective in medicine, are harder to apply to education due to the complexity of teaching and learning. He believes (and I tend to agree) that cause and effect are harder to determine in the educational context. Smith argues  that in medicine  there is a specific problem (an illness or condition) and a predefined intended outcome (change to that condition). This can be problematic in the medical context, but is even harder to measure in education. I would add that the environment as a whole is harder to control and interventions more difficult to replicate. Different teachers could attempt to deliver the same set of interventions, but actually deliver radically different sessions to learners who will interact with the learning in a variety of ways. Can education be thought of as a change of state caused by an intervention in the same way we would prescribe a drug for a specific ailment?

All this is not to say that RCTs cannot play a role, but that you have to think about what you are trying to research before choosing your methodology (some of the interventions Goldacre addressed related to specific quantitative measurable things like teenage pregnancy rates, or criminal activity). Perhaps it is my social scientist bias, bit I woudl still want to triangulate using a range of methods (quantitative and qualitative).

From a personal perspective, I sometimes think that ideas translated from science to a more social scientific context can lose some scientific validity in the process (though this is maybe most true at the level of theory than scientific practice. For example Dwarkins translated selfish genes into the concept of cultural memes, suggesting cultural traits are transmitted in the same way as genetic code. Malcolm Gladwell’s tipping point is a metaphor from epidemiology which he applies to the spreading of ideas, bringing much metaphorical baggage in the process. Perhaps random control trials could provide better evidence for the validity of these theories too?

This month we picked articles from SEDA’s 53 powerful ideas all teachers should know about blog.

Mike read Students’ marks are often determined as much by the way assessment is configured as by how much students have learnt

Many of the points made in this article are hard to dispute. Different institutions and subject areas vary so widely that not only are how marks are determined different between say Fine Art and Medicine, but also between similar subjects at the same institution, and also between the same subject at different institutions. This may reflect policy or process (eg dropping the lowest mark before calculating final grade).  In particular, Gibbs argues that coursework tends to encourage students to focus on certain areas of the curriculum, rather than testing knowledge of the whole curriculum.  Gibbs also feels these things are not always clear to external examiners. He does not feel that QAA emphasis on learning outcomes address these shortcomings.

The article (perhaps not surprisingly) does not come up with a perfect answer to what is a complex problem. Would we expect Fine Artists to be assessed in the same way as doctors? How can we ensure qualifications from different institutions are comparable? Some ideas are explored, such as asking students to write more course work essays to cover the curriculum, and then marking a sample. This is however rejected as something students would not tolerate. The main thing I can take from this is that thinking carefully about what you really need to assess when designing the assessment is important (nothing new really). For example, is it important that students take away a breadth of knowledge of the curriculum, or develop a sophistication of argument? Design the assessment to reflect the need.

Suzi read Standards applied to teaching are lower than standards applied to research and You can measure and judge teaching

The first article looks at the difference between the way academics receive training for teaching and the way research and the way teaching and research are evaluated and accredited. Teaching, as you might imagine, comes off worse in all cases. There aren’t any solutions proposed, though the author muses on what would happen if they were treated in the same way:

“Imagine a situation in which the bottom 75% of academics, in terms of teaching quality, were labelled ‘inactive’ as teachers and so didn’t do it (and so were not paid for it).”

The second argues that students can evaluate courses well if you ask them right things: to comment on behaviour which are known to affect learning. There didn’t seem to be enough evidence in the article to really evaluate his conclusions.

The argument put at the end seemed sensible: that evaluating for student engagement works well (while evaluating for satisfaction, as we do in the UK, doesn’t).

The SEEQ, a standardised (if long) list of questions for evaluating teaching by engagement, looks like a useful resource.

Roger read Students do not necessarily know what is good for them.

This describes three examples where students and/or the NUS have demanded or expressed a preference for certain things, which may not actually be to their benefit in the longer term. He believes that these cases can be due to a lack of sophistication of learners (“unsophisticated learners want unsophisticated teaching”) or a lack of awareness of what the consequences of their demands might be (in policy or practice). The first example is class contact hours. Gibbs asserts that there is a strong link between total study hours (including independent study) and learning gain, but no such link between class contact hours and learning gain. Increasing contact hours often means increasing class sizes which generally means a dip in student performance levels.   Secondly he looks at assessment criteria, saying that students are demanding “ever more detailed specification of criteria for marking” , which he states are ineffective in themselves for helping students get good marks, as people interpret criteria differently. A more effective mechanism would be discussion of a range of examples where students have approached a task in different ways, and how these meet the criteria. Thirdly he says that students want marks for everything, but evidence suggests that they learn more when receiving formative feedback with no marks, as otherwise they can focus more on the mark than the feedback itself.

The solution, he suggests is to make evidence-based judgements which take into account student views, but are not entirely driven by them, to try to help students develop their sophistication as learners and to explain why you are taking a certain approach. This article resonated with me in a number of ways, especially with regard to assessment criteria and feedback. There is an excellent example of practice in the Graduate School of Education where the lecturer provides a screencast in which she goes through an example of a top level assignment, explaining what makes it so good.  She has found that this has greatly reduced the number of student queries along the lines of “What do I need to do to get a first / meet the criteria”.  I also strongly agree with his point about explaining to students the rationale for taking a particular pedagogic approach. Sometimes we can assume that students know why a certain teaching method is educationally beneficial in a particular context, but in reality they don’t. And sometimes students resist particular approaches (peer review anyone!) without necessarily having the insight into how they may be helpful for their learning.

Active learning might be an unhelpfully broad topic but there are some very helpful ideas in these papers.

• Bonwell, C. (1991), Active learning: creating excitement in the classroom, Eric Digest – The article starts by defining what AL is, the key factor being that students must do more than just listen e.g. read. write, discuss, problem solve. It identifies the main barrier to use of AL as risk, for example that students will not participate, or that the teacher loses control.  It suggests ways to address this for example by trying low risk strategies such as short, structured, well-planned activities.
• Prince, M. (2004), Does Active Learning Work? A Review of the Research, Journal of Engineering Education, 93(3), 223-232. Splits active learning into constituent parts and looks at the evidence for (often relatively minor) interventions covering each of these parts, in an attempt to identify what really works. A useful reference for anyone looking for quantitative evidence for active learning type interventions and a useful discussion of what leads to successful (or unsuccessful) problem-based-learning.
• Jenkins, M. (2010), Active Learning Typology: a case study of the University of Gloucestershire. The paper describes how an ‘active learning ‘strategy has been implemented at the University of Gloucester. In the first paragraph Jenkins provides some references on active learning to unpacks its meaning that helped us to better understand the term and put it into context,  for example, …the role of the teacher is not to transmit knowledge to a passive recipient, but to structure the learner’s engagement with the knowledge, practising the high-level cognitive skills that enable them to make that knowledge their own (Laurillard, 2008; 527). page 2. At the same time this is compared to the understanding of ‘active learning’ of the staff at the university which through  a survey were asked to identify their conceptions of active learning. The results identified three categories ‘families’, 1) external (student are active when they learn by doing), 2) ‘internal (student are active when they are engaged in cognitive processes) and 3) holistic (it is a composite of the two, and students are active learning is generally investigative, developmental, creative. An interesting perspective is a distinction in the interpretation where the emphasis is placed on the student or the teacher, Is active learning what the teacher gets the students to do or what learning is done by students? The data showed that there is a split between some staff practising ‘active teaching’ and other practising ‘active learning’. The outcome of the project has produced a framework for staff to work with which is very useful and identifies common elements of active learning in these five categories: Co-learning opportunities, Authenitcity, Reflection, Skills development, Student support.

Challenge-based learning  (Roger) – aim of CBL is to build 21st skills and  make learning relevant by basing it on real challenges to which students find a solution. It is multidisciplinary, collaborative, student directed and works best in a technology-rich environment.  The New Media Consortium carried out a trial with a number of US schools and Universities, which found that CBL does build 21st century skills, it engages students, and teachers felt that it helped students master the material. TheCBL website gives an overview of the key elements.

“‘… who might gain access to hitherto ‘unlearnable’ ideas” – nice phrase from part 1 of the blog post.

What are Learning Analytics (Suzi) – Puts forward the radical/futuristic idea that with the amount of personal data being recorded (videos watched, events attended, blog posts written, etc) and advanced data processing – formal qualifications might no longer be explicitly pursued. Instead some system would look at everything you’ve done and tell you that you are “64% to achieving a phd in psychology, 92% to achieving a masters in science…”

2011 Horizon Report – Four to Five Years: Learning Analytics (Suzi) – Overview with sensible coherent examples which are basically web stats/web personalisation applied to education. Seems an idea in its infancy.

Ambient learning cities (Suzi) – Learning doesn’t need to be a specialist activity – we can embed it back in our lives through the environments we live in. The only projects I could find around this were museum-focused, crowd-sourcing / crowd-curating collections. The RunCoCo project produces software and a process to manage this.

Suggested reading

• All three of these short articles, plus one related article of your own choosing: Innovation in Education and Learning (Department of BIS blog): part 1, part 2, part 3

Evaluating E-learning, A Guide to the Evaluation of E-learning, Graham Attwell, 2006 (Suzi) – Framework of factors to take into consideration, then lists 5-6 clusters of types of evaluation. Not much detail of how these types of evaluation actually work, but may be useful as a starting point. Section 6: SPEAK tool – used in community education – may be useful for “students as agents of change”. Section 9 – evaluating e-learning policies – gives guidelines and prompts for developing a set of questions with which to evaluate a policy. Section 10 – management-oriented evaluation – possibly worth a look during project planning.

Evaluation of e-learning courses– Institute of Education , 2008  (Roger)   Covers evaluating courses, wholly or partly online.  Aims to provide an overview of practical evaluation resources. And its target audience is academics at IOE. Includes a literature review.Recommendations:

• Plan evaluation before the course starts
• Collect feedback from all stakeholders including students, tutors, admins and tech support staff.   For staff this could be done on an ongoing basis through frequent team meetings, and an end of course survey
• Collect student feedback during and at the end of the course
• Consider all relevant aspects of the use of tech for t and l in the course e.g. usefulness of the  content, how well online  activities run e.g. timing, sequencing (could be blended), instructions, the user experience ( levels of engagement, tutor participation, workload)
• Make use of the specific tools available in typical online tools e.g. course stats in BB to get an idea of levels of activity (not quality)

Suggested reading

• A Technique for Quality Evaluation of E-Learning from Developers Perspective
• Online Education Evaluation: What Should We Evaluate?
• Transition and technology—Evaluation of blended learning, delivered by university staff to 6th form students
• Models and instruments for assessing Technology, Enhanced Learning Environments in higher. education
• e-learning benchmarks compared
• e-learning maturity model from Victoria University of Wellington – 5 process categories: Learning, Development, Support, Evaluation and Organisation – each process is broken down into practices, which are then rated
• HEA e-learning benchmarking methodologies
• HEA e-learning benchmarking Bibliography
• Jisc Curriculum design and delivery – http://jisccdd.jiscinvolve.org/wp/2011/11/09/tracks-in-the-snow-finding-and-making-sense-of-the-evidence-for-institutional-transformation/
• JISC – tangible benefits of e-learning
• Educause – Maintaining Quality in Blended Learning: From Classroom Assessment to Impact Evaluationhttp://www.educause.edu/Resources/MaintainingQualityinBlendedLea/213889
• Evaluation of e-learning courses WLE centre

There was discussion about this on the ALT list in May 2011. References given include:

• http://www.eep-edu.org/InnService/Start/what_addval_start.htm   – aimed at schools but still relevant
• https://sites.google.com/site/evaluationtoolkit/
• http://www.as.bham.ac.uk/learningspaces/LEI/Toolkit.shtml – University of Birmingham LT adoption toolkit
• http://www.sciencedirect.com/science/article/pii/S0360131505001338 – Critical success factors for e-learning acceptance UAE University
• http://www.sciencedirect.com/science/article/pii/S0360131506001874 – What drives successful e-learning

Information and digital competencies in HE  (Roger) – read one article in this collection  “HE and the knowledge society. Information and digital competencies”. (by Juan de Pablos Pons, 2010)  Knowledge society requires changes to teaching models – lecturers no longer the only source of knowledge, opportunities for technology to help the social dimension of teaching (“socialization of knowledge”). Universities are too compartmentalised, their structures and offer needs to reflect more the interdisciplinary nature of modern knowledge and research.   Distinguishes between IT competencies (how to use ICT) and “information competencies” (which would relate to more of Doug Belshaw’s 8 elements, e.g. criticality, creativity, cognitive etc. Lastly universities need to recognise that they are educating each student for multiple jobs (and these may not yet even exist)

The essential elements of digital literacies, Slideshare (Nic) This presentation offers a useful description of digital literacy (DL), i.e. DL is often understood as the ability to participate in a range of activities and creative practices that involve understanding, sharing and creating meaning with different kinds of media and technology”. This emphasise ICT and media. The presenation also introduces a ‘5-step process model’ and a matrix that can be used to map steps to different technologies. It also describes 8 essential elements of DL. Ones I found particularly interesting were:  As online texts are not exactly written, cyber literacy needs a different form of critical thinking; society is increasingly look for people who can confidently solve their own problem and manage their own life long learning, qualities ICT is believed to promote; DL must involve systemic awareness of how digital media are constructed. From our discussion, I’ve realised that althgough the term DL is complex and contested, the topic has an important relationship to TEL. Also, DL goes beyond reading (and writing) texts to actively and collaboratively creating meaning. This is where we, as learning technologist, can engage with it at a practical level, i.e. in the design of TEL activities. See a 38 page paper from this guy on ‘What is digital literacy?’ at http://neverendingthesis.com/doug-belshaw-thesis-as-submitted.pdf

The SCONUL seven pillars of information literacy – core model – (Roger) aimed at Library staff engaged in information skills work. 7 pillars : 1. identify a personal need for info, 2. scope – assess current knowledge and identify gaps, 3. plan – construct strategies for finding info, 4. gather info, 5. evaluate – including reading critically and evaluating info, 6. manage – organise information professionally and ethically (academic integrity) , 7. present – including active creation of knowledge.

The SCONUL seven pillars of information literacy – research lens  (Suzi) rather complicated extended metaphor for digital-literacy

The Medium is not the Literacy (Suzi) – article critical of the term from 2002 – the example he gives about news is a bit dated (digital news is more than just reading news stories on a screen now) – but makes a good point about the shortcomings of the term

Digital Natives: Fact or Fiction? (Suzi) – Useful, brief critique of the term “digital natives” stating that it was based on opinion (not science)

Futurity Now: Bruce Sterling on Atemporality (Suzi) – Nice illustration of the changes we face – “old” Feynman on how science works (write down problem, think hard, write down answer) vs what we do now (start by Googling to see if someone already solved it, etc)

Education Technology Standards for Students – Nice clear breakdown of the components of digital literacy (what it should mean for students to be digitally literate) including: creativity, citizenship, critical thinking, research, collaboration, and an understanding of the technology infrastructure.

European e-Competance Framework – “A common European framework for ICT professionals in all industry sectors” (including education)

Suggested reading

• TLRP briefing – digital literacies
• JISC Digital literacies programme
• Learning literacies in a digital age 
• Next generation user skills
• Information literacy in Europe – Literature review 
• What is ‘digital literacy’? A Pragmatic investigation, Doug Belshaw – thesis 
• Developing digital literacies – briefing paperhttp://www.jisc.ac.uk/media/documents/funding/2011/04/Briefingpaper.pdf

How to Make an eBook (Suzi) – How-to guide from Smashing Magazine. Clear and seems to do a good job of covering the most popular formats. Good for getting a grasp on the tech.

JISC e-book observatory project  (Suzi) – Survey by JISC written up in 2009, looking at use of e-books particularly by libraries and with reading done on computer screen. Good stats on quite how dissatisfied students are with short-loan collections, and on the expectation for e-books to be provided through the library.

How do e‐book readers enhance learning opportunities for distance work‐based learners? (Roger) – recent article from “Research in LT” (March 2011) . Small scale study.  Context: PG work-based DL progs. e-books/readers pre-loaded with materials piloted in response to 3 challenges, all of which they were found to effectively address: 1. need for flexibility about when and where to study for highly mobile work-based learners 2. limits of access to key readings e.g. in real libraries or accessible via campus computers 3. maximising benefits of learners’ limited and often fragmented study time. e-book readers pre-loaded with content in e-pub format given to 28 learners in 2009/10. Methodology used included cognitive mapping. All of the predicted benefits above were realised, in addition to cost-saving as students printed less out. Biggest issue was copyright restrictions which limited the ability to make other essential readings available.

Nielson Alertbox on information design for e-books (Suzi) – Essentially: write the content as though for print, create indexes and navigation as though for the (mobile) web. Kindle (at least) is still a bit rubbish for texts that involve jumping around rather than linear reading.

The line between book an internet will disappear, Hugh McGuire, September 2010 (Suzi) – The e-book / print book battle is a false one – e-books will become webpages by around 2015. e-Books currently defined by what you can’t do: link or deep-link to them, copy and paste, search them (especially, search a bunch of them at once). In short: e-books do not live on the internet. epub is just a website, bundled up into a thing. Books need APIs (see also: Google Books API and Open Library API)

Post-artifact books and Publishing, (Zak) Craig Mod

Suggested reading for the session

• Zak’s blog: Notes on ebooks

Derek Bruff’s blog provides lots of information on this topic.

Research-based practices for effective clicker use  (Roger) – Large study from 2007  (approx 4000 students) at large research uni in US. Clickers mostly used in STEM subjects. Key points: 1. Staff improve over time at using clickers effectively. 2. Conceptual questions are most popular with students, and they find these most useful when used to stimulate peer discussion in active sessions (as opposed to factual recall type questions)

Transforming student learning with classroom communication systems, Ian Beatty, 2004 (Suzi) – Description of the teaching method they have developed for leaching Physics to large groups. Lots of practical advice.

Clickers in the Large Classroom: Current Research and Best-Practice Tips, Jane E Cladwell, 2007 (Suzi) – Literature review containing advide we may want to summarise and make available on the website.

Improved Learning in a Large-Enrollment Physics Class (Nic), Louis Deslauriers, et al. (2011) Science 332, 862 (2011). A useful paper that presents robust evidence of significant educational benefits of remodelling large lecture teaching to incorporate a range of clicker techniques. The research methodology is interesting, if not slightly controversial, and there are useful citations to other studies as mentioned in bullets above.

Possible future reading:

• Evidence for the Efficacy of Student-Active Learning Pedagogies, Jeffrey E Froyd, 2007 (referenced by Nic’s article) – Summary of (other people’s) research supporting the use of: collaborative/cooperative learning, inquiry teaching, peer-led team teaching, problem-based learning, process-oriented guided inquiry learning, SCALE-UP
• Support online material for improved learning in a large enrolment physics class, L. Deslauriers, E. Schelew, C. Wieman, 2011 (support material for Nic’s article) – evaluation methods (including survey questions), in-class questions used (with comments from instructors), discussion of the Hawthorn effect (and why they believe it does not apply)

We looked at the Grand research challenges for TEL from Stallarnet and at some of the background papers and IoE TEL programme themes.

Socio-Technical Change: Computing, Bioscience, Maths by Dave Cliff – very accessible article looking at the next 20-50 years in technology, touching on: climate and energy issues; the importance of highly-mathematically-literate researchers; likely increases in computing power / reductions in cost of computing; disruptive technologies every 10 years (cloud computing -> ubiquitous and pervasive computing -> quantum computing); recording life / external personal memories; 3d printing …. and much more (I only read about half the paper) -Suzi

Open Educational Practices and Resources – OLCOS Roadmap 2012  –  overview of OER developments (written in 2007) and recommendations for policy makers, institutions, teachers and students on how to address OER-related challenges.  Emphasis on need for change in teaching practice from teacher-centred to more active participative student-centred approach to help learners develop digital competencies in order to succeed in a knowledge-based society – Roger

Other things that came up:

• RSA Animate – Changing Education Paradigms – animation of Ken Robinson on what education was and will be for
• Stanford AI MOOC – Seb Schmoller has been attending this and writing about it. See also What we can learn from Stanford (Seb’s summary in ALT newsletter).