End of a year or end of an era

Those who follow my blog posts, or used to, will have noticed a distinct lack of activity. This is partly because I am a bit busy in my Head of School role at the Open University. I have much reduced personal responsibility for assessment and indeed during 2019 I made a decision to step away from some of my assessment-related work. I just don’t have the time. This means that anything I post here about assessment is, in some senses, more and more a “view from the edge”.

However, my commitment to equality, diversity and inclusion just grows and grows – and, however hard we try, assessment very often fails to be equitable. I still research the impact of assessment and other things on different demographic groups. So expect “e-assessment (f)or learning” to take a different focus in 2020.

However, first of all, some background as to why this matters so much to me. I’m a woman and a physicist and that immediately makes me unusual (only around 25% of physics undergraduates in England are women). That’s a crying shame – what wasted talent! By the time you think about women profs and leaders in STEM Faculties, the numbers get even smaller. But there are things we can do to address the balance, and I am getting more and more involved in this work.

One of the most scary things I have ever done was to speak at “Succeeding in STEM: three women profs and their unconventional careers” in November. I speak at meetings all the time, so whay was this so scary? Well, because it was so personal. Not even my sister had known until that time how much I suffer with lack of confidence and imposter syndrome. But in recent years I have realised how important it is to “be myself” as a leader – and the fact that I struggle at times is part of that. Sometimes when a member of staff or a student in my School comes to me with an issue to resolve, it affects me deeply at a personal level….and that’s OK. I don’t want to be a souless Head of School, I don’t want to be super woman. I want to be authentic, I want to be me.

I hope that the recording of the “Succeeding in STEM” event (at https://bit.ly/34QOAEb) might help others a bit. However there is one thing that we didn’t get time to discuss, and this is a vitally important point: the issues are not just about sex or gender. We all have all sorts of biases and there are all sorts of reasons why people are overlooked, isolated, bullied – or feel themselves to be so. I am woman and a physicist, but I am also white, British, straight, cis, middle aged, and have become middle class…..etc. etc. It’s complicated! The concept of “intersectionality” is very important; we are all different. Let’s not forget that, and celebrate our glorious diversity at the same time as remembering others who are not the same as ourselves.

Posted in equality and diversity | Tagged , | Leave a comment

“A shepherd has 125 sheep and 5 dogs. How old is the shepherd?”

To my mind this question is clearly unanswerable.

I was therefore shocked to read, in an old but interesting book (Redish, 2003) that many schoolchildren try to find an answer. (“The teacher, wouldn’t give me a problem that has no solution”).

Furthermore they frequently come up with an answer of 25. (“There are only two numbers to work with: 5 and 125. Adding, multiplying and subtracting them doesn’t give something that could be an age. Only dividing gives a plausible answer.”)

I find that worrying. It links to the fact that true problem solving is about more than finding the right formula to use by trial and error and then manipulating it. I’m really not sure that many of our students develop the problem solving skills that we want them to develop; skills that would help in the real world as well as in passing physics exams.

We want our students to be independent and reflective learners, understanding the relevance of what they learn for the real world. How do we encourage these behaviours? I don’t know, but I find it worrying that Joe Redish goes on to report that so called unfavourable behaviours (Hammer, 1996) actually increased after instruction, at the expense of favourable behaviours.

Hammer, D. (1996) More than misconceptions: Multiple perspectives on student knowledge and reasoning, and an appropriate role for education research, Am J. Phys, 64, 1316-1325.

Redish, E.F. (2003) Teaching physics with the Physics Suite. Wiley.

Posted in independent learning | Tagged , , | Leave a comment

Establishing a free-text version of the force concept inventory

The Force Concept Inventory, established by Hestenes et al. (1992) is the original and best well known concept inventory. The idea is that you use the FCI pre- and post-instruction, asking students a bunch of questions which give you a summary of their conceptual understanding of Newtonian mechanics.

For the purpose of current work going on at the Open University, there are a number of significant points:

  1. The FCI (and most other concept inventories) are based on multiple-choice questions.
  2. The questions are supposed to assess underlying understanding. They aren’t mathematical and students aren’t supposed to agonise over them.
  3. The questions are based on pseudo “real world” situations, but yet students are expected to ignore air resistance etc.
  4. The same questions, or very slight variants of them, are used across the world.
  5. These aren’t “assessment for learning” questions of the type I usually play with, indeed feedback isn’t given. One reason for this is that there is a general paranoia in the community that the correct answers will get out.

There are some concerns about the FCI (including a gender gap in attainment which varies from question to question). However it is generally very well accepted.

As reported extensively in this blog, I am not the world’s greatest fan of multiple choice questions, and I have been involved in the development of the Pattern Match question type, in which short free-text answers from students are automatically marked. So, our current challenge is to develop a free-text version of the FCI. Early work (with David Sands from Hull, Ross Galloway from Edinburgh and Open University consultant Christine Leach) led us to decide, at least at first, to alter the original FCI questions as little as possible. We also discovered that it is a lot easier to develop reliable answer matching for some questions than others, so the latest version of the instrument is part free-text question, part MCQ and part other selected-response question types.

So where are we up to: our PhD student Mark Parker is currently at a critical stage of the project. He is gathering responses to the questions from students so as to be able to analyse the responses and improve the answer-matching.

If you have a group of students who would be able to help us in this developmental work, we would love to hear from you. Please email me, sally.jordan@open.ac.uk in the first instance.

Mark is also evaluating student reaction to the instrument and I’m expecting that we might find out a whole range of interesting things.

Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. The Physics Teacher, 30(3), 141-158.

Posted in force concept inventory, short-answer free text questions | Tagged , | Leave a comment

Reflections on the International Conference on Women in Physics

I’ll admit it, the link between this post and the blog’s declared subject matter is a bit tenuous. Some of my research is into demographic differences in attainment in physics, in particular apparent gender differences in attainment, which may or may not be as a result of particular types of assessment. I am also a woman, a physicist and Head of a School that is very proud of its Juno Champion and Athena SWAN Silver status, whilst being aware that there is still much we can do to improve our practice.

Thus it was that I found myself on the UK delegation to the International Conference on Women in Physics (ICWIP) in Birmingham (Photographs copyright Liz Hingley and the University of Birmingham). There were about 200 delegates from around the world, mostly but not exclusively women, itself really unusual in a field where only twenty-something percent of undergraduates are female, and where the “leaky pipe” means that female physics professors are rather rare. I have got so used to operating in a male-dominated world that I wasn’t really sure what I’d make of it and in some ways, with the perspective of hindsight, I’m still not sure what I did make of it. The situation for women in physics is by no means straightforward. I’d like to explore several different aspects.

One of the conference workshop series (not the one I went to) was about unconscious bias. I see unconscious bias on an almost daily basis, for example, today I was working at home and a rather irritating man came to to to door trying to sell me something. To his mind I was clearly the little housewife…Hmmm. At work, I’ve been confused for the assistant rather than the chair of an interview panel. And, yes I’ll admit it, sometimes I suffer from unconscious bias myself. Dr Smith…that’s a man, right. Not necessarily. The whole point about unconscious bias is that it’s an automatic reaction, I guess driven by society’s conditioning.

I also suffer, sometimes very badly, from imposter syndrome. What me…professor…Head of School?…no way. One day I’m going to be found out. I’m the pathetic, lacking in confidence, 18-year old who left home very young for University then surprised herself by doing quite well, but still didn’t have the confidence to stay on to do a PhD.

But yet, I don’t think I have ever suffered any discrimination for being a women, or a mother, and I don’t think either factor has stood in the way of my career. I stayed at home whilst my children were small and then worked part-time, but my husband then took a career break and he now does all the domestic tasks and has also shouldered responsibility for elderly parent care, first mine and now his.

The conference was a curious mixture of talks about women in physics and talks about physics itself, with Gabriela Gonzalez’s lecture on the search for and discovery of  gravitational waves being particularly good. We had inspirational lectures from Jocelyn Bell-Burnell, Julia Higgins and Athene Donald, and all alluded to difficulties in their careers as a result of society’s perceptions of the place of women. Jocelyn in particular talked about the fact that everyone was more interested in her announcement of engagement to be married than her discovery of pulsars.

[Incidentally, Jocelyn, in a fantastic talk given in association with her receipt of the Institute of Physics’ President’s Medal, also attributed the original discovery of pulsars in part to imposter syndrome. It goes something like this: there was a anomalous signal, but she checked, checked and double checked because she struggled to believe she was right…and when another signal popped up with the same characteristics, she recognised it. A more confident PhD student would probably have just put them down to noise.]

Much earlier in my career I had some small amount of reaction of the sort Jocelyn, Julia and Athene described: I married quite young, and my parents therefore assumed that I would be their “stay at home” daughter with no career ambition – not quite how it worked out in the end! However the significant thing is that I’m just slightly younger than Jocelyn, Julia and Athene and societal norms have changed a lot in my lifetime; being female and having a career is much less unusual than it was, certainly in the UK.

So, after much rambling, I get to the point of the conference which will stick with me for ever (alongside its sheer friendliness and supportiveness). The workshop series which I did go to was on “intersectionality” in other words exploring all the aspects of our identify which make a difference. So I’m a woman, a mother, and a physicist. I’m also white, British, middle class, straight and of a particular age. There were some touching and very honest reflections in the workshop from a gay black American woman; from several Indians about the impact of their caste on their life (and also that, culturally, those of Indian extract in the UK are encouraged by their parents to study engineering rather than physics, leaving a situation where one woman feels dreadfully isolated in a physics class whilst lots of people from similar backgrounds are studying engineering at the university down the road); and from a British woman who feels intimidated because she’s young (as I did once).

Then right at the end of the conference we had a surprise visit from Malala Yousafzai, which was extremely moving. What can I, a western woman, really understand what she has been through in the cause of women’s rights to education? The implications of her bravery will haunt her for the rest of her life; she arrived in an armoured car with her father and bodyguard and information about her visit was embargoed until she was safely somewhere else. Truly humbling, but yet on one level she was just another school leaver who had found physics quite difficult.

So, whilst one’s gender and sex are important, so are many other accidents of birth – where you are born, your ethnicity, the colour of your skin, your religion, your age, your sexuality, whether you have a disability etc. etc. It is right that we continue to fight for true equality of experience for all.

Posted in conferences, equality, gender | Tagged | Leave a comment

A call for a holistic student assessment experience

Oh dear! It has been too long since I posted to this blog and now I only have about 30 mins in which to try to catch up. I’ve very quickly updated my biography etc and tried to remove some of the dead links. I have lots of more important things I want to say so, over the next few weeks, look out for

  1. A general review of the 2017 Assessment in Higher Education Conference (which has just taken place and which was fantastic!)
  2. A specific reflection on one of the keynotes – is technology enhanced assessment a sheep in wolf’s clothing or a wolf in sheep’s clothing…
  3. More about my presentation at the conference, in which I explored the boundaries and tensions between assessment and learning analytics.

For now though, I’d just like to remind you of the link between Assessment in Higher Education and the Transforming Assessment webinar series…and to advertise the fact that there is a webinar which presents three papers from the conference this coming Wednesday, 5th July, at 7am GMT (8am BST). Click here for more information.

And what better way to advertise an upcoming webinar than to report on a wonderful previous webinar, presented by Mark Kerrigan from Anglia Ruskin University (also building on work done at the University of Greenwich) about “Map My Programme” which has sought to encourage staff and students to think holistically about assessment across a programme, recognizing that even when submission dates are staggered, assessment tasks from different modules can create confusing overlaps. All too frequently we think about assessment at the subject level without stopping to think about the non-subject-specific transferrable and employability skills. Where are we assessing those? Anyway, rather than relying on my brief report, why not watch the recording of the webinar for yourself. Click here.

Posted in conferences, programme-focused assessment | Leave a comment

EAMS 2016

Back in September I was delighted to be a keynote speaker at EAMS: the first ever conference on E-Assessment in the Mathematical Sciences. For more information about the conference in general click here; for a recording of my keynote click here.

The conference was a pleasant surprise in many ways. Firstly, it was truly international, with attendees from Australia, Finland, Ireland, Japan, Norway, the Netherlands, South Africa and the US as well as from all over the UK. Secondly, it was not as “techy” as I’d feared it might have been, nor did the maths go right over my head (well, not very often…). Thirdly, and most importantly, I had to tone down my keynote, which had been written to be rather critical of much e-assessment practice, because there was some fantastic stuff reported at the conference! I was particularly pleased to be a keynote speaker alongside Christian Lawson-Perfect, Chris Sangwin and Michael Gage. Thus we heard about some of the very good e-assessment systems and question types for mathematical sciences: NUMBAS, STACK and WeBWorK. I added in some detail on Pattern Match.

As I’ve said before in this blog, I am particularly impressed by STACK, and its author Chris Sangwin gave a particularly thoughtful talk on “the interplay between calculation and reasoning”, which fed very neatly into my discussion of “how far is it appropriate to go” in assessing automatically.

Of course there were talks that were less good;  the one point that I’d still want to emphasise is the need to monitor student use of questions very closely, and not to assume that they are behaving in the way that you think they are. However it was a joy to be at a meeting with such lovely people, most of whom seemed driven  by a desire to improve their students’ learning. It was also a great pleasure that Cliff Beevers, who was already an expert in this area when I was just setting out in the very early 2000s, was also at the meeting.

Posted in conferences | Tagged , , , , | Leave a comment

Failure or success?

I can be a bit of a glass half empty sort of person, but recent events have made me reflect on society’s capacity to look for failure in success.

I guess my thinking comes in part from the events surrounding the ExoMars 2016 mission, and the excitement and disappointment felt in the School of Physical Sciences at the Open University, of which I am Head. On 19th October, two things were supposed to happen, about 225 million kilometres away, in the vicinity of Mars. First of all, the Schiaparelli lander (the shiny bit on top of the Trace Gas Orbiter, of which a quarter-sized model is shown in the photograph below), which had separated from the TGO previously, was supposed to land on Mars. Well it did, after a fashion; trouble is it came in far too fast and appears to have exploded on impact. Hmmm, so that was a failure then?

P1060366

Well, actually no. Schiaparelli was a “entry, descent and landing demonstrator module” – it was meant to test and demonstrate the technology. The majority of that technology worked according to plan. The heat shielding operated as anticipated during atmospheric entry, the parachute deployed correctly and further slowed the lander, the front heat shield jettisoned as planned and the landing radar was activated and began returning data. Unfortunately, Schiaparelli then released its parachute earlier than anticipated and only fired its landing thrusters for a few seconds rather than the predicted 30-second burn; shortly afterwards communication was lost.

It is important to note that most of the data which was collected during Schiaparelli’s descent has been recovered. The mission’s AMELIA team, of which the my colleague Stephen Lewis is Co-Principle Investigator, is aiming to use this descent data to reconstruct the lander’s trajectory from an altitude of around 120 km down to the point at which it lost contact. The team can then build up vertical profiles of the Martian atmosphere – e.g. temperature, pressure, wind speed – which will be compared with existing atmospheric models and simulations. This is the first landing attempted during the planet’s annual dust storm season, meaning that the results will offer unprecedented insight into the atmosphere at this time of year.

Most significantly, the majority of the science planned for this part of the ExoMars programme will be undertaken by the Trace Gas Orbiter (TGO). And whilst we were all worrying about the fate of the Lander on 19th October, the TGO had successfully entered Martian orbit. That’s a real success. The OU is closely involved closely on two of the orbiter’s instruments: Manish Patel is Co-Principle Investigator on NOMAD, an Infrared-to-UV spectrometer investigating the composition of the Martian atmosphere, and Matt Balme is a Guest Investigator on the TGO mission, working with data from the CaSSIS instrument, which will be returning high resolution, colour, stereo images of the planet. These stereo images can be used to generate 3D elevation data of the Martian surface.

So, we have lots to celebrate. But the public sees the failure, not the success. This is the point at which I return from my description of events around Mars to my usual standpoint as a commentator on assessment-related matters. We need to remember how much damage a single “failure” (e.g. a poor exam result) can do to an individual, despite previous successes. Even when failure is real, we need to learn to move on and to build success out of failure rather than the other way round.

 

[With thanks to PhD student Rhian Chapman, whose description of the work of the TGO and Lander I have adapted from the School of Physical Sciences’ website (http://www.open.ac.uk/science/physical-science/news/exomars-update)]

 

Posted in ExoMars, Success and failure | Tagged , , | Leave a comment

Keys to transforming assessment at institutional level: selected debates from AHE2016

Hot on the heals of my post about Sue Bloxham’s keynote at the Assessment in Higher Education workshop in June, this post is about the follow-up  Transforming Assessment webinar “Keys to transforming assessment at institutional level: selected debates from AHE2016.

Three talks from the AHE workshop had been selected for the webinar on the basis of the fact that they really did focus on change at the institutional level, and I thoroughly enjoyed chairing the session. If you want to watch the whole thing, click here for more information and the recordings.

The first of the three talks that we’d selected was “Changing feedback practice at an institutional level” in which Sally Brown talked about work at the University of Northumbria, Leeds Met (now Beckett) University and Anglia Ruskin University. Kay Sambell had given this talk at the earlier workshop and their conclusions were that

  • Slow transformative development has more impact than attempts at quick fixes;
  • Having money to support activities and personnel is important, but large amounts of cash doesn’t necessarily lead to major long-term impact;
  • Long-term ownership by senior managers is essential for sustainability;
  • To have credibility, activities need to be based on evidence-based scholarship;
  • Committed, passionate and convincing change agents achieve more than top-down directives.

The third of the three talks was “Changing colours: what happens when you make enhancement an imperative?” in which Juliet Williams talked about the impact of the TESTA (Transforming the Experience of Students through Assessment) Project at the University of Winchester.

However, from the conversations that I had at the workshop in June, it was the middle talk (given at the Webinar by Dave Morrison of the University of Plymouth because Amanda Sykes from the University of Glasgow was unavailable) that had inspired many of the attendees – bearing in mind that these were largely assessment practitioners not experts. The title was “Half as much but twice as good” and the important points I picked up were that

  • Timely feedback is more important than detailed feedback
  • [students are as busy as we are so] Less feedback can be more effective. If a student only reads your feedback for 30 seconds, what do you want them to take?

Capture 3

We ended with a good discussion of how to bring true institutional change.

Posted in conferences, insitutional change, timeliness of feedback | Tagged , , , , | Leave a comment

Central Challenges in Transforming Assessment at the Departmental and Institutional Level

Back on 30th June, Assessment in Higher Education (AHE) held a seminar  in Manchester with the theme of “Transforming assessment and feedback in Higher Education on a wider scale: the challenge of change at institutional level”. The idea behind the seminar was partly to hold a smaller-scale event between our increasingly large bienniel conferences, though we had well over 100 attendees.

AHE are now working in collaboration with Transforming Assessment, and we live-streamed Sue Bloxham’s keynote to a further twenty or so people around the world. Then on 13th July we had a dedicated webinar to which three selected presenters from the seminar contributed. My involvement in both of these events meant that I had a double ‘bite at the cherry’. I heard one set of presentations at the seminar itself, then I heard the three presentations on 13th July , and chaired a discussion of overlapping themes. There was some fantastic stuff.

capture 1As I try to catch up with this blog, I’ll start by describing my take on Sue’s keynote. She started from the precept that assessment remains unfit for purpose – and change is slow. She went on to outline what she described as key barriers to assessment enhancement, where the two  barriers that have most resonance with my own experience being:

 

  • centrally imposed change, which produces resistance. Sue’s proposed solution is that we should put the focus for change on small low-level workgroups.
  • the need for assessment literacy for staff. Here the focus must be on adequate professional development.

Sue went on to describe a  framework which might be drawn upon to create the conditions for transformation at institutional or departmental level, based around

  • key principles
  • infrastructure
  • strategy
  • assessment literacy.

Capture 2

It was inspirational; now we just need to make the change happen. Don’t take my word for it though; you can watch the recording of Sue’s keynote  here.

 

Posted in conferences, insitutional change | Tagged , , | 1 Comment

Fishy feedback

I don’t think that I am ever going to win prizes for my artwork. However this post is about my picture of a fish, shown below:

fish

A month or so ago, I was invited to take part in a live event about assessment and feedback, and for some reason the three of us involved decided to base this around an ‘assessed task’ which was to draw a fish.

There were a few problems – the first one was that we hadn’t been told the criteria against which our drawings would be judged. So my colleague John drew fish and chips. Even though I did better than John did, I was unaware of the detailed criteria.

So, when our work was ‘marked’ none of us did very well. The feedback given related to efforts as marked against the criteria – which we hadn’t seen. So I was praised for my colourful fish (I just happened to have a red pen in my hand) but downgraded because I had not shown any water. So, despite the fact that my marks were better than those of my two colleagues, I was angry to start with – I hadn’t been told to draw a fish in water!  My emotions stopped me from listening to the rest of the feedback, which may or may not have been justified.

Finally, the mark scheme gave credit for  the presence of different sorts of fins and gills. I have no idea whether my drawing, or the other colleague’s one that I was marking, included these points or not. My knowledge of fish anatomy is somewhat limited!

So, very quickly (and not as planned by the activity) we have a list of important points for assessing and giving feedback.

1. Make sure your assessment criteria are clear – and shared with students.

2. Receiving feedback is an emotional experience; bear that in mind.

3. Make sure that students understand the language that you use in your feedback.

 

Posted in assessment criteria, emotional reaction, feedback | Tagged , , | Leave a comment