Rebecca's Research Blog

Way, way back, when I first found out about blogs and I was debating where to put the blog for my academic life, there was a debate about whether these blogs should go on institutional sites, or whether they should go on an external provider. WordPress wasn’t as big back then  – there were numerous options, and it wasn’t clear which ones were the best, which were likely to survive, which needed a bit of technical expertise and which were likely to start charging.

I’ve started several blogs since then, but I think this was my first. It took me all the way through my PhD and beyond. I have found it really helpful in reminding me of that process, reminding me of what it felt like to be a PhD student, reminding me of what was important to me at that time and what went wrong. It allows me to track how my research questions and my research overall evolved.

It’s now mainly a record of my PhD time, although I add to it infrequently. It has moved from being a read/write space to a read space. Not just for me – I point my doctoral students at it, and I have added posts on literature reviews and methodology so that I have advice I can point people to.

However, that brings me to the risks of institutional blogs. The university IT department makes the reasonable points that ‘Maintaining inactive blogs uses limited IT resources which are needed for more urgent work. They are also a security risk because any compromise is unlikely to be noticed.’

This one is currently scheduled for deletion because I haven’t posted to it for a couple of years. That’s not too bad but I noticed another of my OU blogs, which I authored with someone else, was recently deleted without warning. My co-author was the named lead, she has left the university, so no warning came and all that material is gone, including one blog post on viva questions that was very frequently read, referenced, and listened to.

Meanwhile, my external blogs on WordPress are there for me as sources of reference. They’ll survive once I leave the University.  This one, though, will eventually be gone. We hear a lot about the dangers of putting things online where they’ll be around for ever. We hear less about the dangers of putting things online where they could be swept away without notice. So much information, so many reference sources, swept away behind us.

Research question: ‘In what ways is the Covid-19 pandemic changing understandings of the relationships between learning and fun?’

A limited case study of a research group in a UK university. We expect our findings to have implications beyond that group.

Data collection using epistolary interviews (via email – one question per email, enabling thoughtful responses that can build over time) with everyone in the research group who wants to take part. Participation is voluntary and participants can drop out at any time.

Questions for epistolary interviews:

• What have been your main experiences of learning and teaching during the Covid-19 pandemic?
• What have been your main experiences of fun during the Covid-19 pandemic?
• How has learning and teaching changed (if at all) during the Covid-19 pandemic? In your answer, please take into account your own perspective and the wider perspective.
• How have you experienced fun changing (if at all) during the Covid-19 pandemic? In your answer, please take into account your own perspective and the wider perspective.
• How has your understanding of the relationship between fun and learning/teaching changed (if at all) during the Covid-19 pandemic?
• In what ways (if any) do you think the relationship between learning and fun will change after the Covid-19 pandemic?

Method:

According to Yin (who I like following for case study research because he has a clear structure to work to) there are five important components of case study design.

1. Five components of a case-study research design are particularly important
2. Its questions (see above)
3. Its propositions, if any (Prop 1- there is a relationship between learning and fun. Prop 2 – this relationship will be highlighted during the pandemic Prop 3- this relationship may be changed by the pandemic and some of these changes can be foreseen)
4. Its unit(s) of analysis (individual members of the research group)
5. The logic linking the data to the propositions (data are collected from people who have reflected on the relationship between learning and fun and who have thought deeply by what is meant by fun and by learning. Also we engage in fun and learning. Also we are influenced by the pandemic)
6. The criteria for interpreting the findings (thematic analysis to support explanation building, themes drawn from the data but also compared with existing frameworks of fun that were covered in our frameworks of fun paper. Interpretations are originally drawn together by one or two of us but are checked against the understandings of all participants. This is a reflexive case study and this phase of comparing understandings will enrich the analysis)

In my last post on this subject, I considered the distinction between method and methodology. In this one, I’ll look more at what is involved in writing the methodology chapter of a thesis (and I’m writing with social science theses in mind).

The first thing you need to consider is a very high level question. What’s the nature of reality? Broadly speaking, there are two perspectives you can take. One is that there is a reality. The truth is out there. That’s a positivist view.  A straightforward example is ‘one plus one equals two’. The other main view is that there are multiple realities or truths, depending on your perspective and context.  That’s a constructivist view. An example of this might be, ‘In the denary system conventionally used by mathematicians, one plus one equals two. In a binary system, one plus one equals ten. In a kitchen, one apple plus one orange equals one fruit salad. In a pet shop, one female rabbit plus one male rabbit equals a potential problem.’ Unless you know the appropriate context, you can’t assess whether a particular answer is correct or useful.

Once you’ve made it clear what your take on reality is, then other decisions flow from that. Broadly speaking, if you’ve taken a positivist view, you’re likely to want to use quantitative methods to uncover some aspect of the reality that is out there. If you’re constructivist, you’re likely to want to use qualitative methods to uncover some of those different understandings of reality. If you’re using mixed methods, then you’ve got to be careful to align them with your stated view of reality. A common problem in methodology sections is that writers forget to align their view of the world (their ontology), with their view of how to find out about the world (their epistemology).

At this stage, you can turn your attention to your research questions and to your theoretical perspective. You should already have shown that these emerge from your literature review (avoid the mistake of writing a literature review and then writing down some questions you clearly came up with before reading any of the literature). Check that your research questions and theoretical perspective align with the view about reality that you expressed earlier. Explain your methodology (see earlier blog post).

You now need to explain both what you did and, importantly, why you did it. Most of your decisions are likely to be explained in terms of your research question (eg I’ve considered several ways of going about this – and the one most likely to answer the research question is this one). Some will be explained in terms of your chosen method (I chose to do interviews, therefore I had to make a decision about sample size). Some will be explained in terms of your epistemology (some people would investigate this with a statistical analysis, but I believe that it’s important to uncover different views).

The important thing is to explain all your decisions in these terms. Sure, some of them will have been pragmatic decisions, but avoid framing them in that way. If you’ve made it clear that the best way of answering the question is to use a certain dataset, but you couldn’t get access to that dataset, then you need to explain what you did to compensate for that. Or you need to change your research question to one that you can answer satisfactorily with the available data.

Make it clear you knew you had options. In the end you chose interviews, but you could have used focus groups, or ethnography. What did you gain by using interviews? What did you lose by not using the other methods? Once you decided on interviews, which decisions followed? Structured or semi-structured? Face-to-face or online? Transcribed or not? In one language or many? In your methodology chapter, you need to be setting out the decisions that you made and why you made them. If you simply state what you did, it implies you didn’t give it much thought, or you weren’t even aware there was a decision to make.

Show that you know your chosen method(s) well. Terms you may have used loosely in the past need to be much more precise at this point. Lots of people talk about case studies, for example, but if you’re using this approach you need to explain whose interpretation of case studies you’re using and how you will apply it. Don’t just consider data collection; you also need to explain how you carried out your analysis. Be precise in all cases. If you’re doing mixed methods, don’t explain the quantitative analysis in enormous detail and then just say you’ll be doing some thematic analysis of your qualitative data. Conversely, don’t go into enormous detail about your qualitative analysis and then just say you’ll do statistical analysis of the numbers.

As a follow-up to that point – don’t treat your methods as a pick-and-mix. Methods are carefully developed over time to provide results that are credible and trustworthy. They align with certain views of reality. Some methods are difficult and complicated. That doesn’t mean you can get away with doing the easy part and ignoring the rest. Reference a detailed description of the method and demonstrate that you followed it.

Your methodology section will also consider some consideration of ethics. Avoid being formulaic here and writing a standard paragraph saying you’ll keep to some guidelines you’ve never read, and you’ll store the data in a way you’ll never adhere to. What are the real ethical challenges you’ll face? How will you really deal with them? Do the existing guidelines help in any way? How?

There’s a lot to cover, but bear in mind that you already know the answers. You already made these decisions. If you keep a research journal or a research blog, you already recorded these decisions. Take your reader through your decisions step by step. Justify them in terms of the ontology, epistemology, thematic framework, research questions that you selected. Send your reader on to the next chapter knowing exactly what you did and why you did it.

A common mistake that people make when first writing a literature review is to present the literature in a neutral and even way, as if it is all equivalent.

For example:

‘Smith (2009) found that pigs can fly. Jones’ study (2009) suggested that this is not the case.’

This doesn’t help your reader to understand or contextualise this information (for example, Smith may have been looking at airline travel for animals, while Jones was studying aerodynamics). More importantly, it doesn’t help your reader to understand your position. Will your paper / thesis take the position that pigs can or can’t fly?

Ambiguity also occurs if you give an author’s name at the beginning of a sentence, because this often implies you don’t have a position on what they say. For example.

‘Smith (2009) says the moon is made of green cheese’.

It’s clear what Smith thinks, but it’s not clear what you think.

However, if you say:

‘The moon is made of green cheese (Smith, 2009)’

you are making it clear that you think this is a fact, which was first uncovered by Smith.

The other option is to keep the author at the beginning, but only because you are going to disagree with their view.

Smith (2009) argued that the moon is made of green cheese but subsequent observations (Jones, 2012) have shown that it is mainly composed of dog biscuits.’

Here you are showing that you are aware of Smith’s view, but that you agree with Jones.

These points are taken from a talk by Sara Hennessy about assessing rigour in papers submitted to the British Journal of Educational Technology.

• Is the account analytical or purely descriptive?
• Do the authors critique the literature and present a balanced account or does the review gloss over known issues with educational technology initiatives?
• How do the theoretical assumptions and explanations of the case compare with alternative explanations?
• Are links made in the discussion/conclusions back to the conceptual framework?
• Are the findings critically interpreted in relation to the existing literature?
• Do the research questions convey genuine inquiry or do they assume that educational technology is a Good Thing?
• What is the theory of change?
• Were there any potential sources of bias?
• What measures were taken to counter them? eg were any counter examples sought when collecting and analysing data?
• Does reporting seem selective? Are the analyses systematic and explicit? Is the account reflective and evaluative?
• Was there any kind of control in a technology intervention design?
• Are there threats to validity and reliability? Does inadequate control of extraneous factors threaten validity of theoretical inferences from data?
• Are limitations acknowledged?
• Could reactivity have played a role? Did novelty value of shiny new tools increase motivation?
• How robust are the claims – how adequately is the argument supported by the evidence provided?
• Does the sampling strategy permit empirical generalization to a larger population?
• Who is the audience? Is the work relevant and current?
• Are there clear conclusions that generalized beyond specific case/context? Do they apply in other institutions?
• Is the work applicable / of interest in other countries? Is the focus overly parochial?
• Is educational technology serving only the privileged in developing countries who already have access?
• SES and gender inequity, rural/urban divide, language, computer literacy, bandwidth and intermittent connectivity / electricity maintenance, technical support, gatekeepers / stakeholders, culturally appropriate content….
• Is the research innovative or, at least, original
• What is the significance and contribution to existing knowledge – theory? methodology? empirical data?
• Has the innovation been tested with real users?
• Are there any convincing learning outcomes?
• Are there explicit implications for practice? Policy?
• Many reports of interventions motivated by ‘how can I use…’ rather than educational need
• How is it used, by and with whom, how often, for what purpose, under what conditions, with what support…
• What is the role of the teacher? Has pedagogy-focused professional development been offered? What cultural shift in teacher and learner roles is necessary?

I’m just back from the LAK17 conference in Vancouver. While I was there, I talked to a professor who had given feedback to doctoral students on their presentations in previous years. In most cases, it was the first time they had ever had feedback on their presenting skills. Thinking back to some of the talks I have heard from senior academics, I would guess that some have never had any help iat all in this area.

For me, an important part of presenting is the interactive aspect. If I see people looking puzzled, I try to go into more depth. If I see people looking bored, I’ll try to shift the pace. If a group of people arrives late, I’ll do a brief recap. If something of note happens in the room (loud noises outside, a phone going off , a butterfly circling overhead) I’ll try to react. A presentation is an opportunity to engage an audience and also to engage with an audience. I try to achieve that. I’m not claiming that I always succeed, but it’s always something I aim for.

The opposite of this approach is someone who reads a prepared script. I can see the attraction – particularly if you are nervous or if you are presenting in a language you don’t speak well. However, it’s usually far more difficult for an audience to follow. You put in more words. You use longer words. You start adding in citations. Your audience may be extremely familiar with the major works of every single person you cite – but it’s unlikely. If you cite two or three people in a single sentence, even the most skilled audience will be struggling to keep up. If everything you want to say is in the paper, people could just read your paper. Your presentation needs to offer something extra.

Small text doesn’t work. If you want your audience to read it, it needs to be big. If you are presenting to a large audience in a large room, your text needs to be very big. Unless it’s there as an example of illegible text, then small text serves no purpose. Long text has similar drawbacks. Either you have to pause awkwardly while the audience reads it, or you have to read it to the audience while they are reading it themselves, or you talk over it and the audience isn’t listening because they are reading the text.

You can run into the same problem with inexplicable images. For example, you put up a picture of a bear because you are talking about a big problem and you see a bear as an example of a big problem. While you talk about the issue, your audience is busy running through possible explanations for your choice of image. Woodland creature? About to hibernate? Likes honey? Lives in Canada? I once spent 20 minutes trying to figure out the use of an image on the second slide of a presentation. It was a presentation about a strong piece of work – in fact, it later won the prize for best paper – but I didn’t hear a word of it, because I was so distracted by the image.

Images also routinely get used without accreditation. People who scrupulously attribute the sources of their ideas use pictures without any reference to their creators. Sometimes, they even use ones that are have watermarks making it clear they are commercially available and should not be used without permission. It doesn’t take much effort to find images licensed using Creative Commons (you can filter your searches on both Flickr and Google Images in order to return these images). And when you use the images, credit them. Credit the work that went into them. Creative commons usually requires the author’s name and a link to the appropriate licence. This isn’t arduous, and it’s acknowledgment you would give to others whose work you have used.

A final thought. Your final slide. This is the one that stays up through the questions and answers that follow your talk. It is the slide that the audience spends most time looking at. Don’t waste it by filling it with ‘Thanks!’ Reiterate your key point. Provide a link to your work. Offer your contact details. Pose a question. Help the audience to engage with you once your presentation has ended.

There’s a difference between ‘method’ and ‘methodology’ – and it isn’t easy to grasp.

Partly that’s because the methodology chapter of a thesis contains your method and. when you’re grappling with the chapter, it’s difficult to see what you can add to that. You know how you’re going to collect your data. You probably even know how you’re planning to analyse your data. So how do you phrase that in the theorised way that your supervisor is asking for? At first sight, it does look as if your supervisor is trying to make a simple matter needlessly complex.

So let’s take a simple example. Imagine you meet a woman who owns a grocery shop, and she says she’ll pay you to find out how much fruit he has in her shop. At first sight, this too looks simple. You’re going to count the fruit. And, for the benefit of your supervisor (you are a doctoral student, after all), you note that this will be a quantitative approach.

You take in your notebook and pen, and you carry out a count and tally up your results. Because you’re a doctoral student with a little time on your hands you count them twice. You bring in your friend, and she counts them as well. A really reliable result. You go and tell the shopkeeper that she has 100 apples, 79 oranges, and 82 bananas. She tells you that isn’t the answer you wanted.

You’re a doctoral student, so you go and sit down and have a coffee and complain to a friend. And the friend, who happens to have been watching QI on television recently, says she thinks a banana isn’t rechnically a fruit. And maybe an apple is a fruit and maybe it’s a vegetable. Oh, and did you take pumpkins and cucumbers into account? So you grumpily stomp back to the shop and go through every type of produce with the shopkeeper and ask whether she defines it as a fruit. Then you count all the items defined as fruit. This time, the answer is 785. Or, annoyingly, 784 when you count a second time. The shopkeeper rejects both answers.

Time for another coffee, and another chat to your friend. How can you be expected to now what the shopkeeper wants? Well – your friend points out – you could ask why the shopkeeper wants this information and in what form she wants it. And it turns out the government is taxing fruit (which is defined in a particular governmental way) by the kilo. So you adopt a new system of classification and a new measure, and you tell the shopkeeper the answer is 350kg. She’s happy, and she pays you – which will keep you in coffee for a while.

Too give a meaningful answer you had to define your terms, and take the context and environment into account, and produce an answer that would be useful to the end user. Those are some of the things that you need to do in a methodology chapter.

Inspired by a Tweet I read recently about the distinction between a thesis and a PhD, I have been thinking about the difference between the two.

The university really focuses on the thesis, which must :

• be of good presentation and style
• be a significant contribution to knowledge and/or to understanding
• demonstrate capacity to pursue further research without supervision
• contain a significant amount of material worthy of publication or public presentation.

What else? Well, our university specifies you must be a registered student, you must live in the UK, you must pass your probationary period, you must spend a minimum amount of time as a registered student, you must make satisfactory progress, you must have a viva, you must make any specified corrections and you must present your thesis according to the guidelines.

All very thesis focused.

Vitae has a Researcher Development Framework that covers knowledge and intellectual abilities;  personal effectiveness;  research governance and organisation; engagement, influence and impact. The university encourages students to engage with this but, apart from reporting satisfactory progress at probationary review, it isn’t enforced or assessed.

Typically, students are assessment focused. They learn what they will be assessed on. It’s not surprising, then, that many doctoral students focus their entire attention on the thesis. That is the centre of their activity – everything else that takes place at the university is a distraction and has lower priority. In extreme cases, they only visit the university for supervision sessions, talk to nobody but their supervisors about their research, and focus totally on putting their thesis together and passing their viva.

But what then? A PhD is one line in a CV – perhaps five or six if you bulk it out with a description of your research. Permanent academic jobs in the UK and in many other countries may not be as rare as hens’ teeth, but they come pretty close. Even fixed-term contracts are difficult to get.

For employers, the PhD is not just one line in a CV, it’s also one line in a long job specification.

Academic employers want to know that you can publish papers, put together grant proposals, attract funding, increase impact via social media, create course materials, teach, mentor, work as part of a team, initiate projects, provide connections to a wider academic community and work on several projects at the same time.

The people getting the academic jobs are the people who can produce evidence that they can do all those things, and that they have already done those things. The people who treated their PhD as a period of academic apprenticeship, when the thesis is just one activity among many. The people struggling to get a toehold in the academic sector are the ones who have simply written a thesis.

Link to a useful article by the Thesis Whisperer, aimed at doctoral students beginning to work on their literature review.

How to become a literature searching ninja

CALRG seminar by Dr Gwyneth Hughes, Reader in Higher Education, Institute of Education, UCL

‘Ipsative’ assessment is about comparing your current performance with your past perfomance. It comes from the Latin word ‘ipse’, meaning herself or himself

Assessment is predominantly taken to be a measurement of learning, and can be considered to be one of the cornerstones of a meritocracy.

A focus on marks, grades and performance distracts attention from the learning process. It can reduce the motivation of students who consistently receive low marks.

Ipsative assessment distinguishes between learning and attainment, it also helps to build motivation and self-esteem. It involves feedback on how a learner has progressed. One approach is the use of learning portfolios in which students provide evidence of how they have learned and developed.

A funded project on assessment found that students were rarely given written feedback on progress. However, assessors found it very difficult, because they did not know what feedback students had been given in the past – particularly if the feedback had been provided by other educators. Feedback is not stored centrally.

They developed a Moodle plug-in that provides a reports dashboard, bringing together all previous feedback.

Assessment Careers: Enhancing Learning Pathways through Assessment: funded project

Ipsative Assessment: book by Gwyneth, published by Palgrave