I’m just trying to write some things down after our tutorial today. Amongst other things there were two books recommended which offered some ideas about research methods. These were:

• Universal methods of design [electronic resource]: 100 ways to research complex problems, develop innovative ideas, and design effective solutions, Bruce Hanington and Bella Martin
• Creative Research Methods in Social Sciences, Helen Kara

I think I’m clear that a big part of my research is going to be through a focus group. However, as I will offer individual interviews as well and Hanington and Martin say, ‘The power of focus groups lies in the group dynamic that it creates.’ (Hanington and Martin, 2012, p.92). I’m not 100% sure that’s the right term or if it is, am I undermining it by doing individual interviews too. Would I be mixing methods too much? I’m not sure.

Despite this, the idea of a structured group discussion, in person, with around 6-8 participants is pretty clear in my mind. I’m sure I want to do this students, I need to work out if I have the capacity to do this also with staff.

Finding participants:

One of my questions is about the potential range of people I want to hear from. Is it too broad to focus on people who are neurodivergent, those from a non-academic background and those with English as an additional language. Lindsay said to go ahead with all of them, but Rachel said perhaps it’s best to focus on a smaller group and it might turn out that the groups overlap – intersectionality would most likely be a factor. Carys said that current practice in the Disability team is to ask for responses from all participants because not everyone identifies as disabled and by not asking for contributions from all, I could be inadvertently excluded disabled students.

If I have the permission of the BA course team, I am going to recruit by going in, in person to a teaching session, explaining the project and the aims – improving the design of the course Moodle pages and others at CSM – the methods and time needed. I’ll need to provide this information in different formats (written and verbal, perhaps a short video?) to be more inclusive, and provide an easy way to contact me. It might be best to focus on 2nd years in the first instance as they have experienced the previous version and still have reason to be invested. Q. Does it matter if some have experienced the previous version and some haven’t?

For staff, I could present the project in a team meeting, and ask if anyone who is willing to talk to me about it can let me know privately. I’m assuming that staff members may not want to disclose any neurodiversity or disability to the rest of the team.

Bibliography:

Hanington, B., and Martin B. Universal Methods of Design : 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions, Quarto Publishing Group USA, 2012. ProQuest Ebook Central

I’m really enjoying the process of trying to work out how to ‘do’ thematic analysis of my data, but it’s also very clear that I’m barely scratching the surface of what I’d need to know, do and understand to be able to do this in a really meaningful way. I am doing my best however, to be diligent and thoughtful and conduct the analysis in the best way I can, given the time constraints and my starting point.

I’m primarily focussing on Braun and Clarke’s approach to thematic analysis because it offers the flexibility to be able to incorporate multiple types of data and, as far as I understand, it isn’t attached to a particular theoretical model. It therefore allows me to work out where I am in relation to various criteria as I go along and try to make sense of my data. Using the slide below from Braun and Clarke’s lecture shown below as a framework, I understand my data coding to be primarily inductive in that I didn’t know what the participants would say from the start and I’m basing my codes on their contributions. However, I don’t think it’s purely inductive as inherent in my research question is the pre-conceived idea of what our Moodle template actually consists of, or its defining features. Therefore my coding is inevitably influenced by my research question i.e. to what extent these features offer support to students and staff who might otherwise experience barriers.

In addition, I believe I am taking an experiential orientation to the data in that my data is very deliberately based on the experience of participants (I started my interviews with a request to talk through what participants do on Moodle and what barriers or supports they encountered. This was followed by prompts, which were based on certain areas I was interested to discuss, but participants had a free rein to select the topics and avoid others.)

Familiarisation with the data:

My data collection techniques were: semi-structured interviews, a short questionnaire (mostly quantitative but with one free text box) and drawings of participants’ experience of using Moodle. Three participants gave me drawing and one gave me a ‘voice drawing’ an audio description of what it’s like to use Moodle from her perspective.

Coding:

My approach to coding is semantic as opposed to latent. Again however, I believe this isn’t cut and dried. As Braun and Clarke (2018) point out, these are all spectrums and I think I am somewhere between the two. My codes and categories are largely descriptive and meaning based (semantic end of the spectrum) but they are also partially formed (through the interview process) and informed by my existing knowledge of digital accessibility, inclusive practice and my research question, which I believe pushes my approach somewhat towards a latent coding approach.

Fryer describes the contrast between data-led and theory-led coding: ‘(data-led coding) can have the benefit of allowing the data to surprise us, to challenge our preconceptions, or to move beyond previous theorising—whereas theory-led coding has more of a tendency to encourage us to find what we seek.’ (Fryer, 2022). However, like Braun and Clarke, he says is it not a binary choice and an element of both approaches is almost inevitable.

In my process, it took me two or more attempts to establish the codes I wanted to use. The process of categorising (Lochmiller 2021) involved more code development. The image below shows part of my coding process using the platform Miro. Participants are given different coloured sticky notes as identifiers and codes are added as coloured tags. In most cases, related codes are given the same colour. E.g. Contextual information: usefulness, Contextual information: presentation.

The reason I used Miro for this purpose was to keep the data as movable items (individual sticky notes), but also because Miro will cluster items according to tags. In the screenshot below, each sticky note has been grouped according to the tag ascribed. At this point it’s possible to see if the codes include comments from more than one participant (by looking at the colour coded sticky notes), and also if there are codes which are too large or too small and should perhaps be split or combined. At this particular stage, I could also separate out the data I had yet to code (the ‘no tag’ group) and so I could add these to the existing groups.

In adding the additional data, I made some amendments to the codes. For example, a new code was introduced – site layout – and Finding what you need was divided up, along with appearance. Within the new code Site layout, there were some sub groups as shown below. Later, some of these data would be moved again, during the analysis phase.

This is the point where the categories came in, shown below in white boxes next to the groups of codes. This process helped me to manage the data and at the same to revisit it and check the coding and groupings. I changed the categories several times and tried to consistently refer back to the research question, shown on the top left in a grey box. At this point, I also added the drawings which I think best illustrated the points made in the category.

Generating and reviewing themes:

The process of analysis is obviously a thread through the different stages of TA, but the process of generating themes is one I found very difficult. Braun and Clarke are very clear that themes do not ’emerge’ and they have have moved away from describing the process are ‘searching’ for themes, as they specifically emphasis the active role of the researcher and as such have chosen the more active term ‘generating’ themes (Braun and Clarke, 2018). I like this distinction although it also makes me slightly uneasy as it’s a new way of looking at data from research for me.

In working out what the themes should be, I found this description helpful; ‘Patton‟s (1990) dual criteria for judging categories – internal homogeneity and external heterogeneity – are worth considering here. Data within themes should cohere together meaningfully, while there should be clear and identifiable distinctions between themes.’ (p. 20 Braun and Clarke, 2006)

Eventually I managed to generate 4 themes which I think are both distinct from each other and individually bring together different codes in a meaningful way. For a full description of the themes and the final report, please see Project findings: report and summary.

Reflections:

In doing this again, I would do the process of transcribing differently. At the start I listened to the interviews and wrote notes directly on the sticky notes. I didn’t realise that at this stage I was effectively transcribing and I felt it was ‘cheating’ to write actual quotes on the sticky notes as I listened to the interviews. However, it because clear that this was actually wrong and what I needed was a full transcription to be able to see some of the quotes I did have in context without having to listen over and over, and also the notes were not useful, I needed them all to be actual quotes.

As a result in the analysis stage, I had to keep returning to listen to the interviews, understand the context and also check the accuracy of the quote. This did mean I was very familiar with it by the end but it wasn’t efficient in terms of timing.

As I said at the start, I need to understand more about the context of thematic analysis but I found it a useful and enjoyable (up to a point…) process.

Bibliography:

Braun, V., and Clarke, V. (2006) ‘Using Thematic Analysis in Psychology’. Qualitative Research in Psychology 3, no. 2 (January 2006) pp.77–101 Available at: https://doi.org/10.1191/1478088706qp063oa.

Braun V. and Clarke V. (2018) Thematic analysis: An introduction. June 25 2018 Available at: https://www.youtube.com/watch?v=5zFcC10vOVY (Accessed: 29 December 2023)

Clarke V. (2017) What is thematic analysis? Dec 9 2017. Available at: https://www.youtube.com/watch?v=4voVhTiVydc. (Accessed 27 December 2023)

Fryer, Tom. ‘A Critical Realist Approach to Thematic Analysis: Producing Causal Explanations’. Journal of Critical Realism 21, no. 4 (8 August 2022): 365–84. https://doi.org/10.1080/14767430.2022.2076776.

Lochmiller, Chad. ‘Conducting Thematic Analysis with Qualitative Data’. The Qualitative Report, 20 June 2021. Available at: https://doi.org/10.46743/2160-3715/2021.5008.

References:

Friese S, Soratto J, Pires D, 2018 Carrying out a computer-aided thematic content analysis with ATLAS.ti MMG Working Paper 18-02

Nickerson, C. 2023, Social Constructionism Theory: Definition And Examples. Available at: https://www.simplypsychology.org/social-constructionism.html (Accessed 29 December 2023)

So, I’ve just been reading about neurodiversity, disability and user experience design and Learning Management Systems e.g. Moodle. It’s interesting to get myself out of ‘tunnel vision’ about the range of features I feel are designed to make our Moodle design more inclusive, and to think about this more broadly and from different perspectives. This is helping to inform and expand the items I want to ask about in the questionnaire and focus group. One of the students responded to my email delightfully saying they were looking forward to seeing what I ask them about – now I feel a bit more pressure to make that interesting! Especially since they are design students.

I’m thinking, in terms of structure for the session, the first half I want to be about their experience, what they do and what are enablers and what are barriers about the Moodle pages. In line with ‘Standpoint theory’, I want to create some significant space for the participants to shape the discussion and identify the highlight the issues.

‘Standpoint theory views members of underrepresented or marginalized groups as epistemically privileged because they are able to see more clearly the institutional structures and systems that marginalize them.’ (Parson, 2005 p.23).

However, as we progress through the session I want to ask some more detailed questions, or actually to provide prompts for the students to pick up and discuss. This is partly because there may be other aspects of the design that students are not particularly aware of or which they may feel uncomfortable about bringing up, thinking they are too minor or because they don’t want to give criticism.

Parson argues that ‘If seeking to identify how an individual or group of individuals is challenged, research questions should guide the exploration of the experiences and challenges of the groups whose experiences we would like to improve.’ (Parson 2019, p. 22). I’m hoping that the use of prompts will allow sufficient scope for the participants to have this agency, without putting all the responsibility of the session on them.

Focus groups and semi-structured interviews

Although these are different research methods, I’ve put them together because there are some overlaps in the way I want to approach them, and I need to design a version of each that fulfils the same goals.

Whilst ideally I’d like to see what happens in the more open focus group structure, I also recognise that not everyone wants to discuss their neurodivergence or disability with classmates or peers and so I have offered 1:1 interviews as an alternative. In addition, as part of my contingency planning I’m also opening up the project to staff and 1:1 interviews seem like a more appropriate format for speaking to staff.

I am struggling to structure my interviews and focus group though, which is quite unusual for me. This is a part of planning that I usually really enjoy, thinking about the aims of the session and the different ways you might go about getting there. I think one of the reasons I’m struggling is that I want there to be space for the participants to talk, and if they do, that will be fine, I think I’ll be able to follow their lead. What I’m struggling with seems to be the structure, what are the questions that I can ask about this specific topic that aren’t so micro that they stall the conversation (e.g. to what extent is the size of these buttons helpful…? :)), or too personal (e.g. what affect does your neurodivergence have on your experience of Moodle?)

I am, however, reassured by Adams’ view, ‘Once developed, the interview guide, no matter how extensive its preparation, should still be considered a work in progress’. (Adams 2015 p.499). He is very clear that the structure of interviews and also focus groups, can and should change and be reviewed as the sessions progress. I had been worried that somehow this needed to be the same (a bit like a job interview), which would make it very unnatural and I think difficult to be in.

So, the upshot is that I am hoping to provide prompts on individual pieces of paper so that participants can select the different things they want to talk about. This should work either in a focus group situation or in a 1:1.

Another aspect of the session which I think is important is about creating a welcoming atmosphere in room and amongst participants and myself. This has always been something that’s important to me, from being a teacher, and I would like to continue it here. I will ensure that the room is set up to try and suggest an equal footing between us all (no teacher chair at the head of the room). If we are in a focus group scenario, I will be using individual, pair and group work to try and maximise the opportunity for those who are from less dominant social groups to participate fully and in the way they wish.

I’m interested in how Maha Bali and Mia Zamora have incorporated this apparently simple idea into a much wider and deeper concept of Intentional Equitable Hospitality (IEH) in an educational context. They say, ‘IEH begins with the notion that the teacher or workshop facilitator is a “host” of a space, responsible for hospitality, and welcoming others into that space.’ (Bali and Zamora 2022)

Whilst this is an apparently simple idea, it goes much deeper, ‘(IEH) prioritizes the values of social justice while fostering learner/participant agency within the learning space, while never forgetting the ways in which power and oppression work outside of that learning space, and how they influence it.’ (Bali and Zamora 2022)

A change of plan… semi-structured interviews

Since writing the first part of this post, it turns out that my plan for a focus group is likely to be replaced by 1:1 semi-structured interviews. Whilst no-one has explicitly said that a focus group would be uncomfortable, the practical considerations of trying to arrange to meet participants has meant that I will largely be conducting semi-structured interviews. I’ve kept the broad structure and principles of my plan, but amended it for the different scenario.

In the video below, I’ve tried to explain the rationale for my design:

Designing the online questionnaire:

In designing this form, I knew I wanted it to be short (easily completable within 5mins), and mostly tick boxes. I also wanted to ask about neurodiversity and/or disability but I was unsure about the wording for this question.

When considering who my participants might be, I asked for some advice from Carys Kennedy, who used to manage the disability service and she very helpfully sent me a sample question for how she asked about disability when conducting online feedback. I made some tweaks to this to suit my needs, but this was a great start for me to feel confident about the appropriate wording for this question. I had included an ‘Other’ box but changed it for ‘Prefer not to say’ and ‘None of the above’, with the free text box instead, after feedback from Carmen Fernandez, Assistive Technology Coordinator.

Carmen also helped me to see more clearly that I needed to remove a section at the start which attempted to explain why I needed the name of the participant and also their neurodiversity/disability. Inadvertently I was implying that there would somehow be a connection between what they said and their neurodivergence more broadly, even though I was actively trying to avoid saying this. It was unclear and I removed it.

I was concerned that this introductory text was too long, but after talking to my tutor group, I was reassured that it seemed ok and keeping it there was part of due diligence on my part. One of my classmates recommended changing the colour to dark green, which she said was easier on the eye. She also said that the use of tick boxes made it seem not too onerous. Mostly, from my tutor group, I was reassured that repeating topics in the questionnaire and in the focus group/interview was actually fine, and would give some valuable quantitive data and a different medium for participants to share their thoughts.

In writing the online questionnaire, I was concerned that my focus would either be too narrow (focussing on specific elements of design) or too broad, when probably the participants would not have considered their use of Moodle in detail and might not know how to answer. I read some more about UX design, specifically for neurodivergent people (see UX design for neurodiverse, visually impaired and disabled users post) and with more feedback from Carmen I was able to clarify what I wanted to ask, and to feel more confident about doing so.

Online questionnaire

Bibliography:

Adams, W. ‘Conducting Semi‐Structured Interviews’ In Handbook of Practical Program Evaluation, edited by Kathryn E. Newcomer, Harry P. Hatry, and Joseph S. Wholey, 1st ed., 492–505. Wiley, 2015. Available at: https://doi.org/10.1002/9781119171386.ch19. (Accessed 22 November 2023)

Bali, M. and Zamora, M. (2022) ‘2: Intentionally Equitable Hospitality as Critical Instructional Design’ in Designing for Care, Pressbooks Available at: https://pressbooks.pub/designingforcare/chapter/intentionally-equitable-hospitality-as-critical-instructional-design/ (Accessed 7 January 2024)

Parson, L (2019) ‘Chapter 2: Considering Positionality: The Ethics of Conducting Research with Marginalized Groups’ in Strunk and Locke, eds. Research Methods for Social Justice and Equity in Education. Cham: Springer International Publishing pp. 15-32

‘Interwoven, neurodiversity and the creative mind’ Chris Kelly, Artist and workshop lead

In my project I hope to hear in detail about the views of people who are neurodivergent and so it makes sense to use a dialogic research method, where questions can be posed and follow up questions asked, based on responses. Using an interface such as Moodle is also a fairly intuitive exercise for most people and so it may be that the participants haven’t articulated or considered carefully their behaviour on the platform and what does and doesn’t work. Of course it’s equally possible that they have become frustrated with it or conversely, have appreciated changes in organisation. I feel that given the potential diversity of experiences, plus the range of potential participants, the opportunity to have an in-depth conversation is a key element of my project.

Focus groups and interviews: The main alternative to a focus group would be to do semi-structured interviews with each participant. This would of course offer benefits in terms of confidentiality and anonymity and it may allow participants to feel more comfortable to talk more freely. Talking about the advantages of semi-structured interviews, Adams says they are suitable, ‘If you need to ask probing, open-ended questions and want to know the independent thoughts of each individual in a group.’ He also mentions it suits ‘uncharted territory’ which I can’t claim this is for everyone else, but I feel it is for me. I would like to open up the conversations and see what happens.

However, my overall feeling is that given the nature of the topic and the diversity of participants, there is more to be gained from a group discussion. We can’t make Moodle perfect for every individual, what we can do is make it good enough to be inclusive and flexible enough to meet everyone’s needs. Although there are many definitions of focus groups, I feel that this one describes the tone of what I’d like to achieve.

‘Agar and MacDonald (1995) suggest that a focus group lies somewhere between a meeting (reflecting the fact that it is specifically organized in advance and has a structure) and a conversation (reflecting the fact that the discussion has nonetheless a degree of spontaneity, with individuals picking up on one another’s contributions)’ (Sim and Waterfield, 2019, p. 2)

There is also the additional element that the ‘outcomes’ of the focus group should include both dialogue and interaction, that the act of the group conversation leads to something more than the sum of it’s parts. I feel that this could be beneficial in discussing information management on the course as hearing about how others experience and interact with the page, may prompt participants to realise or articulate something about their own behaviour and preferences.

As I mentioned above, it is partly for this reason that I am proposing a focus group rather than individual interviews, despite the risks (see below). In addition, I watched the video above of Chris Kelly’s account of a crochet workshop he held with neurodiverse students where several of the participants talk about the, sometimes unexpected, benefit of coming together as a group, even though neurodiversity is explicitly about diversity, it was still possible to find things in common and to feel a benefit of being in a community. Obviously I can’t be sure this will happen, but it feels like a really positive possibility and it will be interesting to see if there is any sense of this that comes from the focus group.

I think for myself also, the information I hope to discover, isn’t totally clearly defined, it’s not an account and nor is it quantifiable. I feel that the ability to be flexible and responsive when facilitating the focus group will be important to the development of my thoughts, as well as meeting the as yet unknown needs/wishes/preferences of the participants. Furthermore, this flexibility feels like an important element of inclusive research. Although I can’t realistically manage full participatory research for this project,

‘… allowing participants to shape the direction of data collection allows them continued access to the research process.’ (Larson 2019, p.28)

I wonder if there is scope to offer a little more flexibility in the research methods. A simple, but limited, way would be for participants to choose questions from a selection, at least enabling participants to focus on the aspects they chose to.

Kerschbaum and Price write about ‘centring disability’ when conducting qualitative interviews. This has made me think about whether I can be more flexible about the methods I use. For example, I have been agonising about whether to invite students to a focus group, with an interview as an option, but maybe I need to frame this more as a positive choice, rather than a way of ‘dealing with’ privacy. For example, they say ‘Centring disability is about much more than simply compensating for or including disabled researchers and participants’ (Kerschbaum and Price 2017, p.98). It’s not an awkward thing to deal with, it’s a choice.

I see more also how my methods of recording and the interviews and gathering data, might need to be more flexible, but that also might bring about something more positive, for example, using captions or interviewing online. Kerschbaum and Price’s argument is that thinking of these arrangements as something to fully pin down in advance, reduces the chance of an opportunity for extra insight. As they say, ‘we recognise our strengths that stem from disability and and we make access a higher priority than maintaining consistency across different interview modalities’ (Kerschbaum and Price 2017 p.100). For example, perhaps it’s possible to keep some elements of the modality open, depending on participants’ responses, thus making space for greater agency and accessibility.

Risks: There are risks to using a focus group, including that participants may not want to disclose their diagnosis to peers, or any difficulties they may have in accessing information. There is also the risk that participants are less open about what they experience and may follow along with more dominant participants. There is of course also the risk that confidentiality is not maintained and therefore even after the focus group, there may be repercussions that are difficult to control and mitigate.

Strategies: In order to address some of these issues, I will start the focus group with a ‘briefing’ which outlines the purpose, the idea of a confidential space to talk and the importance of trust between participants. I’ll also briefly remind participants of this before they leave. In addition to this, I’ll plan in some time for participants to talk in pairs before sharing with the whole group, to make space for everyone to contribute, and also incorporate some individual responses, before asking the group to discuss. I’ll also try to create a friendly, welcoming space through arranging the room nicely, having snacks and drinks available and having things on the table for people to doodle, fidget or whatever they like to relax. I’ll also start the session with some gentle chat to try and generate a relaxing atmosphere.

Another strategy here would be to make use of an online questionnaire to offer an opportunity to participants to contribute confidentially. As well as sharing their experiences this way, they could also disclose their neurodiversity on the form, meaning they wouldn’t need to do this in front of their peers. Finally, it would also give me the opportunity to ask some targeted questions without interrupting the flow of the discussion. This might also help to make the focus group more relaxed, more of a conversation than a Q&A session.

Drawing: One thing I am interested in doing is asking participants to draw their experience of Moodle. I think this is because I often think of experiences visually, through a metaphor or a rough drawing e.g. scribbles for confusion. I also looked at the ‘draw and write’ technique described by Helen Kara as ‘a tried and tested technique of gathering data that enables children to express their views and opinions in their own terms’ (Kara, 2015, p.89). Although Kara describes using the technique with children, which is not relevant to me, I was drawn to the idea of participants expressing themselves in their own terms. After listening to Dr. Jenni Good talk about using drawings as a form of data collection in her Cross-Programme talk, I feel more confident in pursuing this. As my topic is quite dry, I think it might open up a different perspective on how it actually feels to use Moodle, and also, potentially, might offer participants a medium they could be more comfortable with using.

I am concerned though that this might make participants feel uncomfortable or self-conscious, so I will make it very clear that the level of drawing is unimportant and also they are free to do it in any way they choose. Of course it will also be fine for anyone to say they’d rather not participate in this element.

References:

Adams, W. ‘Conducting Semi‐Structured Interviews’ In Handbook of Practical Program Evaluation, edited by Kathryn E. Newcomer, Harry P. Hatry, and Joseph S. Wholey, 1st ed., 492–505. Wiley, 2015. Available at: https://doi.org/10.1002/9781119171386.ch19. (Accessed 22 November 2023)

Kara, H. (2015) Creative Research Methods in the Social Sciences: A Practical Guide. Bristol: Policy press.

Kerschbaum S. and Price M. (August 2017) Centring disability in qualitative interviewing Research in the Teaching of English, Volume 52

Parson, L (2019) ‘Chapter 2: Considering Positionality: The Ethics of Conducting Research with Marginalized Groups’ in Strunk and Locke, eds. Research Methods for Social Justice and Equity in Education. Cham: Springer International Publishing pp. 15-32

Sim J. and Waterfield J. (2019) Focus group methodology: some ethical challenges Quality & Quantity Available at: https://doi.org/10.1007/s11135-019-00914-5 (Accessed 6 Nov 2023)