Finally! Your assessment project is almost done. You had a goal, designed a project, collected and analyzed your data from (let’s say) five focus student groups. And now you have written up a report including some implications and suggestions for change, and are ready to share with your team and to start talking about next steps.
But wait! The analyzing and writing process can take several weeks or months, meaning that a large gap between when you ‘heard’ the student voices in the focus groups and when you finish a summary and interpretation of what they shared. And in this process of preparing a draft report, what might have seemed very important initially, can become a bit less clear and evident as you are immersed in all the data. Remember in trying to summarize data from focus groups and interview we must try to ensure that what we report on is based upon an accurate representation of what the students shared with us. There are lots of ways to help ensure a fair representation of the data from the students, like having audio transcripts and taking personal notes, but there are two other steps (that don’t take very long) and can add to the trustworthiness and credibility of your final reports.
One approach is to share a summary of the key findings, themes and representative quotes with your participants. This is often called a member check. In this case we are asking the members (students in the focus groups) if they “see” themselves in the findings and summaries you have created. You want them to tell you what you got right and what you might have (unintentionally) under or overstated. This process also gives the students one more chance to share with you. Why is this extra reflection useful? Think about how many times you have been in an interview or conversation and then later had that feeling afterwards of “Why didn’t I mention that?” Using this process, you get the benefit of their feedback for accuracy and the time they have had to reflect on the focus group discussion.
A second approach is to share your transcripts with a colleague who is acting as a “peer debriefer.” Ask them to read the transcripts, or whatever summaries of the focus groups you have. Have your colleague identify what they think the major findings and learning’s from these same conversations. This can be helpful to check against some of our own bias and potential blind spots. Who should you choose to help out? A good person to ask is someone who knows of, but isn’t totally involved in your program. When they have shared back with you, you can then discuss where your interpretations are aligned and where they might diverge. This process will not likely lead to total agreement, and it may lead to some re-writing on your part. However, it is a very useful step that should make you even more confident that what you share with your team, the data you use to inform decisions about what to do and not do, are based on the best analysis possible.
These are two ways we can help to ensure that the process we use to learn more about our programs and services results in the fairest representation of the stories the students share before we begin to make changes.
This post is part of our #SAassess series on the importance of assessment in student affairs as a state of mind. A variety of knowledgeable and relatable perspectives will be portrayed throughout the month of November. We hope you will gain inspiring insights and take time to reflect on how you make meaning of your data collection and assessment practices. For more information, check out the intro post by Kim Irland. Be sure to read the other posts in this series too!