Q&A: Impact of Captions & Transcripts on Student Learning & Comprehension
At the University of South Florida St. Petersburg (USFSP), the accessibility committee is working diligently to make the university’s educational content accessible to all students.
In the webinar, “Impact of Captions & Transcripts on Student Learning & Comprehension,” Karla Morris, Lyman Dukes III, and Casey Frechette, three researchers leading a study at the university, discuss their findings. They show how closed captioning and interactive transcripts impact a student’s ability to retain, recall, and comprehend educational content.
Below is the Q&A portion of the webinar, where the presenters answer viewers’ questions.
What was the most surprising aspect of the study?
Casey: I would say that a real challenge we faced was retention across the entire semester we were collecting data.
It was a challenge, in the beginning, to encourage students to opt-in. We had a pretty strong opt-in rate on the whole, but getting students to follow through throughout the semester, particularly to come back and take the post-test, was a real challenge.
I don’t have the numbers in front of me, but we had at least two times as many students start the research as we did finish it all the way through. Something on the order of 200 students signed up, opted in, took the pre-tests, but they didn’t finish all the way through and take the post-test.
You showed the 3% and 8% test score increases, but how do you define the benefits that students receive? How were those benefits defined and measured?
Casey: Those percentage increases are including recall, comprehension, and transfer of questions. Those particular changes were basically saying student learning increases, and student learning increased more so in the interactive transcript group.
What we’re not able to say just yet, because we haven’t done this part of the analysis, is are there differences when we drill down into those three types of learning? Do we see differences in recall, for example, versus comprehension?
We’re also at a point now where we’re about to do a dive into the attitudinal data that we’ve collected to see if student perceptions of their experiences changed or may have been enhanced. Were they feeling more motivated? Were they feeling more focused? Those facts are not yet accounted for because we haven’t done that part of the analysis just yet, but that’s on our radar next.
Will you share data on how many students turned off the captions?
Lyman: We do have that data. We’re certainly in a position to report that the data will be included in the 3Play report that we’ll complete sometime in the fall of 2019.
Did you offer any incentives to the students for using captions?
Karla: Yes, we actually did. Through 3Play Media, we were able to acquire funding to purchase $25 Amazon gift cards – and we had a finite number for those. I think the total was 50 for all of the groups. In the end, we told them that if they participated and completed all of the checkpoints and the post-tests, that they would be randomly entered into a drawing to win one of those gift cards.
You spoke about the power of captioning and interactive transcripts for higher education. Do you think the results of the study could be recreated in a high school setting?
Karla: Yeah, I wouldn’t see why not – especially for our virtual schools. I certainly think as long as the students know how to use the tools and the faculty know how to advise students on how to use the tools to benefit their learning, that we could certainly see that result.
Casey: Just to piggyback on that, I would also say that there could be some distinct advantages in exposing students to these tools earlier on. There is a certain learning curve and a certain acclimation process that we’ve observed across our pilot studies and our current research. I think that the earlier that students are familiar with using these tools to become more active in their own learning, the better.
Lyman: Just to add to what Casey said, it was one of the reasons why we also wanted to orient students to the tools, particularly the interactive transcripts, prior to the students actually employing those tools in a live course environment.
Was there a group without closed captions and the interactive transcript which shows their percentage increased from pre- to post-assessment?
Lyman: There was not. While in an ideal sort of study situation, one would have control of who didn’t receive either of the treatments, it’s important to ensure that each student had the reasonably equitable benefit of tools that would allow them to perform at their highest level within the course.
We made the decision to not include control of that nature, and frankly, I’m not sure that our institution would have given us permission to do so if we had asked. They would have, I suspect, wanted all students to have access to either the captioning or the interactive transcript tool.
This study focused on online courses, but how you think this could work for a regular classroom space?
Karla: I think it would be interesting to examine a course in a face-to-face setting. If you’re going to caption or transcribe a live lecture, that’s something that could be shown on the screen for all to see, rather than just on a laptop for the student who needs the captioning.
Casey: I think that as live transcription software becomes better and becomes more accurate, this is a really interesting possibility for us to be thinking about in live classroom settings. Also, just to underscore it, I would say in any case where there is a hybrid kind of class, where maybe it’s an in-person class but there’s material posted online, of course, we see tremendous value in these tools.
One of my classes is video storytelling. We watch a number of examples in class. Even as we’re discussing it today, I’m realizing I’m not always thinking about enabling closed captioning or showing examples with closed captioning when we’re reviewing media in class. I think it’s really great to be thinking about how all of this could apply in that face-to-face setting.
How are you using the findings to talk to faculty to get buy-in from them?
Karla: I think that’s definitely something we are going to be thinking about as we compile our final report for 3Play Media. The good thing here at USFSP is we have faculty that know we’ve been working on these areas. Just this morning, a faculty member stopped by and said, “What did you find? Does this mean that we’re doing interactive transcripts starting in the fall?”
We’ll probably plan to share the report through formal communications. We’ll probably offer professional development workshops, maybe partner with our Center for Innovative Teaching and learning, and really just include it as a part of the conversation when we do course design and instructional design intakes with our faculty members. We’ll also reach out to our network of champions and make sure that is our faculty.
Casey: One thing that I think we’ve tried to focus on is the broad range of benefits that we’re seeing in these assistive tools. An earlier study that we published was subtitled “Examining the Value of Closed Captioning for all Students”, and this reflects some of those qualitative comments that we saw, where students who may not have English as their first language, students who find themselves studying in different environments, and students who see themselves as visual learners are all benefiting from these tools.
Lyman: It’s also worth pointing out that recent data indicates that 7 out of 10 students with documented disabilities leave high school and actually choose not to disclose upon entering college. So while these tools certainly help all students, people affiliated with institutions of higher ed should be aware they’ve got a large percentage of students with disabilities on their campuses that aren’t disclosing, and therefore, aren’t receiving services. So universal design tools of this nature are actually benefiting them, even though they’re not officially documented students with disabilities. So having these tools for many reasons is quite important.
What percentage of students in your study had a disability?
Karla: That is a good question. I’m afraid that we don’t have an answer to at this moment. It’s not part of the data that we’ve analyzed just yet.
Where were students watching videos? Was it mainly on desktop or on their mobile phones?
Casey: That’s another really interesting question, and we don’t explicitly ask that in our survey. If I’m not mistaken, I don’t think that was one of our questions, although I do believe that we have that as part of our tracking data. It’s a really good prompt for us as we move into the next phase of our analysis to examine things like screen size, operating system, and browser. I believe that we’ve captured that as part of the observational data, and it may be another interesting covariant to look at as we get deeper into the results.
Are you providing students with verbatim captions? If so, have you faced any resistance to them?
Karla: We are providing verbatim, and we haven’t had any resistance.
Lyman: If I recall, if there was resistance, it would be to captions that are not verbatim as opposed to verbatim captions.
Casey: Another issue that has come up for us in conversations with our colleagues is whether to allow for printing of full transcripts. That’s something that we haven’t formally dealt with in our research, but it’s a factor that we know has come up. Some faculty do have concerns about maintaining control of their content. It hasn’t emerged as a show-stopper by any means, though. All the faculty that we’ve worked with have been happy to provide full transcriptions of the lectures that we have included in our research. We should also note that although we only studied the first module video for each section, all of the videos in these classes were captioned, and transcriptions were provided for all of them.
If you’re working with a limited platform, what sort of advice do you have for providing captions and/or interactive transcripts?
Lyman: There is good news when we look at all of the major video platforms, from YouTube to Vimeo. From what I’m observing, their closed captioning and interactive transcript options are getting better over time. Vimeo, in particular, has made a lot of strides in recent years to increase the functionality of their captioning.
There’s good momentum in general, and one of my tips would be to keep an eye out on third-party platforms that can be used if you’re facing limitations within whatever software you’re using to deliver your content. There’s probably an option out there that’s free or low cost that does enable the kinds of tools we’re talking about today.
Was downloading transcripts an option in your study? If so, what is one benefit of providing the download option?
Karla: We disabled that option in our study because of the concern that Casey had brought up – the concern for redistribution of that content.
Casey: I think that one implication of this is around the note-taking. Downloading transcripts could be used as part of a studying process, part of a note-taking process, and then particularly for students who like to be more tactile in their learning – highlighting content, taking notes in sidebars. Alongside the concerns around copyright or protection of content, it’s also important to keep looking at what the learning benefits of being able to print out partial transcripts and using that as a study guide or note-taking tool might be.
Lyman: I’ve used a very simple means of providing students access to printing out content that’s included in an online PowerPoint lesson in this particular case. What I do in my case is to provide very rich descriptions in the notes version of PowerPoint, providing a deeper dive regarding what’s included in the slide itself. That doesn’t necessarily address the specific question here but is the means by which I provide students an opportunity to print information that may be talked about over a presentation so they can use it for study purposes.
How did the university respond to the findings from the study?
Karla: They’re so preliminary that we haven’t shared the findings with them just yet, but I think once we have all of the data analyzed and the report prepared, we’ll be sharing it. I think that their response in general to all of our work has been really positive. It’s really advanced the profile of the institution, it’s support for our students, and really showed the initiatives that we have in place.
Lyman: Accountability metrics are very much in play in our state. Our funding as an institution is tied to a set of accountability metrics that apply to all the public universities in the state of Florida.
My expectation is that the institution will be very interested in the findings and we hope that they do provide some benefit in the near term to students on our campus.
Watch the full webinar, “Impact of Captions & Transcripts on Student Learning & Comprehension.“
Canadian CRTC Caption Laws
Canada’s deaf and hard of hearing population have been advocating for higher captioning standards since the 1960s. The Canadian Radio-television and Telecommunications Commission (CRTC) has done much work in the last decade to give Canadians a chance to voice their opinions and…
How to Design a Sustainable Process for Accessible Web Design
Designing an accessible website is more important than ever before. In fact, 71% of people with disabilities leave a website immediately if it’s not accessible. Slow a steady wins the race, and accessible web design is more than just a one-time auditing…
How to Get Started with 3Play Media’s Audio Description
Audio description is an audio track that narrates the relevant visual information in media and assumes the viewer cannot see the onscreen content. It can be compared to a sports broadcaster narrating the visual action of a game over audio. The descriptions…