Australian Research Study: How Useful Are Closed Captions for Learning?
Adding to the rapid-growing number of studies surrounding closed captioning in higher education, an Australian academic study on how closed captions impact student learning in mathematics has just been released.
The study involved 135 students at the University of New South Wales (UNSW) in Sydney and investigated the perceived level of usefulness that both accurate and automated captions lent in a mathematics course over a 2-month period.
It was discovered that students broadly agreed that captions are a useful learning feature allowing flexibility of where and when a video is watched, while helping to understand speaker accents, and clarify explanations that are difficult to hear in the recording.
Dr. Chris Tisdell, who is one of two authors on the study, Associate Dean of Education, and Associate Professor of Mathematics at UNSW, will present on the study and main takeaways from the insights as they apply to closed captioning use in higher education everywhere.
This presentation will cover:
- Data & results from the student study
- Levels of student engagement with closed captions
- How and why students used closed captions in the study
- How students perceived the usefulness of accurate vs. automatically-generated captions
- The percent of overall students responding who report using closed captions as a helpful learning aid
- Takeaways from the study
Dr. Chris Tisdell
Study Author | University of New South Wales
Patrick Loftus (Moderator)
Marketing Assistant | 3Play Media
Webinar Q&A: Australian Mathematician Studying How Closed Captions Help Students Learn
We reached out to Dr. Chris Tisdell after discovering an article published on his research paper about how students find closed captioning helpful as learning tool.
We found his research particularly interesting because we had also just released results from our nationwide student closed captioning use survey and from our institutional survey on administrative implementation of closed captioning.
Chris, who teaches mathematics courses at the University of New South Wales in Sydney, Australia, graciously agreed to present on his research findings in a webinar with us.
Since 2008, he has also run a YouTube channel with a massive following and millions of viewers which is dedicated to teaching mathematics to people all over the world through fun and educational video content.
Chris’ report, co-authored with Dr. Birigt Loch from Swinburne University of Technology in Melbourne, is based on results from a university level mathematics course in which students used closed captions on course video content and then were asked how helpful they were for learning. Read on for highlights from our Q&A:
Did international and ESL students respond that they found captions most useful for “understanding” accents, or was it more correlated to helping them better comprehend the English content?
- CHRIS TISDELL: I think it’s a combination of both.
I’ve been in higher education for about 15 years. And in Australia, we have a lot of international students. And one of the common complaints here is that the lecturer, the tutor, they speak with a strong accent.
English might not be the first language of their teacher or of their tutor. So, accents are really important. And being able to manage accents and understand a thick accent is really important.
There were some students who identified that they’ve found benefits. One of the benefits in using closed captioning was “you’re kind of accent-independent,” which I found was great. Even my videos on YouTube — just around the world, I get people saying, “oh, nice accent,” or “funny accent,” or “I can’t understand your accent.” So I think it’s a big issue. And we’re a very diverse world. So we all speak a little bit differently.
Are you planning to conduct this study again with a control group and focus on if it improved their learning?
“ Now, some people think that student perceptions don’t hold much weight. I would argue against that. […] I think that listening to the student voice is important.”
- CHRIS TISDELL: [Not having a control group] is another limitation of the research. We would love to do that. Part of having the control group is a little bit tricky because we need to make sure that we are fair in the sense that we’re not excluding people from gaining access to certain resources. But that is something that I would love to pursue maybe in the next phase of the project. And that’s certainly a deeper, and more serious, and more important element.
So at the moment, we’re just looking at perceptions. Now, some people think that student perceptions don’t hold much weight. I would argue against that. Our university, we get the best students in the state with the highest entry scores. We teach them to be critical thinkers. I think that listening to the student voice is important. But certainly, I would love to have a bit more of a controlled approach and be able to take the next step.
Do you have best practices for captioning math content and equations?
- CHRIS TISDELL: Not yet. That is one discussion point that I would love to unpack a bit more. Mathematics is tricky because it has difficult or sometimes complicated expressions. It’s the same with physics, or chemistry, or any of the technical and scientific fields. So, when you’re working with a basic ASCII type keyboard, it’s very tricky.
We’ve tried to replicate equations where we can. It’s not like you’re captioning a speech by Obama, where it’s just basic words. It’s very tricky.
One of the things you’ll notice if you go and watch some of my videos is that the equations are already on the screen. So, if a viewer can already see an equation on the screen, is it necessary to actually have the captioning again? That’s one of the questions that we’ve faced. I don’t have an answer to that.
But certainly if you’re going to take this to the more scientific fields with closed captioning, the technical equations and whatnot, they’re going to be a barrier. And that’s something we need to explore, whether it’s LayTeX integration with closed captioning, which is a beautiful typesetting program. Maybe that’s the future.
Can you talk about how uploading transcripts to YouTube works? How does it auto-time? Does the transcript require specific formatting for this to work properly?
3Play Media’s round trip integration with YouTube provides an automated workflow for adding captions and subtitles.Your YouTube videos can be processed in a matter of hours and captions will be automatically sent to YouTube and added to your videos.
- CHRIS TISDELL: The sky’s the limit here. You can get special files, SRT files, or something like that with all time-stamped text and whatnot. We took the simplest approach possible. So, what we did was we got the students to work with a Word file — just a simple Word file. And then they would send the Word file to me, and I would match it up with the video. And basically, you’re just copying the text out of the Word file and putting it into a little box when you want to edit the closed captioning for a particular video on YouTube. It was fine.
The syncing was fine. Automatically, it was very, very easy to do. So, you can have really amazing programs like VTT or SRT files. But we didn’t go that approach. We went the simplest approach that we could all do, which was using text in a Word file that we copied and paste into the appropriate YouTube video place for each video. It was very easy. And the alignment between the captioning and the words, YouTube did a very good job of doing that. And it was automatic.
Did anyone note that the translations were not helpful because of their quality?
CHRIS TISDELL: Yes. We were relying on Google Translate pretty much. I think that’s embedded in the YouTube translation service. That was a big [issue] for some of the international students or ESL students.
The quality of the translations is not quite there yet. So, maybe in five years or whenever, it’ll be much better. But some of the students said, “OK, some of the translations are wrong, but we can still get an idea of what’s going on.” So, it’s not 100% accurate, but you can get a general idea of what’s going on.
Were there any instances where students strongly disagreed that captioning was useful to their learning?
CHRIS TISDELL: Certainly for the translations, there were some “strongly disagrees”. Maybe 1% of students strongly, strongly disagreed, about 1% mildly disagreed. And there were no “disagrees”.
How do you think research will help generate buy-in for captioning at universities that currently only caption for accommodation requests?
“ First of all, [closed captioning] is something that legally we should be providing. Secondly, the students want this. There is a huge market for this.”
- CHRIS TISDELL: My university is the same. You have to ask for these sorts of things if you want them. I think, as a society, universities should be making evidence-based decisions and having good research — not just from this study, but I know you guys have been doing stuff
- , as well — that is evidence-based and evidence-informed decision-making.
First of all, this is something that legally we should be providing. Secondly, the students want this. There is a huge market for this. The students see a lot of benefit in this. That’s a pretty power– And the evidence is that the students highly value this. That’s a pretty compelling argument. So I think now we’re in a position to really move this forward. And that’s what I’m hoping to do.
What areas do you see for future research?
- CHRIS TISDELL: I think that the translations are still sort of a horizon event here, in a sense. How can we improve the experience when using translations? How can we improve the accuracy of translations? That’s really important.
I think we still haven’t solved the captioning of technical parts, like equations, scientific formulae — that sort of thing. I think that’s important. I know you guys at 3Play Media are doing a lot with the interactive transcripts. I see that as sort of really going amazing places.
Do you think that these results would be the same for other subject matters? Or, do you think that they were so positive for math because of its complexity?
- CHRIS TISDELL: One of the challenges with social science research is the ability to replicate experiments. It’s a bit tricky.
This is sort of discipline-based; in math, and science, and engineering — STEM.
It would be interesting to look at the research that’s been done outside of STEM. So, if somebody’s thinking about doing that, I’d be very happy to help. I would hope it would be the same. I’m not sure why it wouldn’t. But that remains to be seen.